os/bluestore: trim SSD devices on mkfs

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20687 from Songweibin/wip-misc-cleanup

mon,osd,crush: misc cleanup

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20485 from jcsp/wip-23017

mgr: quieten logging on missing OSD stats

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20573 from liewegas/wip-mgr-ovh

qa/suites/rados/rest/mgr: provision openstack volumes

Merge pull request #20692 from idryomov/wip-krbd-huge-image-test

qa: krbd huge-image test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: krbd huge-image test

We've had multiple overflows in ceph_calc_file_object_mapping().
It wasn't being used by rbd, but it now is.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: rename blkroset to basic

More simple fixed-1.yaml based tests are coming.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #20680 from cbodley/wip-qa-dnsmasq

qa: clean up dnsmasq task and fix EPERM error

Merge pull request #19252 from joke-lee/website_redirect

rgw: fix s3 website redirection error

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20635 from mdw-at-linuxbox/wip-master-rgw-openssl

rgw: curl* reuse and for debian, use openssl not gnutls.

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20673 from ceph/wip-rm23148

docs fix ceph-volume missing sub-commands

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

crush: should break the loop if leaves overfull

Signed-off-by: songweibin <song.weibin@zte.com.cn>

osd: drop redundant assert check

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #20684 from liewegas/wip-bluestore-cache-debug

os/bluestore: no trim debug noise if there is no trimming to be done

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: no trim debug noise if there is no trimming to be done

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20637 from wjwithagen/wjw-fix-MDSmonitor.h

mon/MDSMonitor: fix clang build failure

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #20671 from batrick/rados-doc-flags

doc: init flags to 0 in rados example

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20653 from xiexingguo/wip-fix-upmap

mon, osd: fix potential collided *Up Set* after PG remapping

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20655 from xiexingguo/wip-fix-calc-upmap

osd/OSDMap: skip out/crush-out osds

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20506 from jan--f/prometheus-rm-outdated-daemon-metrics

pybind/mgr/prometheus: don't export metrics for dead daemon; new metrics

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20667 from tchaikov/wip-qa-standalone

qa/standalone: s/delete_erasure_pool/delete_erasure_coded_pool/

Reviewed-by: David Zafman <dzafman@redhat.com>

qa: clean up dnsmasq task and fix EPERM error

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20648 from trociny/wip-snap_set_diff-empty_snapset

librados/snap_set_diff: don't assert on empty snapset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20613 from Songweibin/wip-23038

rbd: import with option --export-format 2 fails to protect snapshot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20608 from Songweibin/wip-rbd-snap-purge

rbd: allow remove all unprotected snapshots

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20578 from trociny/wip-pybind-group

librbd: small cleanup for recently merged code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20621 from cbodley/wip-rgw-data_lock

rgw: don't hold data_lock over frontend io

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #20449 from cbodley/wip-rgw-beast-coroutine

rgw: switch beast frontend back to stackful coroutine

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/install add missing lvm subcommand for ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados add missing lvm subcommand for ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20546 from yehudasa/wip-rgw-gc-aio

rgw: gc use aio

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #20643 into master

* refs/pull/20643/head:
ceph-volume-client: fix command argument order

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #20650 from ceph/wip-rm22841

ceph-volume be resilient to $PATH issues

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc: init flags to 0 in rados example

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20656 from yaozongyou/rbd-fix-return

librbd: fix missing return in NotifyMessage::get_notify_op

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: allow remove all unprotected snapshots

allow remove all unprotected snapshots when exiting
protected snapshots in the same image.
Fixes: http://tracker.ceph.com/issues/23126
Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: import with option --export-format fails to protect snapshot

Fixes: http://tracker.ceph.com/issues/23038
Signed-off-by: songweibin <song.weibin@zte.com.cn>

mon, osd: fix potential collided *Up Set* after PG remapping

The mgr balancer module are basically doing optimizations based on
the snapshots of OSDMap at certain moments, which turns out to be
the culprit of data loss since it can produce bad PG mapping results
sometimes while in upmap mode.
I.e.:
1) original cluster topology:

-5       2.00000     host host-a
 0   ssd 1.00000         osd.0       up  1.00000 1.00000
 1   ssd 1.00000         osd.1       up  1.00000 1.00000
-7       2.00000     host host-b
 2   ssd 1.00000         osd.2       up  1.00000 1.00000
 3   ssd 1.00000         osd.3       up  1.00000 1.00000
-9       2.00000     host host-c
 4   ssd 1.00000         osd.4       up  1.00000 1.00000
 5   ssd 1.00000         osd.5       up  1.00000 1.00000

2) mgr balancer applies optimization for PG 3.f:

            pg-upmap-items[3.f : 1->4]
3.f [1 3] + -------------------------> [4 3]

3) osd.3 is out/reweighted etc., original crush mapping of 3.f changed
   (while pg-upmap-items did not):

            pg-upmap-items[3.f : 1->4]
3.f [1 5] + -------------------------> [4 5]

4) we are now mapping PG 3.f to two OSDs(osd.4 & osd.5) on the same host
   (host-c).

Fix the above problem by putting a guard procedure before we can
finally encode these *unsafe* upmap remappings into OSDMap.
If any of them turns out to be inappropriate, we can simply cancel it
since balancer can still re-calculate and re-generate later if necessary.

Fixes: http://tracker.ceph.com/issues/23118
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20640 from tchaikov/wip-async-dpdk-temp-str

msg/async: avoid referencing the temporary string

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #20531 from xiexingguo/wip-ec-scrub-sleep

osd/ECBackend: inject sleep during deep scrub

Reviewed-by: Sage Weil <sage@redhat.com>

qa/standalone: s/delete_erasure_pool/delete_erasure_coded_pool/

it's a regression introduced by ac56a202

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: fix clang error

The specific specialisations in both .h and .cc does not really fly
with Clang

/home/jenkins/workspace/ceph-master/src/mon/MDSMonitor.cc:79:22: error: template parameter redefines default argument
template <int dblV = 7>
                     ^
/home/jenkins/workspace/ceph-master/src/mon/MDSMonitor.h:76:23: note: previous default template argument defined here
  template<int dblV = 7>
                      ^
1 error generated.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

pybind/mgr/prometheus: extend daemon metadata with hostname and ceph version

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: add fs and mds metadata metrics

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: new mon metrics

Remove mon_quorum_count and replace it with per-MON quorum status
(mon_quorum_status). Also add mon_metadata metrics.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: add Metrics class to manage Metric instances

The central change of this commit is that per-daemon metrics are now
managed by first appending the metric (using Metrics.append) to a
staging area. Then the metrics for specific paths (metric names) are
overwritten by the staged metrics (by calling Metrics.reset). This gets
rid of metrics from daemon that are no longer in the cluster. I.e. when
ceph no longer reports metrics for one OSD daemon (because it was
removed from the cluster) the prometheus module will no longer export
metrics for that daemon.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

mon: drop unused variable - 'args'

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #20630 from joscollin/wip-doc-monitoring

doc: Update monitoring.rst

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>
Reviewed-by: Lenz Grimmer <lenz@grimmer.com>

doc: Update monitoring.rst

Updated monitoring.rst with
* Latest command output
* misc doc fixes

Signed-off-by: Jos Collin <jcollin@redhat.com>

rbd: fix return is missing in NotifyMessage::get_notify_op

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

osd/OSDMap: skip out/crush-out osds

We might do 'ceph osd out <osd.x>' or 'ceph osd crush reweight <osd.x> 0'
for various reasons, and hence can produce 0-weighted OSDs.
Skip those OSDs when trying to calculdate PG upmaps so we won't be able to
hit the *assert* below:

/build/ceph-13.0.1-2232-g64665c7/src/osd/OSDMap.cc: 4179: FAILED assert(target > 0)

See also:

http://pulpito.ceph.com/xxg-2018-02-28_09:02:53-rados-wip-fix-upmap-distro-basic-smithi/2235497/

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph-volume process always try to expand the executable

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util ensure behavior of new which utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.system add a helper to find system executables

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20646 from tchaikov/wip-qa/run-standalone.sh/python-path

qa/run-standalone.sh: set PYTHONPATH for FreeBSD also

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20634 from tchaikov/wip-delete-pool-in-standalone-test

qa/standalone: extract delete_pool()

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #20619 from theanalyst/doc/release-notes/12.2.4

doc: 12.2.4 release notes

Reviewed-By: Alfredo Deza <adeza@redhat.com>
Reviewed-By: Lenz Grimmer <lgrimmer@suse.com>

librados/snap_set_diff: don't assert on empty snapset

Instead treat the diff as a full-object delta.

Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/run-standalone.sh: set PYTHONPATH for FreeBSD also

do not set PYTHONPATH=/usr/lib/python2.7/dist-packages anymore, it will
be overridden later on anyway.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19015 from hansbogert/master

mon: Add `ceph osd get-require-min-compat-client` command

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #20607 from shangfufei/test_stack

msg/async:fix the incoming parameter type of EventCenter::process_events()

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20509 from liewegas/wip-less-filestore-idempotent

qa/tasks/filestore_idempotent: shorter test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20498 from liewegas/wip-22616

os/bluestore: trim cache every 50ms (instead of 200ms)

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20616 from tchaikov/wip-cmake

cmake/BuildBoost: fixes to ready seastar

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20636 from tchaikov/wip-23169

Revert "core: hint the dout()'s message crafting as a cold code."

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #20631 from tchaikov/wip-cmake-systemd

cmake: s/sysconf/sysconfig/

Reviewed-by: Nathan Cutler <ncutler@suse.cz>
Reviewed-by: Lenz Grimmer <lenz@grimmer.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

ceph-volume-client: fix command argument order

Fixes: http://tracker.ceph.com/issues/22038
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg/async: avoid referencing the temporary string

* get_val<std::string>(...) returns a temporary std::string, so we cannot
  keep a reference to it after evaluating this method. so convert it to
  an integer right away in the same expression.
* use std::stoull() with base = 16, so we can parse a hex string
  representing up to 64 bits.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15849 from theanalyst/wip-doc-es

doc: rgw add some basic documentation for sync plugins & ES

Reviewed-By: Yehuda Sadeh <yehuda@redhat.com>

doc: 12.2.4 release notes

And updating the release table

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Revert "core: hint the dout()'s message crafting as a cold code."

This reverts commit b265ed2955570c17b954270e6a1449637790e9b8.

with b265ed29, we cannot use `dout(10) << __func__ << ...` anymore for
logging the name of the inner most function. what we will have is
always `operator()`, because the lambda closure will be the inner most
function with the the hinting change.

Fixes: http://tracker.ceph.com/issues/23169
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/standalone: extract delete_pool()

some tests, like osd-backfill-stats.sh are using delete_pool(), but
they don't have this function defined. and this function is defined
in standalone tests separately, so would be simpler if we can
consolidate them in ceph-helper.sh.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20411 from majianpeng/sync-nvml-submodule

cmake: sync nvml submodule to latest code.

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: make PMEMDevice work based on latest pmdk.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

cmake: sync nvml submodule

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

cmake: s/sysconf/sysconfig/

it's a regression caused by 638aadf

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20618 from tchaikov/wip-systemd-deb-env

cmake,deb: install system units using cmake

Reviewed-by: Nathan Cutler <ncutler@suse.com>

rgw: reuse CURL* for keystone.

When using keystone with https (the recommended setting),
it is inefficient to start up a new SSL connection for
each and every operation.  Keeping a CURL* structure around
should reduce the cost of doing this.  This logic tries
to do so, but it also tries to free them fairly aggressively
(5-10 seconds).  This should still greatly reduce load on
keystone at peak times while not tying up excess resources.

Fixes: https://tracker.ceph.com/issues/23162
Signed-off-by: Marcus Watts <mwatts@redhat.com>

build/link against curl w/ openssl not nss or gnutls. (debian)

curl + gnutls has some big significant performance hits at least
when doing keystone validation.  nss has long-term memory growth
issues.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #20602 from gregsfortytwo/wip-23141-rotational

os/bluestore: correctly check all block devices to decide if journal is_rotational

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20595 from jcsp/wip-23037

mgr: die on bind() failure

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20593 from tchaikov/wip-23121

ceph-objectstore-tool: don't destroy SnapMapper until the txn is completed

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #20540 from ovh/bp-optracker-cleanup

common/OpHistory: move insert/cleanup into separate thread

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20539 from zeichenanonym/mgr-prometheus-fix-metadata-does-not-exist

mgr: prometheus: Don't crash on OSDs without metadata

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20530 from wido/issue-23078

common/dns_resolv.cc: Query for AAAA-record if ms_bind_ipv6 is True

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20394 from liewegas/wip-pg-log-size

osd: min_pg_log_entries == max == pg_log_dups_tracked

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20290 from rzarzynski/wip-common-subsystemmap

core: make the main dout() paths faster and more maintanable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20273 from adamemerson/wip-any-singleton-will-do

common: Switch singletons to use immobile_any and cleanups

Reviewed-by: Jesse Williamson <jwilliamson@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20528 from tchaikov/wip-22462

mon: add MMonHealth back

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20415 from wumingqiao/wip-fallocate

filestore: include <linux/falloc.h>

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20620 from ceph/wip-rm23140

ceph-volume: adds raw device support to 'lvm list'

Reviewed-by: Alfredo Deza <adeza@redhat.com>

librbd: normalization for group C API

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20599 from dillaman/wip-librbd-devstack-tempest

qa/workunits/rbd: switch devstack tempest to 17.2.0 tag

Reviewed-by: Mykola Golub <mgolub@suse.com>

rgw: don't hold data_lock over frontend io

get_obj_data::data_lock protects access to the read_list, which the
librados finisher thread appends to on completions

locking isn't needed over calls to d->client_cb->handle_data() in
get_obj_iterate_cb(), because this callback is called synchronously
from the frontend/process_request() thread - no other thread will
be submitting io to the d->client_cb callback

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20612 from fangyuxiangGL/com

rgw: download object might fail for local invariable uninitialized

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19845 from fangyuxiangGL/escape

rgw: rework json/xml escape usage follow #19806

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20352 from yehudasa/wip-22928

rgw: use explicit index pool placement

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20440 from cbodley/wip-qa-rgw-beast

qa/rgw: remove some civetweb overrides for beast testing

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20471 from mdw-at-linuxbox/wip-master-swacldel

rgw: Allow swift acls to be deleted.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: download object might fail for local invariable uninitialized

This is an omission in #20226

Fixes: http://tracker.ceph.com/issues/23146
Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>