ceph-volume tests/functional add mgrs daemons to lvm tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2b7a22ef10e38a7bb4d08c9ebe9736e6452b005f)

Merge pull request #23778 from ceph/backport-luminous-23075

luminous: ceph-volume batch command

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.api update lvm sizing tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8d4525a59d1d7eb1d571848698c69cace29bfa53)

ceph-volume api.lvm convert and use extents for sizing calculations

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 99e034d7eed9508ee016431c040e3fddca9bca69)

ceph-volume util move the str_to_int utility out of lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cb5b30ab43f37adc45fb6114452b37722c65501e)

Conflicts:
src/ceph-volume/ceph_volume/util/__init__.py

ceph-volume lvm.batch update help docs from auto to batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 226a5389ab3379943d3af35e3e1a86481a3980b2)

ceph-volume lvm.main rename auto to batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ae98cc3512d5e66609640066ccbca1a36acfa220)

ceph-volume lvm rename auto to batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d7707650a59855eac75e37ddb4053c62b0dc11e9)

ceph-volume lvm.strategies add stubs for filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fb6a2da7a13cc7edfe496583addbd29c21ecde1a)

ceph-volume lvm.strategies add bluestore with validators

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9f13951414e5e4378a0473c9ddef7e641f4e9d3f)

ceph-volume lvm.auto create a strategies module

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6043a2255dc8e9b4e3b09a4642a16718fd93370d)

ceph-volume lvm.prepare do not pass the name when creating a vg

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 30d17788a915a2f49856fbd20f61638d42553b74)

ceph-volume api.lvm create_vg should optionally name vgs on the fly

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 01aece2b9ab034e8d4419f262a33764664ad93a2)

ceph-volume lvm.main make auto a subcommand

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0cd5a698b163e46b5282fb50d3599fd9f6a4d2b1)

ceph-volume lvm.auto add --yes prompt to execute strategies

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d38e8d95a42357483689c5280c6c29888d14c58a)

ceph-volume lvm.auto help menu and dynamic strategy loading

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9a426d0666d35a8b47d00752c6930b675f8bafda)

ceph-volume lvm.auto move auto as a sub-command of lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7fa5317e2f911f94c867a344c2c5cc5441a63db1)

ceph-volume tests verify Size() behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 079e4c3f3126a00fc9a3036e3f426c07ecc0659c)

ceph-volume util.disk capture real size when inspecting disks

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 165b944c6196c7b760e9ab6b9b2cf9eb6092190c)

ceph-volume util.disk create a Size class for sizing representations and calculations

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c13b12c5bed2f7f2133ba2f4c4b29159fc410f53)

ceph-volume main remove auto as a top-level command

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b9bc2eb628355a8f6d87a41eae2f7bcf83156a81)

ceph-volume api.lvm extract sizing method into its own utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2e9da026339df13e6a6cfb3b9c785d30e49cb1ff)

Merge pull request #23757 from ceph/backport-luminous-23443

luminous: ceph-volume: `lvm batch` documentation and man page updates

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23717 from yuriw/wip-27057-luminous

luminous -  qa/tests: Whitelised 'slow request'

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #23737 from theanalyst/wip-luminous-26887

luminous: qa/tasks/thrashosds-health.yaml: whitelist slow requests

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/man/ceph-volume add batch information

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e00d9bbf53f189ca8d2faff3c673ce61cfeaaa72)

doc/ceph-volume include batch doc in the toc tree

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d3bad7fa766e2ae072aec053dc86393c519978eb)

doc/ceph-volume decument batch reporting and bluestore support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f965f9db2246df2359f8309719fc18613336b1d2)

qa/tasks/thrashosds-health.yaml: whitelist slow requests

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit c0d9f7177c47a2b613fb1a3e9fa56eb90b36973a)

 Conflicts:
- qa/tasks/thrashosds-health.yaml
  keeping MON_DOWN already present

qa/tests: Whitelised 'slow request'
for upgrade/kraken-x
for krbd

Fixes: http://tracker.ceph.com/issues/27057
Fixes: http://tracker.ceph.com/issues/27054
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #23720 from ceph/revert-23696-wip-27057-luminous

Revert "qa/tests: Whitelised 'slow requests'"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Revert "qa/tests: Whitelised 'slow requests'"

Merge pull request #23696 from yuriw/wip-27057-luminous

qa/tests: Whitelised 'slow requests'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: Whitelised 'slow request'
for upgrade/kraken-x
for krbd

Fixes: http://tracker.ceph.com/issues/27057
Fixes: http://tracker.ceph.com/issues/27054
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #23266 from rhcs-dashboard/wip-25117-luminous

luminous: mgr: add units to performance counters

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23595 from dillaman/wip-24387-luminous

luminous: librbd: utilize the journal disabled policy when removing images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #23479 from rhcs-dashboard/wip-22522-luminous

luminous: mgr/dashboard: add configuration setting browser

Reviewed-by: John Spray <john.spray@redhat.com>

librbd: utilize the journal disabled policy when removing images

Fixes: http://tracker.ceph.com/issues/23512
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit cc6ef6d57a71e19dc748890299f41e4b6403058a)

Conflicts:
src/librbd/image/RemoveRequest.cc: trivial resolution
src/test/librbd/image/test_mock_RemoveRequest.cc: trivial resolution

Merge pull request #23379 from dzafman/wip-24875-luminous

luminous: osd: do_sparse_read(): Verify checksum earlier so we will try to repair and missed backport

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23501 from theanalyst/wip-rgw-reorder-luminous

luminous:rgw: have a configurable authentication order

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22655 from liewegas/wip-24481-luminous

luminous: mon/HealthMonitor: do not send MMonHealthChecks to pre-luminous mon

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #23454 from vumrao/wip-vumrao-luminous-25223

luminous: filestore: add pgid in filestore pg dir split log message

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #23459 from badone/wip-OSDHealthMetric-race-luminous

luminous: mgr/MgrClient: Protect daemon_health_metrics

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: have a configurable authentication order

This implements a configurable authentication order, currently used only for s3
authentication and only supporting external & local authentication, though there
is potential for more finegrained control by allowing for a map of various
engines and the control strategy (required vs sufficient vs fallback)

The current implementation just focuses on setting control fallback if the
engine is the last in the order (and hence the stack) and just sets sufficient to
every other element, so that errors from the last sufficient engine is returned.

The configuration option is rgw_s3_auth_order which takes a comma/space seperated
list of authentication engines where currently we support the keywords `external`
and `local`.

Luminous specific changes:
std::string_view -> boost::string_view
auto function return -> return type std::vector<std::string>
if initializer list dropped

Fixes: http://tracker.ceph.com/issues/23089
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 9c7fc682ca23259037115db3437c2bc9dd91fa22)

luminous: filestore: add pgid in filestore pg dir split log message

Fixes: http://tracker.ceph.com/issues/25223
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
(cherry picked from commit d62ea5cc05857bc1a1f9cf4a0e3d4df315bef8af)

Merge pull request #23418 from dzafman/wip-24845

luminous: tools/ceph-objectstore-tool: split filestore directories offline to target hash level

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/dashboard: add configuration setting browser

Signed-off-by: Rubab Syed<rubab.syed21@gmail.com>
(cherry picked from commit 14bb815060a4d156bb5f7cc5d66e77466cba4351)
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
Conflicts:
      src/pybind/mgr/dashboard/base.html
      src/pybind/mgr/dashboard/monitors.html
      src/pybind/mgr/dashboard/osd_perf.html
      src/pybind/mgr/dashboard/rgw_detail.html

Conflict resolution:
- Keep config references while removing references to new monitor and
rgw endpoints and pages.

Merge pull request #23130 from ivancich/wip-fix-dir-suggest-lumious

luminous: rgw: fail to recover index from crash luminous backport

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #22932 from smithfarm/wip-24814-luminous

luminous: rgw: REST admin metadata API paging failure bucket & bucket.instance: InvalidArgument

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #22930 from smithfarm/wip-24632-luminous

luminous: rgw performance regression for luminous 12.2.4

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22934 from smithfarm/wip-24810-luminous

luminous: Invalid Access-Control-Request-Request may bypass validate_cors_rule_method

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22984 from pdvian/wip-24808-luminous

luminous: rgw: fix gc may cause a large number of read traffic

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23000 from pdvian/wip-24830-luminous

luminous: rgw: "radosgw-admin objects expire" always returns ok even if the pro…

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23019 from pdvian/wip-24833-luminous

luminous: rgw: radogw-admin reshard status command should print text for reshar…

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23144 from cbodley/wip-25032

luminous: rgw: add curl_low_speed_limit and curl_low_speed_time config to avoid the thread hangs in data sync.

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #23207 from linuxbox2/luminous-rgw-22790

luminous: rgw: do not ignore EEXIST in RGWPutObj::execute

Reviewed-by: Casey Bodley <cbodley@redhat.com>

luminous: mgr/MgrClient: Protect daemon_health_metrics

Without holding the lock update_daemon_health() can race with
send_report() corrupting the daemon_health_metrics vector.

Fixes: http://tracker.ceph.com/issues/23352
Signed-off-by: Kjetil Joergensen <kjetil@medallia.com>
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 4996506a6b4ab309110039ea29a075f14d09a379)

Conflicts:
src/mgr/MgrClient.cc Function name is different in luminous

Merge pull request #22729 from pdvian/wip-24495-luminous

luminous : osd/Session: fix invalid iterator dereference in Sessoin::have_backoff()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #22918 from smithfarm/wip-24804-luminous

luminous: Python bindings use iteritems method which is not Python 3 compatible

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23054 from liewegas/wip-osdmap-warning-luminous

luminous: messages,mon,osd: silence gcc-8 warnings related to memset()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23113 from smithfarm/wip-24824-luminous

luminous: tests: test_ceph_argparse.py broken on py3-only system

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23155 from smithfarm/wip-25033-luminous

luminous: tests: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) in powercycle

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #23202 from smithfarm/wip-25074-luminous

luminous: Boost system library is no longer required to compile and link example librados program

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23313 from cernceph/dvanders_chown

luminous: rpm: silence osd block chown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22546 from smithfarm/wip-24501-luminous

luminous: osd: eternal stuck PG in 'unfound_recovery'

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #22708 from pdvian/wip-24474-luminous

luminous: librados: fix buffer overflow for aio_exec python binding

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22723 from liewegas/wip-hobject-hash-luminous

luminous: common/hobject: improve hobject hash

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #22773 from smithfarm/wip-24714-luminous

luminous: mon: Add option to view IP addresses of clients in output of 'ceph features'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23014 from batrick/i24190

luminous: client: increase verbosity level for log messages in helper methods

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23169 from ukernel/luminous-24856

luminous: mds: handle discontinuous mdsmap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22885 from smithfarm/wip-24696-luminous

luminous: mds: low wrlock efficiency due to dirfrags traversal

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22895 from smithfarm/wip-22386-luminous

luminous: ceph-fuse mount failed because no mds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23013 from batrick/i24136

luminous: MDSMonitor: uncommitted state exposed to clients/mdss

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23015 from batrick/i23790

luminous: mds: unset deleted vars in shutdown_pass

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23212 from batrick/i25036

luminous: mds: print mdsmap processed at low debug level

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23213 from batrick/i25039

luminous: mds: dump recent events on respawn

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23214 from batrick/i25041

luminous: mds: increase debug level for dropped client cap msg

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22680 from VictorDenisov/backport_23772

luminous: let "ceph status" use base 10 when printing numbers not sizes

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #22686 from pdvian/wip-24471-luminous

luminous: spdk: fix ceph-osd crash when activate SPDK

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22774 from theanalyst/wip-luminous-24717

luminous: qa/workunits/suites/blogbench.sh: use correct dir name

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #22848 from smithfarm/wip-24739-luminous

luminous: Bring back diff -y for non-FreeBSD

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #22860 from smithfarm/wip-24770-luminous

luminous: bluestore: set correctly shard for existed Collection

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #22862 from smithfarm/wip-24772-luminous

luminous: osd: may get empty info at recovery

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #22864 from smithfarm/wip-24774-luminous

luminous: Mimic build fails with -DWITH_RADOSGW=0

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge pull request #22904 from ifed01/wip-ifed-backport-24260

luminous: os/bluestore: fix flush_commit locking

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22909 from ifed01/wip-ifed-fix-23540-luminous

luminous: os/bluestore: fix incomplete faulty range marking when doing compression

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23025 from smithfarm/wip-24864-luminous

luminous: common: Abort in OSDMap::decode() during qa/standalone/erasure-code/test-erasure-eio.sh

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23112 from smithfarm/wip-24979-luminous

luminous: tests: ceph-helpers.sh tries to use dirname without mandatory parameter

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #23374 from ceph/backport-luminous-22764

luminous: ceph-volume: refuse to zap mapper devices

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23395 from alfredodeza/backport-luminous-rm24020

luminous: ceph-volume expand on the LVM API to create multiple LVs at different sizes

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mds: update MDSRank::cluster_degraded before handling mds failure

Migrator checks MDSRank::is_cluster_degraded() to decide if it needs to
send message to other mds. MDSRank::cluster_degraded should get updated
before calling Migrator::handle_mds_failure_or_stop().

Introduced by commit 7de9da4a "mds: handle discontinuous mdsmap"

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 93458c7306c3822175ed3c048f0937af8f02c341)

Merge pull request #23394 from alfredodeza/backport-luminous-rm24152

luminous: ceph-volume enable the ceph-osd during lvm activation

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23397 from alfredodeza/backport-luminous-rm25216

luminous: ceph-volume lvm.activate conditional mon-config on prime-osd-dir

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

client: increase verbosity level for log messages in helper methods

Fixes: http://tracker.ceph.com/issues/21014
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit e276fdaaa8b737ea845e19388796796ca30d2162)

Conflicts:
src/client/Client.cc

ceph-volume tests patch __release__ to mimic always for stdin keys

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 99c17ecbaf443824ea8e155e06c11b20a8603d76)

ceph-volume lvm.activate conditional mon-config on prime-osd-dir

The --no-mon-config should only be done in Mimic releaes and later

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 020d6b1e5555e4547e8a0a19bd29eea89e1b7a31)

ceph-volume api.lvm fallback to 1 part if none are define when creating lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 552d987233545f6f33b9220ea4ea98753f96962f)

ceph-volume tests verify create_lvs behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8c487124d795778dbbf461ffb95841b1c8025e08)

ceph-volume api.lvm default to null tags on lvs created

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 490e73eff1b5b7cc98db5d3df292759e1773c0ab)

ceph-deploy api.lvm allow multiple lv creation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e073c56d80d8f37e0f96093f8dd4a95aef81ea02)