rgw: parse old rgw_obj with namespace correctly

Fixes: http://tracker.ceph.com/issues/22982
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit c598e53c95ec09772353a12e9162464ccb76e0a8)

Merge pull request #20501 from dzafman/wip-23054

luminous: Snapset inconsistency is detected with its own error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20548 from dzafman/wip-23093

luminous: last-stat-seq returns 0 because osd stats are cleared

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20405 from liewegas/wip-osd-new-class-luminous

luminous: mon: accept crush device class to osd new

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20353 from adamemerson/wip-22603-22604-luminous

luminous: Backport of cache manipulation: issues #22603 and #20353

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20568 from ceph/backport-20513

luminous: ceph-volume warn on mix of filestore and bluestore flags

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20266 from linuxbox2/luminous-mploc-22655

luminous: rgw: return valid Location element, CompleteMultipartUpload

Reviewed-By: Radoslaw Zarzynski <rzarzynski@redhat.com>

Merge pull request #20236 from linuxbox2/luminous-rgw-22121

luminous: rgw: URL-decode S3 and Swift object-copy URLs

Reviewed-By: Radoslaw Zarzynski <rzarzynski@redhat.com>

Merge pull request #19104 from nickjanus/cors_not_found

luminous: rgw: Fix S3 compatibility bug when CORS is not found

Reviewed-By: Matt Benjamin <mbenjamin@redhat.com>

ceph-volume tests verify exclude_group_options works with groups of flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 11ba052010fe9a5c78c359bd05aa3042a69a575c)

ceph-volume tests add exclusion checks in prepare

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 43be2aafc187892a1ff61039dabea9efe0c9b1b3)

ceph-volume tests add exclusion checks in create

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2f6248c94be9b57766b820b23826714897153a51)

ceph-volume util.arg_validators create an arg group exclusion check

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dec6e086cf221e9b90f1b227aa63717d387f9f8e)

ceph-volume lvm.create use the new group option exclution

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f6ac244bb51742429c01f5d8726cbe86210f4266)

ceph-volume lvm.prepare use the new group option exclution

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e6b3c46368ea3e12fcebfef8fa60dd0824f3e1b4)

ceph-volume lvm.common remove --journal-size as it is not used/supported

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3400f31787c197c52037f26a69fa1e2c1a6c294f)

ceph-volume lvm.common define arg groups for objectstores

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3e7c995ebe252cb785da12548113ac855aa5219b)

Merge pull request #20559 from ceph/backport-wip-rm23080

luminous ceph-volume correctly normalize mount flags

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.util update tests for mount flags

Include parametrized flags for ensuring a combination of values will
still be normalized regardless on how they are on ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6671cb4fd1636505b1a10d81d3bd0db23bb2fbd0)

ceph-volume util.prepare normalize mount flags for CLI

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6bd226a3b4d37627bf9f039133de7792af3e9710)

Merge pull request #20545 from ceph/backport-20537

luminous: ceph-volume: adds support to zap encrypted devices

Reviewed-by: Alfredo Deza <adeza@redhat.com>

test: Fail upon flush_pg_stats timeout

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr/ClusterState: record osd_stat for out osds too

I'm not quite sure why we were doing this. :/

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a7612d33d1e4184af7faa6712fc50d6487a09c40)

rgw: Add cache introspection and manipulation

Provide the ability to examine and delete elements from the cache.

Fixes: http://tracker.ceph.com/issues/22603
Fixes: http://tracker.ceph.com/issues/22604
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 69fafcaaed6c5ee32f12ac857dbf4af7e13483f8)
Conflicts:
Resolved by removing a few C++17isms.
Make list helper public to deal with CentOS 7 GCC Bug

ceph-volume: use lv.encrypted property with lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 64659882d78ad4376573fc21ef5107dfb324bba2)

ceph-volume: adds an encrypted property to api.lvm.Volume

This property will return a boolean indicating if the lv
is encrypted or not.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 5f1c0219d048ca507329e8eb66f47cd1c8c31d4e)

ceph-volume: add zapping to lvm dmcrypt tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 1d5606885a8b1e42a4530b1a525f0273c09e5761)

ceph-volume: test zapping for both lvm create and lvm dmcrypt

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 70f68e2c19800d990c50991b9e159398549897dc)

ceph-volume: adds support to zap encrypted devices, lvs, and partitions

Fixes: http://tracker.ceph.com/issues/22878
Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 4e28bfafc72bba24bd06013c4084906f05eddb57)

osd: Add new snapset_inconsistency error check

Includes new test case

Caused by: 5f58301a1364e948834dabe503200dda07fc2790
This changed attr consistency checking to exclude system keys,
which required snapset to be handled just like object info.

Fixes: http://tracker.ceph.com/issues/22996
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 33e747724a0cb92e8200c41e16231973ff0aa2ad)

Conflicts:
src/osd/PGBackend.cc (trivial)

Correction:
qa/standalone/scrub/osd-scrub-repair.sh (Corrupted snapset looks different due to legacy support)

12.2.3

cmake: disable DOWNLOAD_NO_PROGRESS if cmake ver is lower than 3.1

see https://cmake.org/cmake/help/v3.1/module/ExternalProject.html and
https://cmake.org/cmake/help/v3.0/module/ExternalProject.html. the
former has DOWNLOAD_NO_PROGRESS, while the latter does not.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d051bf0441c933e27fb0f54bc6ed999d1e8eca89)

Merge pull request #20452 from ceph/backport-wip-rm22785

luminous ceph-volume consume mount/format options from ceph.conf

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests remove unused import

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ad49283b2ee1e4a1a8a54d0c71039a2afc778b29)

Merge pull request #20451 from ceph/wip-yuri-PR20053-luminous

qa/tests: Applied PR 20053 to stress-split tests

Reviewed-By: Jason Dillaman <jdillaman@redhat.com>

qa/tests: Applied PR 20053 to stress-split tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

ceph-volume util.prepare extend flags when found

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 287c952bc8c3422c6eed72a1e0b22612d0728314)

ceph-volume util.constants update mount flags to include rw, normalize on list

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 457a5173d2e08192273c22f1abe52a64773188b4)

ceph-volume configuration normalize underscore vs. whitespace named options

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b80e3bf1175a65181d1f2ed38b6a416c6b730d82)

ceph-volume tests check for mount/format usage from ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 676c93fcf3041de07e8125d36d8dc7ad474db9b1)

ceph-volume tests add ceph.conf stub for custom conf files

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a8eecc64716564479308fafba5159e271c42feac)

Merge pull request #20429 from ceph/backport-wip-rm22988

luminous ceph-volume use realpath when checking mounts

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests verify realpath usage in mounts

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 16eabbc3abb7d03a0808963a9aae6b2ffb3a875b)

ceph-volume util.system optionally use realpath on devices

To check a mounted device it is needed to verify on a combination of
realpath and plain devices against realpath and plain paths. In LVM, two
different paths might refer to the same devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 53e691e1d5122ff533e69ae08c73e18c1ed74765)

Merge pull request #20438 from ceph/backport-wip-rm22876

luminous: ceph-volume: lvm zap will unmount osd paths used by zapped devices

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: use terminal.MultiLogger in `lvm zap`

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 3f2e7074ee42ff8248afaed2aa9bfd919096fc38)

ceph-volume: use api helper methods for pvs and lvs in lvm zap

This is just a cleaner way of doing this instead of using PVolumes
and Volumes directly.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit c8a6da4e1bb645ca5ee8cb8f5d861584fe70210d)

ceph-volume: fix documentation typos for lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 5552cefc4ce6d6d819f43671cec83ca23b119665)

ceph-volume: when zapping unmount osd directories

If you zap an lv, device or partition and it's currently mounted as a
ceph osd directory then it will be unmounted so the zap can complete.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit d4639ac116407f71ba6bff06a3202c1490d57dee)

ceph-volume: adds a util.system.unmount function

This function will unmount the given path

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 6909baef542f3cd9b7d97113a6152280e72013f6)

Merge pull request #20407 from cbodley/wip-qa-multisite-trim-luminous

luminous: qa/rgw: disable log trim in multisite suite

Reviewed-by: Yuri Weinstein <yuri.weinstein@gmail.com>

qa/rgw: disable log trim in multisite suite

the multisite tests run manual trim operations with radosgw-admin, which
can race with internal log trimming to produce tests failures

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit f8909bb6a6320822da7a4c4dab242f84732ebf5d)

mon/OSDMonitor: accept crush_device_class in 'osd new' json input

If provided, set the OSD device_class at OSD creation time.  This is
simpler than writing a file that the OSD has to read in and use to
set its initial device class, and also avoids a bit of sticky state
at the OSD that will make it keep trying to reset its device class on
startup if it ever gets cleared.

Note that we now ignore json input fields we don't understand, so remove
a test case.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 59bdada5d52710bd683f2b6ef6a8edaef3c8564a)

mon/OSDMonitor: fix 'osd crush set-device-class' for new osds

We should set the max_devices value for a new device.  (Actually,
finalize() will fix this for us on decode, but better to be
pendantic.)

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6aa7c885e57070c52ccecec1ceca9a53e878d30c)

crush/CrushWrapper: fix finalize clobbering of max_devices

This hides device names when they don't appear in the hierarchy,
e.g. after setting a name and class for a new device that hasn't
yet been put in the tree.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 1e48ee9c592891fcaa5ae82fc53d45231156a62c)

Merge pull request #20368 from ceph/backport-20367

luminous: ceph-volume: adds custom cluster name support to simple

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20357 from pdvian/wip-22938-luminous

luminous: rgw: unlink deleted bucket from bucket's owner

Reviewed-By: Orit Wasserman <owasserm@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

ceph-volume: use a custom cluster name in simple functional tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 7f1dc6b3ab885253d28c223bda7e6e1232a84609)

ceph-volume: set conf.cluster name during simple activate

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit b1cee74fa2225fe10a024c7291bd23aadcc23441)

Merge pull request #20338 from cbodley/wip-22930

luminous: rgw: fix for empty query string in beast frontend
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: unlink deleted bucket from bucket's owner

if a bucket is deleted by an admin/system user instead of its
owner, the unlink would fail and the deleted bucket remained
visible to the original owner

Fixes: http://tracker.ceph.com/issues/22248
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4701e85a3ea72852a61a106724fa22497f3d5789)

Merge pull request #20204 from tchaikov/wip-pr-18614-luminous

luminous: osd,mgr: report pending creating pgs to mgr

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20249 from pdvian/wip-22864-luminous

luminous: mds: fix scrub crash

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20256 from pdvian/wip-22860-luminous

luminous: osdc: "FAILED assert(bh->last_write_tid > tid)" in powercycle-wip-yuri-master-1.19.18-distro-basic-smithi

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20299 from pdvian/wip-22859-luminous

luminous: mds: set higher priority for some perf counters

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20300 from pdvian/wip-22867-luminous

luminous: MDS : Avoid the assert failure when the inode for the cap_export from other…

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #20339 from batrick/i22242

luminous: mds: limit size of subtree migration

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #20340 from batrick/i22240

luminous: mds: prevent filelock from being stuck at XSYN state

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #20341 from batrick/i22089

luminous: mds: don't report repaired backtraces in damagetable, write back after repair, clean up scrub log

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #20337 from liewegas/wip-bluestore-aio-read

luminous: os/bluestore: propagate read EIO errors up the stack

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20350 from ceph/backport-20264

luminous: ceph-volume dmcrypt support for simple

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: do not test custom cluster names with simple

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 63046e0d7bd05a42477dd1fbd4af11f81046b08d)

ceph-volume: sleep 2 minutes after reboot in simple tests

This will allow for some time for OSD services to start back up after
reboot. If not we see race conditions where tests fail because they
were run before the OSD finished startup.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 85b319a2c577c1fa86c6ccbbac61f015dd24ad21)

ceph-volume: adds the simple dmcrypt_plain and dmcrypt_luks tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 3052010a80a1783b6ebd629a603638ab5806f8fd)

ceph-volume: fix ceph-volume simple scan help menu test

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 2ee70bf2d1958c0655ddb5b7cc4d2ed3f4bc58b8)

ceph-volume tests.functional add simple xenial filestore dmcrypt plain support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 00b14177bdbe6d66424001b1c35ca8fa7b46977f)

ceph-volume tests.functional add simple xenial filestore dmcrypt luks support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0b2f869ccbb30c27bd19f2a05bb893957912142e)

ceph-volume tests.functional add simple xenial bluestore dmcrypt plain support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0174f9e1646f4b51c712506f629b48c329f20ea1)

ceph-volume tests.functional add simple centos7 filestore dmcrypt plain support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4c26162002709ccdd1b48c05126befdab7f269da)

ceph-volume tests.functional add simple centos7 filestore dmcrypt luks support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a911d7c6a99e2ea235400614568d9cd362470cac)

ceph-volume tests.functional add simple centos7 bluestore dmcrypt plain support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8e0213ab053e80038bdae8d1b06d448fd397e3bc)

ceph-volume tests.functional add simple centos7 bluestore dmcrypt luks support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 76151c461bfe1e5ae4dc277c9d9357efb81e8c92)

ceph-volume tests.functional add simple bluestore dmcrypt luks support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d6e24022e8dfe845823fa0e87f3fba028c4b25b5)

doc/ceph-volume lvm prepare fully supports encryption now

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0b2075759e8788b2c4ae46f94fdee953cc82b21e)

doc/ceph-volume scan update for encryption support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e0d3bb82255a937cd937d33f966979fb541d1443)

doc/ceph-volume remove notice that dmcrypt is not supported

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a7d11ca27df1d22c8109cad5bf7be8fd23395c71)

doc/man/ceph-volume add simple documentation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9524021aa15280d98f2b04d1fd66552733adf8a2)

ceph-volume simple.scan update help menu to indicate device support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 09f35cb249674eda044b8fceea6f402ffac7731d)

ceph-volume tests for keyring parsing

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b90c1a86843d8f0c8c7fb68fe059bdd72f3730fd)

ceph-volume tests for validate_devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bece7af0523b04d39c26011990cb1fe323aecdd8)

ceph-volume simple.activate b64decode keys for activation as well

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 636ebc127797c8020cfb7aff09955b72beac8bdf)

ceph-volume tests add validation for lsblk parsers

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 03102e4417194e16deb8b738fcdaf87e699e6073)

ceph-volume tests validate parsing of cryptsetup

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cce6a12f4a0d8d9a3d7dcb509b42ed2536d9257f)

ceph-volume tests add a stub for process.call

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b80b61b90f8539b363c03c96e63f31c38c707fc8)

ceph-volume util.encryption add notes about extra b64decode call for ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c2367ef4a4db1e9c62e33dc211474f652d6bc6af)

ceph-volume util.encryption parse legacy encrypted with dirs too

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 32cb810f8ab7e4a710bfb2590a065dea41369fce)

ceph-volume terminal create a logger to get terminal+log messages in one call

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f6dd0ff15f0aeda9f660e923153f1d3dae68e583)

ceph-volume simple.activate support dmcrypted devices for both plain and luks

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5ece73b533ee4b577c0ce61e3685a880e1de4851)

ceph-volume simple.scan parse the keyring out of the keyring file

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a1124362816cbaaad7255b7ffc3698b68601ffe5)

Conflicts:
src/ceph-volume/ceph_volume/devices/simple/scan.py

ceph-volume simple.scan support dmcrypt OSDs when scanning

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6c87d66d76ee1fc70f8dc6211d89dbd489c6ed3a)