Merge pull request #22515 from ceph/backport-mimic-21803

mimic: ceph-volume initial take on auto sub-command

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22477 from alfredodeza/mimic-bluestore-no-meta-paths

mimic: os/bluestore: don't store/use path_block.{db,wal} from meta

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests verify get_devices captures device metadata

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8deb088f101894763d27a6c31a3e8a2e96fc8320)

ceph-volume tests.conftest allow custom dirs in tmpfile fixture

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9d820e82ac6bc0f46890df1cdaa104e88cab2a7e)

ceph-volume tests ensure human_readable_size works up to terabytes

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4ad42f4244062995de003d8141cddcdc1d2b80fc)

ceph-volume tests verify mapper and /dev devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9cfd37344c69ebf1f3d8440f295d79ae05b545e9)

ceph-volume tests add tests for get_block_devs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 672c4c4d1c31b4631b1238ed0efb98fef34f2a3d)

ceph-volume tests verify _map_dev_paths mappings

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2c4d46955e8d7f52b090cd8b5b5561a2dba443e5)

ceph-volume devices.auto initial take on auto sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e3b04c9677c273f91e0a9de35dcd68c578d124db)

ceph-volume util.disk create a device metadata utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c6559538db28d6b8bc8529e6b2a62e8619c69bd6)

ceph-volume util.disk add dev mapping helpers for path resolution

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c06e63fed8a11e80444e8e7ec2e82373d75aa501)

ceph-volume tests verify get_file_contents utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0177f71caba3fd64de285d6808b5f859c55ed2fe)

ceph-volume util.system capture file contents

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 39792573dabac3b85a41760c965b60746ca1f059)

ceph-volume devices.auto initial take on auto sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5649bd724b5d300c7080052433f9b9d1c6774cff)

ceph-volume main include commented out auto subcommand

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ad1b8cae055ee6c822370d91027613680ccd3dd3)

ceph-volume tests ensure is_lv detection on LVs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 47060a1ef4562b29771bdfb5a375a225f18c228c)

ceph-volume api.lvm create helpers to detect an lv from a device path

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 68ca7a691e6e333209b67eb5cea8594f37fb93c1)

ceph-volume util make chown links affect links as well as paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7f79932f2c522da222a27c15182434914ca9a738)

ceph-volume lvm.activate chown bluestore symlinks

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2a0911d04cc1eeb19ea2e1d6069095f9e2af26fe)

os/bluestore: make bad symlink target error messages more informative

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit aea215103caf669c3e289cc07f978511d61f68b0)

os/bluestore: don't store/use path_block.{db,wal} from meta

This reverts most of 8d07fa5ad0772b4d979dc10695d230074d1f1e69.

ceph-volume goes to great lengths to ensure that the symlinks in the
osd dir are accurate.  Having these values here is an opportunity to
get them out of sync.  And that can happen very easily if the initial
mkfs was performed using a /dev/sdX device name (which is unstable
across reboots).  Even after ceph-volume corrects the symlink, bluestore
will continue to use the stale device path.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ef510e43f0ee14b49e99beed9ae8feda6db3429a)

Merge pull request #22507 from ceph/backport-mimic-22437

mimic: ceph-volume tests destroy osds on monitor hosts

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22535 from alfredodeza/mimic-cv-ansible-deps

mimic: ceph-volume tests.functional install new ceph-ansible dependencies

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.functional install new ceph-ansible dependencies

Make note that ceph-ansible's requirements.txt can't be used just yet

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 22310f43165e474e8e12732be57217b26e2b5424)

Merge pull request #22529 from ceph/backport-mimic-22062

mimic: ceph-volume dmcrypt and activate --all documentation updates

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume explain encryption and activate all with examples

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0a4cebd1dda876cfc19fbfb97c668d4dae05097c)

doc/ceph-volume activate is able to use --all for existing OSDs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d8b9a7c0745d6a901988a08a8218d9da585e5aa8)

doc/ceph-volume encryption is enabled with the --dmcrypt flag

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 72cef7acf7a9a965960ec3c7c1b99ce8e7fa3922)

doc/ceph-volume prepapre accepts a --dmcrypt flag

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e97d0717d323a08a46aa997c63d2416da7e4941f)

doc/ceph-volume activate does not need a dmcrypt flag

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ac0c925c576868107e585ba458fbff956467bccb)

Merge pull request #22508 from ceph/backport-mimic-22426

mimic: ceph-volume expand on the LVM API to create multiple LVs at different sizes

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22514 from ceph/backport-mimic-21890

mimic: ceph-volume describe better the options for migrating away from ceph-disk

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc/rados add an anchor to reference replacing osds

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0cc7bdc67ae3f2b1aeab8cc6046b6d172915e548)

doc/ceph-volume describe the options for migrating away from ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 537ff9cc3b64c59f9dbb81ac3cf180460ecaa3f3)

doc/ceph-volume high-level description of simple activate at index.rst

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dd6f368158db7f102b485179166ebe73d6ea622f)

ceph-volume api.lvm fallback to 1 part if none are define when creating lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 552d987233545f6f33b9220ea4ea98753f96962f)

ceph-volume tests verify create_lvs behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8c487124d795778dbbf461ffb95841b1c8025e08)

ceph-volume api.lvm default to null tags on lvs created

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 490e73eff1b5b7cc98db5d3df292759e1773c0ab)

ceph-deploy api.lvm allow multiple lv creation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e073c56d80d8f37e0f96093f8dd4a95aef81ea02)

ceph-volume api.lvm add a sizing method for VGs to calculate LV creation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e61b9e5c771e167990caad3d85c0bfd27682d643)

ceph-volume exceptions create an error for size allocations

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b035bb0f54d4a7777ce242798bdeb18ad0dc28e0)

ceph-volume tests verify vg sizing behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 08832f21e52f28d65a16aadae54a10c8fd16259f)

ceph-volume api.lvm force vgs to display units in 'g'

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit da9f90fcb40e1e82c322f6020253b63da69238da)

ceph-volume tests functional osd destroy on mons only for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 953c7ef54925059e6de75c2e11d9d40d909bbbca)

ceph-volume tests/functional destroy filestore osds on the monitors

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 41277c65b4c745523e3a83bba00eea3482df4100)

ceph-volume tests/functional destroy bluestore osds on the monitors

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b665e1caa70a8a4b5a8c972b769bf33d5d6edde4)

Merge pull request #22425 from alfredodeza/mimic-rm24417

mimic: ceph-volume tests do not include admin keyring in OSD nodes

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #22449 from wido/mimic-dashboard-port

mimic: mgr/dashboard: Listen on port 8443 by default and not 8080

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Listen on port 8443 by default and not 8080

Port 8080 is a common alternative HTTP port used for web traffic.

The dashboard however uses SSL (which can not be turned off) and for
that purpose we should use 8443.

Signed-off-by: Wido den Hollander <wido@42on.com>
(cherry picked from commit b7cc66e809e9f15b211adef5bac369c32a4bacaa)
Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #22398 from alfredodeza/mimic-rm24416

mimic ceph-volume lvm.activate Do not search for a MON configuration

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #22374 from dzafman/wip-scrub-omap-mimic

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume tests do not include admin keyring in OSD nodes

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 08760632743040520588e1bc03adc17746ff2eca)

ceph-volume lvm.activate Do not search for a MON configuration

This system might not have one nor can we be sure that we have
a client.admin keyring on the system.

Just prime the directory and have the OSD then use it's own cephx
key to communicate with the MONs.

Signed-off-by: Wido den Hollander <wido@42on.com>
(cherry picked from commit 722d79d16a0da042bbc189709efd3eeff1366558)

Merge pull request #22372 from tchaikov/mimic-schedule-links

mimic:  doc: fix the links in releases/schedule.rst

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22274 from votdev/fix_table_pagination_style_mimic

mimic: mgr/dashboard: Fix some datatable CSS issues

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Revert "qa/standalone/scrub/osd-scrub-repair.sh: drop omap_digest flag"

This reverts commit 886606bfd76b3f327b9376c4cc3875ae4991365a.

Signed-off-by: David Zafman <dzafman@redhat.com>
Conflicts:
        qa/standalone/scrub/osd-scrub-repair.sh (manually made equivalent changes)

(cherry picked from commit 843598b69ba0c6dd3532b4cf3e75475189a3f45d)

osd: Handle omap and data digests independently

Caused by: be078c8b7b131764caa28bc44452b8c5c2339623
The original attempt above to fix the omap_digest handling when
data_digest isn't present had 2 errors.  First, it checked
is_data_digest() and is_omap_digest() instead of digest_present and
omap_digest_present which indicate the source digest is available.
Second, MAYBE could only be set if both digests are available.

Fixes: http://tracker.ceph.com/issues/24366
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 01f9669928abd571e14421a51a749d44fa041337)

Merge pull request #22299 from tchaikov/mimic-pr-22286

mimic: cmake: enable RTTI for both debug and release RocksDB builds

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

doc: fix the links in releases/schedule.rst

they were broken in e37e49b2

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 27aac7c36cbf9d28ceb81d037de8d9a13ca46564)

doc: fix some redirected links reported by "build-doc linkcheck"

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 776329abe17987bcc021dac2b444fc7ab24c1b28)

cleanup: Remove debug option osd_debug_scrub_chance_rewrite_digest

This option seems pointless and there are no test cases that use it.

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 6adeaed32f70923d012bf9410bfa8651694be3cf)

13.2.0

Merge pull request #22309 from ukernel/mimic-24340

mimic: mds: fix some memory leak

Merge pull request #22322 from ukernel/mimic-24345

mimic: mds: properly journal root inode's snaprealm

Merge pull request #22326 from jan--f/cmake-fix-ceph-test-build-mimic

mimic: cmake: fix cython target in test/CMakeFile.txt

Reviewed-By: Kefu Chai <kchai@redhat.com>

cmake: fix cython target in test/CMakeFile.txt

The cython target is called cython_modules in python2 environments and
cython3_modules in python3 environments. Reflect that naming in
src/test/CMakeFile.txt. Otherwise the test target can not build in
python3 environments.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 743b9108c0f217ef6eeaadf645a538991e7d5eff)

Conflicts:
  src/test/CMakeLists.txt - Resovled by changing line 548 to
  "cython${PY_BINDING_INFIX}_modules)" and picking no other changes.

Merge pull request #22314 from jan--f/mimic-move-crypto_plugins

mimic: cmake: move crypto_plugins target

Reviewed-By: Kefu Chai <kchai@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

mds: properly journal root inode's snaprealm

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/24343
(cherry picked from commit 4f5029bdcc598731603918f4a8378b6e00375ca6)

cmake: move crypto_plugins target

the crypto_plugins target was defined in
src/crypto/isa-l/CMakeLists.txt, but this is only included
if(HAVE_INTEL AND HAVE_BETTER_YASM_ELF64 AND (NOT APPLE))

moving it out of the if() block allows the os target to depend on it
even if no plugins are built

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 61516bcfac4dff53d0e24f6c4623127465c46d9d)

mds: fix leak of MDSCacheObject::waiting

Fixes: http://tracker.ceph.com/issues/24289
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 8f3c8bf6eafd3545c3c786b8520e8ff2c40af2a0)

mds: fix some memory leak

Fixes: http://tracker.ceph.com/issues/24289
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit e7c149b93dc384ee4a2c8250c502548d12535123)

cmake: enable RTTI for both debug and release RocksDB builds

Overwise ceph build in Release mode is failing.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 83841bf3de4f664df1c5b1b7997861acf2ede767)

Merge pull request #22288 from tchaikov/mimic-24321

mimic: qa: wait longer for osd to flush pg stats

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22262 from a2batic/link-mimic

mimic: mgr/dashboard: Fixes documentation link- to open in new tab

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22264 from liewegas/wip-mimic

ceph_release: s/rc/stable/

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #22277 from tspmelo/fix-lint-error

mgr/dashboard: fix linting problem

Reviewed-by: Kanika Murarka <murarkakanika@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: wait longer for osd to flush pg stats

pg sends pg-stats to mgr every 5 seconds, so we cannot check for the
number of pgs right after creating the pool, at that moment, the number
of pgs could be 0, that's why manger.wait_for_clean() returns right
away, and leaves us with 0 pgs: the pgs serving the pool are still being
created. that's why `manager.get_num_active_clean()` returns `0`
sometimes. so, we should force osd to flush their stats to mgr, and wait
until the pg stats converages.

Fixes: http://tracker.ceph.com/issues/24321
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 380747268a1e4a3e8db618d66006c104091fafd5)

Merge pull request #22184 from tchaikov/mimic-23464

mimic: common/crc/aarch64: Added cpu feature pmull and make aarch64 specific…

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: fix linting problem

Problems were introduced in 50fbc04bf45b8cbf4db57c492652615625cd72c6,
when cherry picking some changes from master.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix some datatable CSS issues

Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 5e84bc5fee6a44b5a18d1f4f2e917af0deeac12c)

mgr/dashboard: Fixes documentation link- to open in new tab
Adds 'target' attribute to open link in new tab.
Fixes : https://tracker.ceph.com/issues/24288

Signed-off-by: Kanika Murarka <murarkakanika@gmail.com>
(cherry picked from commit 3677a11fff46bc63cc996e3cb0025b373f74b45f )

ceph_release: s/rc/stable/

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22260 from liewegas/wip-rbd-cli-tests-mimic

mimic: test: fix CLI unit formatting tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test/cli-integration/rbd: fixed unit formatting

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4aea23cf1b488c0fe57b3cbe910ddcc4cb5360eb)

qa/workunits/rbd: fix unit formatting tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d5b7d672b949767dc65359f26f5af3cad887e444)

librbd: clones should not inherit implicit feature bits from parent

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit bb357062853cc225fd7489eba9a70aa7ec7226e3)

Merge PR #22225 into mimic

* refs/pull/22225/head:
msg/async/Event: clear time_events on shutdown

Merge PR #22226 into mimic

* refs/pull/22226/head:
tests/crypto: print compile warning when NSS is unavailable.
tests/crypto: add tests for the no-bl encrypt/decrypt, part 2.
tests/crypto: add tests for the no-bl encrypt/decrypt.
auth: use OpenSSL for CryptoAESKeyHandler's no-bl encrypt/decrypt.
auth: extend CryptoKey with no-bl encrypt/decrypt.
auth: CryptoAESKeyHandler switches from NSS to OpenSSL.
auth: the outbuf of AES should be multiple of block size
auth: cache the PK11Context for CryptoAESKeyHandler

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #22227 into mimic

* refs/pull/22227/head:
mgr/telemetry: Check if boolean is False or not present

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22242 into mimic

* refs/pull/22242/head:
doc: rgw: fix tagging support status

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #22245 into mimic

* refs/pull/22245/head:
mgr/dashboard: Add unit tests for all frontend pipes

Merge PR #22247 into mimic

* refs/pull/22247/head:
qa: use $TESTDIR for testing mkfs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22254 into mimic

* refs/pull/22254/head:
osd/ReplicatedBackend: fix use-after-free on InProgressOp

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedBackend: fix use-after-free on InProgressOp

- op in flight to disk...
- on_change() clears the InProgressOp
- C_OSD_OnOpCommit calls op_commit() w/ bare pointer
- crash!

Fix by refcounting InProgressOp and clearing on_commit when it is
canceled.

Fixes: http://tracker.ceph.com/issues/24219
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e84c2d097440aea5980fba2a2ef065769dbf1271)

qa: use $TESTDIR for testing mkfs

as tmpfs does not support O_DIRECT, but bluefs use this flag for
accessing the underlying file.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 1692f49b7b51ca599bf2e180d7a69778a5a5ba68)

Merge PR #22235 into wip-sage-testing-20180525.132253

* refs/pull/22235/head:
mgr/iostat: enable module by default on new and vstart clusters
doc: add documentation for iostat

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

common/arch/arm: Improve CPU feature check.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit d998580c7dbeff5ced5e9eb1e11d160a2a8d7852)

common/crc/aarch64: Added cpu feature pmull and make aarch64 specific crc function to depend on it.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 90b1c1f1c4a57a59f598cd93dc9ce8a1532db238)

Merge pull request #22169 from tchaikov/mimic-23386

mimic: crush: fix device_class_clone for unpopulated/empty weight-sets

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22199 from dzafman/wip-minor-mimic

mimic: test: wait_for_pg_stats() should do another check after last 13 secon…

Reviewed-by: Kefu Chai <kchai@redhat.com>