ceph-volume tests.functional install new ceph-ansible dependencies

Make note that ceph-ansible's requirements.txt can't be used just yet

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 22310f43165e474e8e12732be57217b26e2b5424)

Merge pull request #22529 from ceph/backport-mimic-22062

mimic: ceph-volume dmcrypt and activate --all documentation updates

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume explain encryption and activate all with examples

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0a4cebd1dda876cfc19fbfb97c668d4dae05097c)

doc/ceph-volume activate is able to use --all for existing OSDs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d8b9a7c0745d6a901988a08a8218d9da585e5aa8)

doc/ceph-volume encryption is enabled with the --dmcrypt flag

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 72cef7acf7a9a965960ec3c7c1b99ce8e7fa3922)

doc/ceph-volume prepapre accepts a --dmcrypt flag

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e97d0717d323a08a46aa997c63d2416da7e4941f)

doc/ceph-volume activate does not need a dmcrypt flag

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ac0c925c576868107e585ba458fbff956467bccb)

Merge pull request #22508 from ceph/backport-mimic-22426

mimic: ceph-volume expand on the LVM API to create multiple LVs at different sizes

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22514 from ceph/backport-mimic-21890

mimic: ceph-volume describe better the options for migrating away from ceph-disk

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc/rados add an anchor to reference replacing osds

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0cc7bdc67ae3f2b1aeab8cc6046b6d172915e548)

doc/ceph-volume describe the options for migrating away from ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 537ff9cc3b64c59f9dbb81ac3cf180460ecaa3f3)

doc/ceph-volume high-level description of simple activate at index.rst

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dd6f368158db7f102b485179166ebe73d6ea622f)

ceph-volume api.lvm fallback to 1 part if none are define when creating lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 552d987233545f6f33b9220ea4ea98753f96962f)

ceph-volume tests verify create_lvs behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8c487124d795778dbbf461ffb95841b1c8025e08)

ceph-volume api.lvm default to null tags on lvs created

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 490e73eff1b5b7cc98db5d3df292759e1773c0ab)

ceph-deploy api.lvm allow multiple lv creation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e073c56d80d8f37e0f96093f8dd4a95aef81ea02)

ceph-volume api.lvm add a sizing method for VGs to calculate LV creation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e61b9e5c771e167990caad3d85c0bfd27682d643)

ceph-volume exceptions create an error for size allocations

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b035bb0f54d4a7777ce242798bdeb18ad0dc28e0)

ceph-volume tests verify vg sizing behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 08832f21e52f28d65a16aadae54a10c8fd16259f)

ceph-volume api.lvm force vgs to display units in 'g'

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit da9f90fcb40e1e82c322f6020253b63da69238da)

Merge pull request #22425 from alfredodeza/mimic-rm24417

mimic: ceph-volume tests do not include admin keyring in OSD nodes

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #22449 from wido/mimic-dashboard-port

mimic: mgr/dashboard: Listen on port 8443 by default and not 8080

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Listen on port 8443 by default and not 8080

Port 8080 is a common alternative HTTP port used for web traffic.

The dashboard however uses SSL (which can not be turned off) and for
that purpose we should use 8443.

Signed-off-by: Wido den Hollander <wido@42on.com>
(cherry picked from commit b7cc66e809e9f15b211adef5bac369c32a4bacaa)
Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #22398 from alfredodeza/mimic-rm24416

mimic ceph-volume lvm.activate Do not search for a MON configuration

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #22374 from dzafman/wip-scrub-omap-mimic

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume tests do not include admin keyring in OSD nodes

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 08760632743040520588e1bc03adc17746ff2eca)

ceph-volume lvm.activate Do not search for a MON configuration

This system might not have one nor can we be sure that we have
a client.admin keyring on the system.

Just prime the directory and have the OSD then use it's own cephx
key to communicate with the MONs.

Signed-off-by: Wido den Hollander <wido@42on.com>
(cherry picked from commit 722d79d16a0da042bbc189709efd3eeff1366558)

Merge pull request #22372 from tchaikov/mimic-schedule-links

mimic:  doc: fix the links in releases/schedule.rst

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22274 from votdev/fix_table_pagination_style_mimic

mimic: mgr/dashboard: Fix some datatable CSS issues

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Revert "qa/standalone/scrub/osd-scrub-repair.sh: drop omap_digest flag"

This reverts commit 886606bfd76b3f327b9376c4cc3875ae4991365a.

Signed-off-by: David Zafman <dzafman@redhat.com>
Conflicts:
        qa/standalone/scrub/osd-scrub-repair.sh (manually made equivalent changes)

(cherry picked from commit 843598b69ba0c6dd3532b4cf3e75475189a3f45d)

osd: Handle omap and data digests independently

Caused by: be078c8b7b131764caa28bc44452b8c5c2339623
The original attempt above to fix the omap_digest handling when
data_digest isn't present had 2 errors.  First, it checked
is_data_digest() and is_omap_digest() instead of digest_present and
omap_digest_present which indicate the source digest is available.
Second, MAYBE could only be set if both digests are available.

Fixes: http://tracker.ceph.com/issues/24366
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 01f9669928abd571e14421a51a749d44fa041337)

Merge pull request #22299 from tchaikov/mimic-pr-22286

mimic: cmake: enable RTTI for both debug and release RocksDB builds

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

doc: fix the links in releases/schedule.rst

they were broken in e37e49b2

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 27aac7c36cbf9d28ceb81d037de8d9a13ca46564)

doc: fix some redirected links reported by "build-doc linkcheck"

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 776329abe17987bcc021dac2b444fc7ab24c1b28)

cleanup: Remove debug option osd_debug_scrub_chance_rewrite_digest

This option seems pointless and there are no test cases that use it.

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 6adeaed32f70923d012bf9410bfa8651694be3cf)

13.2.0

Merge pull request #22309 from ukernel/mimic-24340

mimic: mds: fix some memory leak

Merge pull request #22322 from ukernel/mimic-24345

mimic: mds: properly journal root inode's snaprealm

Merge pull request #22326 from jan--f/cmake-fix-ceph-test-build-mimic

mimic: cmake: fix cython target in test/CMakeFile.txt

Reviewed-By: Kefu Chai <kchai@redhat.com>

cmake: fix cython target in test/CMakeFile.txt

The cython target is called cython_modules in python2 environments and
cython3_modules in python3 environments. Reflect that naming in
src/test/CMakeFile.txt. Otherwise the test target can not build in
python3 environments.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 743b9108c0f217ef6eeaadf645a538991e7d5eff)

Conflicts:
  src/test/CMakeLists.txt - Resovled by changing line 548 to
  "cython${PY_BINDING_INFIX}_modules)" and picking no other changes.

Merge pull request #22314 from jan--f/mimic-move-crypto_plugins

mimic: cmake: move crypto_plugins target

Reviewed-By: Kefu Chai <kchai@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

mds: properly journal root inode's snaprealm

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/24343
(cherry picked from commit 4f5029bdcc598731603918f4a8378b6e00375ca6)

cmake: move crypto_plugins target

the crypto_plugins target was defined in
src/crypto/isa-l/CMakeLists.txt, but this is only included
if(HAVE_INTEL AND HAVE_BETTER_YASM_ELF64 AND (NOT APPLE))

moving it out of the if() block allows the os target to depend on it
even if no plugins are built

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 61516bcfac4dff53d0e24f6c4623127465c46d9d)

mds: fix leak of MDSCacheObject::waiting

Fixes: http://tracker.ceph.com/issues/24289
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 8f3c8bf6eafd3545c3c786b8520e8ff2c40af2a0)

mds: fix some memory leak

Fixes: http://tracker.ceph.com/issues/24289
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit e7c149b93dc384ee4a2c8250c502548d12535123)

cmake: enable RTTI for both debug and release RocksDB builds

Overwise ceph build in Release mode is failing.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 83841bf3de4f664df1c5b1b7997861acf2ede767)

Merge pull request #22288 from tchaikov/mimic-24321

mimic: qa: wait longer for osd to flush pg stats

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22262 from a2batic/link-mimic

mimic: mgr/dashboard: Fixes documentation link- to open in new tab

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22264 from liewegas/wip-mimic

ceph_release: s/rc/stable/

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #22277 from tspmelo/fix-lint-error

mgr/dashboard: fix linting problem

Reviewed-by: Kanika Murarka <murarkakanika@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: wait longer for osd to flush pg stats

pg sends pg-stats to mgr every 5 seconds, so we cannot check for the
number of pgs right after creating the pool, at that moment, the number
of pgs could be 0, that's why manger.wait_for_clean() returns right
away, and leaves us with 0 pgs: the pgs serving the pool are still being
created. that's why `manager.get_num_active_clean()` returns `0`
sometimes. so, we should force osd to flush their stats to mgr, and wait
until the pg stats converages.

Fixes: http://tracker.ceph.com/issues/24321
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 380747268a1e4a3e8db618d66006c104091fafd5)

Merge pull request #22184 from tchaikov/mimic-23464

mimic: common/crc/aarch64: Added cpu feature pmull and make aarch64 specific…

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: fix linting problem

Problems were introduced in 50fbc04bf45b8cbf4db57c492652615625cd72c6,
when cherry picking some changes from master.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix some datatable CSS issues

Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 5e84bc5fee6a44b5a18d1f4f2e917af0deeac12c)

mgr/dashboard: Fixes documentation link- to open in new tab
Adds 'target' attribute to open link in new tab.
Fixes : https://tracker.ceph.com/issues/24288

Signed-off-by: Kanika Murarka <murarkakanika@gmail.com>
(cherry picked from commit 3677a11fff46bc63cc996e3cb0025b373f74b45f )

ceph_release: s/rc/stable/

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22260 from liewegas/wip-rbd-cli-tests-mimic

mimic: test: fix CLI unit formatting tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test/cli-integration/rbd: fixed unit formatting

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4aea23cf1b488c0fe57b3cbe910ddcc4cb5360eb)

qa/workunits/rbd: fix unit formatting tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d5b7d672b949767dc65359f26f5af3cad887e444)

librbd: clones should not inherit implicit feature bits from parent

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit bb357062853cc225fd7489eba9a70aa7ec7226e3)

Merge PR #22225 into mimic

* refs/pull/22225/head:
msg/async/Event: clear time_events on shutdown

Merge PR #22226 into mimic

* refs/pull/22226/head:
tests/crypto: print compile warning when NSS is unavailable.
tests/crypto: add tests for the no-bl encrypt/decrypt, part 2.
tests/crypto: add tests for the no-bl encrypt/decrypt.
auth: use OpenSSL for CryptoAESKeyHandler's no-bl encrypt/decrypt.
auth: extend CryptoKey with no-bl encrypt/decrypt.
auth: CryptoAESKeyHandler switches from NSS to OpenSSL.
auth: the outbuf of AES should be multiple of block size
auth: cache the PK11Context for CryptoAESKeyHandler

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #22227 into mimic

* refs/pull/22227/head:
mgr/telemetry: Check if boolean is False or not present

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22242 into mimic

* refs/pull/22242/head:
doc: rgw: fix tagging support status

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #22245 into mimic

* refs/pull/22245/head:
mgr/dashboard: Add unit tests for all frontend pipes

Merge PR #22247 into mimic

* refs/pull/22247/head:
qa: use $TESTDIR for testing mkfs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22254 into mimic

* refs/pull/22254/head:
osd/ReplicatedBackend: fix use-after-free on InProgressOp

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedBackend: fix use-after-free on InProgressOp

- op in flight to disk...
- on_change() clears the InProgressOp
- C_OSD_OnOpCommit calls op_commit() w/ bare pointer
- crash!

Fix by refcounting InProgressOp and clearing on_commit when it is
canceled.

Fixes: http://tracker.ceph.com/issues/24219
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e84c2d097440aea5980fba2a2ef065769dbf1271)

qa: use $TESTDIR for testing mkfs

as tmpfs does not support O_DIRECT, but bluefs use this flag for
accessing the underlying file.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 1692f49b7b51ca599bf2e180d7a69778a5a5ba68)

Merge PR #22235 into wip-sage-testing-20180525.132253

* refs/pull/22235/head:
mgr/iostat: enable module by default on new and vstart clusters
doc: add documentation for iostat

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

common/arch/arm: Improve CPU feature check.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit d998580c7dbeff5ced5e9eb1e11d160a2a8d7852)

common/crc/aarch64: Added cpu feature pmull and make aarch64 specific crc function to depend on it.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 90b1c1f1c4a57a59f598cd93dc9ce8a1532db238)

Merge pull request #22169 from tchaikov/mimic-23386

mimic: crush: fix device_class_clone for unpopulated/empty weight-sets

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22199 from dzafman/wip-minor-mimic

mimic: test: wait_for_pg_stats() should do another check after last 13 secon…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22170 from tchaikov/mimic-24211

mimic: os/bluestore: simplify and fix SharedBlob::put()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Add unit tests for all frontend pipes

Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit b6305630c417bcb92c55a6b461c63ca0911a8b03)

doc: rgw: fix tagging support status

As of mimic, policy and LC conditionals for tagging are merged, so let's drop
the unsupported status.

Fixes: http://tracker.ceph.com/issues/24164
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 17dab9b3f24c1454a12f42184fd262119eb97e3e)

mgr/iostat: enable module by default on new and vstart clusters

Signed-off-by: Mohamad Gebai <mgebai@suse.com>
(cherry picked from commit a776755b704065bb0b3dc648b3c5e1efd887e003)

doc: add documentation for iostat

Conflicts:
        doc/releases/mimic.rst: remove file
        doc/mgr/index.rst: trivial resolution

Signed-off-by: Mohamad Gebai <mgebai@suse.com>
(cherry picked from commit 1e2bf1ec8c0dbe18b816ff9eb3243b73bb40d237)

Merge PR #22136 into mimic

* refs/pull/22136/head:
client: avoid freeing inode when it contains TX buffer heads

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22137 into mimic

* refs/pull/22137/head:
mds: properly setup client_need_snapflush for snap inode
Revert "mds: properly setup need_snapflush for snapped inode"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22138 into mimic

* refs/pull/22138/head:
mds: reply session reject for open request from blacklisted client
qa/tasks/cephfs: add timeout parameter to kclient umount_wait

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22139 into mimic

* refs/pull/22139/head:
client: delay dentry trimming until after cap traversal

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22140 into mimic

* refs/pull/22140/head:
client: fix issue of revoking non-auth caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22141 into mimic

* refs/pull/22141/head:
mds: broadcast quota to relevant clients when quota is explicitly set

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22153 into mimic

* refs/pull/22153/head:
mon/MonClient: set configs via finisher

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22154 into mimic

* refs/pull/22154/head:
client: increase verbosity level for log messages in helper methods

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/telemetry: Check if boolean is False or not present

Now we disable the module if the boolean is True

Signed-off-by: Wido den Hollander <wido@42on.com>
(cherry picked from commit 20f5696a735094cbaa0a698f41094cd5a5ac96f4)

tests/crypto: print compile warning when NSS is unavailable.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit ff56a6b7e1e502170779802fb121b4f8f0cdc1b6)

tests/crypto: add tests for the no-bl encrypt/decrypt, part 2.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 2535d11713aa837015e5028923ac97a271f41081)

tests/crypto: add tests for the no-bl encrypt/decrypt.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit e70ef0fb483c34e4944a996eb84ea09f7d1b30ca)

auth: use OpenSSL for CryptoAESKeyHandler's no-bl encrypt/decrypt.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 24e0532f81671fcea31a1fd2fb17e8b41fe5e9d9)

auth: extend CryptoKey with no-bl encrypt/decrypt.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit b5acef206c94142464f7ff314bb11c843c5420e8)

auth: CryptoAESKeyHandler switches from NSS to OpenSSL.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 4860bb70e1f47377ff69e1dc44e9b11bc69a7c2a)

auth: the outbuf of AES should be multiple of block size

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 3cfcd02af6381022081d51367f7338bde1fcb1d2)
(cherry picked from commit 7635485d34c99671030a10e1da67945c19b4fc88)

auth: cache the PK11Context for CryptoAESKeyHandler

in the flame graph, 0.50% of total time is used by
CephxSessionHandler::check_message_signature(), in which 0.27% is used
by PK11_CreateContextBySymKey(). so we should cache the PK11Context.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit f8c2036f7663be54030e13f1f9c1511cb40070f6)
(cherry picked from commit 7f6751bb5522c00b78d33af402470a659a07d4a3)

msg/async/Event: clear time_events on shutdown

Works around crash from http://tracker.ceph.com/issues/24162

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 923bd4649aacfb02a783fab9a0a71d1395d6e474)

Merge pull request #22135 from tchaikov/mimic-24175

mimic: mgr/status: output to stdout, not stderr

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22160 from liewegas/wip-24037-mimic

common: OpTracker doesn't visit TrackedOp when nref == 0.

Merge pull request #22183 from tchaikov/mimic-24025

mimic: cmake/modules/BuildRocksDB.cmake: enable compressions for rocksdb

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>