Merge pull request #24233 from liewegas/wip-35998-mimic

mimic: mgr/DaemonServer: fix Session leak

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24339 from ifed01/wip-ifed-fix-repairer-mimic

mimic: os/bluestore: fix bloom filter num entry miscalculation in repairer

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #24352 from ifed01/wip-ifed-bluestore-tool-fixes-mimic

mimic: ceph-bluestore-tool: fix set label functionality for specific keys

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24386 from pdvian/wip-36127-mimic

mimic: msg/async: clean up local buffers on dispatch

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #24420 from karunjosy/wip-36308-mimic

mimic: doc: Typo error on cephfs/fuse/

Reviewed-by: Nathan Cutler <ncutler@suse.com>

doc: Fix typo error on cephfs/fuse/

Fixes: http://tracker.ceph.com/issues/36180
Signed-off-by: Karun Josy <kjosy@redhat.com>
(cherry picked from commit d312cdbd36d9c0e76409feac1f3e289b247dc2dc)

Merge pull request #24002 from pdvian/wip-35705-mimic

mimic:rbd: fix error import when the input is a pipe

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24122 from pdvian/wip-35712-mimic

mimic: librbd: journaling unable request can not be sent to remote lock owner

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24137 from pdvian/wip-35714-mimic

mimic: librbd: ensure exclusive lock acquired when removing sync point snaps…

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24240 from smithfarm/wip-35077-mimic

mimic: qa: install dependencies for rbd_workunit_kernel_untar_build

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #24244 from dillaman/wip-36117-mimic

mimic: test/librbd: not valid to have different parents between image snapshots

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #24321 from smithfarm/wip-36120-mimic

mimic: [rbd-mirror] failed assertion when updating mirror status

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24356 from dillaman/wip-35957-mimic

mimic: librbd: fixed assert when flattening clone with zero overlap

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #24381 from alfredodeza/mimic-rm36247

mimic ceph-volume: skip processing devices that don't exist when scanning system disks

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

msg/async: clean up local buffers on dispatch

The AsyncConnection keeps local (member variable) bufferlists of incoming
messages before they're placed into the Message's front/data/middle buffers.
Previously these were reset only when a new Message is being received, which
means in steady state we store a full Message for every Connection even if
it's inactive!

Instead we obviously want to drop our local references to Message state
once it's been dispatched, so that it can go away.

Fixes: http://tracker.ceph.com/issues/35987
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 47ed036753223c44c7bf66c64d4a4adfe7267c0a)

ceph-volume util.disk when there are no devices mapped, skip to the next one

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9b0f472abadde26fce2a603fca5c466ebb770d4a)

ceph-volume tests.util verify devices that don't exist don't break get_devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0176c1965e6ed532b38dfee03af0e9e3a85149d6)

Merge pull request #24357 from alfredodeza/mimic-rm36249

mimic ceph-volume:  activate option --auto-detect-objectstore respects --no-systemd

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.activate pass no_systemd args when detecting objectstore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 29de6820a9a7108c8de41036add9332f9b910359)

ceph-volume tests.lvm verify activate works with --auto-detect-objectstore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 60dbeb6184797ccffeaeea2e85ef2d7fb4a1c54c)

librbd: fixed assert when flattening clone with zero overlap

Fixes: http://tracker.ceph.com/issues/35702
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8e9035443322f8cbdb8c7aff5da0d1c7599a2592)

Conflicts:
src/librbd/Operations.cc: trivial resolution
src/test/librbd/test_librbd.cc: trivial resolution

Merge pull request #24334 from alfredodeza/mimic-rm36098

mimic ceph-volume:  lsblk can fail to find PARTLABEL, must fallback to blkid

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

tools/ceph-bluestore-tool: update size label on bluefs expand

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 71c3b58da4e7ced3422bce2b1da0e3fa9331530b)

tools/ceph-bluestore-tool: add support for proper setting size, btime,
description, osd_uuid fields

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit ffac450da5d6e09cf14b8363b35f21819b48f38b)

tool/ceph-bluestore-tool: avoid mon/config access when calling global_init
This caused a permanent stuck on tool's startup.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit fa65e7bf65bea514b66f56053d5cbf2c16045f4c)

ceph-volume simple.scan better detection for ceph-disk data devices

Uses the new ``CephDiskDevice`` to look into PARTLABEL from both lsblk
(the default) falling back to blkid, which in some cases has the right
value when lsblk fails.

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c502e5f64aa4f0d832c14680e1731e58d8a3770b)

ceph-volume tests.util verify CephDiskDevice behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8f072eb3f940a6714089b37a6fbacd549420a454)

ceph-volume tests.conftest add blkid patching to device_info fixture

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5bfc8c26e639dba955f83f53893ce311e6e047f1)

ceph-volume util.device add a ceph-disk device object to detect types

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 200e6bfd934b34aa8175330aef32c51505ec3392)

ceph-volume tests.util verify new blkid API calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ead3c7e7d19635a741a9da6fc330ec12a68b3d77)

ceph-volume util.disk expand blkid's API calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dbde6a5b611fdd4bcd942df5dbab94ab5076fdff)

Merge pull request #24337 from alfredodeza/mimic-rm36251

mimic ceph-volume add new ceph-handlers role from ceph-ansible

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.functional add new ceph-handlers role from ceph-ansible

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 956dc5e65c6e35e27ab6524d4fb9fdd05d616cf6)

os/bluestore: fix bloom filter num entry miscalculation in the repairer

This could  cause an assertion due to an access to uninitialized bloom
filter. This happened  when detecting errors involving physical extents close to the end of a large partition.
E.g. it was observed in http://tracker.ceph.com/issues/25001 while trying to repair an OSD.

Fixes: https://tracker.ceph.com/issues/35971
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 50919813a15d9c29c182062f88885890c55eb76e)

Merge pull request #24306 from alfredodeza/mimic-batch-report-fix

mimic ceph-volume: fix journal and filestore data size in `lvm batch --report`

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

rbd-mirror: instantiate the status formatter before changing state

This will avoid a possible race between pre-queued status updates
firing between the time the state has been changed and the formatter
has been instantiated.

Fixes: http://tracker.ceph.com/issues/36084
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit bbdc545ec1602f928c89c85d61c975abdf67be5b)

ceph-volume: only run ceph-common once during tests

Using import_role we can have just one play and avoid rerunning
ceph-common over and over again in daemon specific plays.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit a81386123e66e2d61391371cec4485b3148ff5a7)

ceph-volume: fail fast during tests

This also rsyncs the ceph-volume code to the testing vms before
a ceph.conf is generated because ceph-volume is needed now for this
to figure out the number of osds when you're using 'lvm batch'.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit eed244473d11a153cd7a37e0e9bfaf804a20db89)

ceph-volume: when using batch --report return journal/data size as ints

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit a5e8bc97e70195cee54c4b66d10684d6cd3f5e38)

Merge pull request #24279 from alfredodeza/mimic-rm27210

mimic ceph-volume custom cluster names fail on filestore trigger

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24210 from pdvian/wip-35855-mimic

mimic : doc: remove deprecated 'scrubq' from ceph(8)

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24254 from yuriw/wip-yuriw-p2p-mimic

qa/tests: added v13.2.2 to the mix

Merge pull request #24258 from pdvian/wip-35925-mimic

mimic: mgr/dashboard: Progress bar does not stop in TableKeyValueComponent

Reviewed-by: Volker Theile <vtheile@suse.com>

ceph-volume tests/functional update dmcrypt test playbooks to use --cluster

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 96e29adf72804cb4fca62c04416b8fe2d23587bb)

ceph-volume tests/functional/batch update tests to use 'test' cluster name

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5f72d438da94323c0b04f261b1a60b13eaf03966)

ceph-volume tests/functional/lvm update tests to use 'test' cluster name

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7b0d2dc66968f81192f152fa166fbc2cb4dcb438)

ceph-volume tests/functional update playbooks to use --cluster

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ba10b6820fd7f352ed6ba78246a6976c54783cbd)

ceph-volume main move out the load_ceph_conf_path to the configuration module

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 532908099eb63bd805db8aecae1ef12301076410)

ceph-volume tests.devices patch configuration.load to avoid raising non-existing ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fe80e5399e9dd1f6b9f94404d732a76f961d9eb4)

ceph-volume tests reset the cluster name after every unit test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 877ee1a27e0651355812e27fbc6ac0a87ea56a68)

ceph-volume tests.functional add custom cluster name to centos7-filestore-create

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e61848dc97b638c6b8e60f0896d325f436142864)

ceph-volume lvm.activate load the config always by reading the cluster name from the LV

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a88cbfc73a258b51b20e060e269234962444ce63)

ceph-volume configuration be able to set conf.ceph

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4f8226f359d94a86a39492ed653d9a9e5d8dd8f9)

mgr/dashboard: Progress bar does not stop in TableKeyValueComponent

Fixes: https://tracker.ceph.com/issues/35907
Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 7243d9b67dc8c15283e30f889eb4d3fd00a5701a)

Conflicts:
src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-key-value/table-key-value.component.ts :
: Resolved in TableKeyValueComponent

qa/tests: added v13.2.2 to the mix

Fixes: http://tracker.ceph.com/issues/36169
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

13.2.2

test/librbd: not valid to have different parents between image snapshots

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3b8f4f136b52d591dc39e6c4f8bd59c1e63c158a)

Fixes: http://tracker.ceph.com/issues/36117
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa: rbd_workunit_kernel_untar_build: install build dependencies

Commit f0fe0936e64d ("qa: use recent kernel to kernel build testing")
bumped the kernel to 4.17.

Fixes: http://tracker.ceph.com/issues/35074
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit a4df8c3562602acd996940cd4d9ae5384a085302)

mgr/DaemonServer: fix Session leak

This bug does not exist in master.

Fixes: https://tracker.ceph.com/issues/35998
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24208 from alfredodeza/mimic-rm36088

mimic ceph-volume  batch: allow journal+block.db sizing on the CLI

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24205 from alfredodeza/mimic-rm24794

mimic ceph-volume: earlier detection for --journal and --filestore flag requirements

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc: remove deprecated 'scrubq' from ceph(8)

The option was removed in a30cbe9adea34782be8699c010d4281d245feb6b

Fixes: http://tracker.ceph.com/issues/35813
Signed-off-by: Ruben Kerkhof <ruben@rubenkerkhof.com>
(cherry picked from commit d05cd1a3e612f52eab9bde42c184022205f65384)

doc/man/ceph-volume update batch command with new sizing flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 042c2cbe475023cd3c34102b710b93c60d92077e)

ceph-volume tests.devices update fixture to include new size flags for strategies

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5df56a09c7b445c36d2b9d2874962c40b681a65c)

ceph-volume lvm.batch.filestore consume --journal-size

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2378c282c78f49f4d911316a7bc85e49a3f8e7f9)

ceph-volume lvm.batch.bluestore consume --block-db-size

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7e52bc559b21b18a06ff1dffd764dbe8aefe96a2)

ceph-volume lvm.batch add sizing flags for journal and block.db LVs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0a16dbd7a8f597cde6debbdcd968cdaf44f6ce47)

ceph-volume tests.lvm verify that it is reported when --journal is not used with --filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ae657d2fe012b0e78959ead9d8d3baa1095a31ff)

ceph-volume lvm.prepare report back when --journal is not used when --filestore is set

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2fc1ee48ac10bd8008ff5d54382db1d83e57ec1b)

Merge pull request #23669 from pdvian/wip-26920-mimic

mimic: mgr/dashboard: Escape regex pattern in DeletionModalComponent

Reviewed-by: Tiago Melo <tmelo@suse.com>

librbd: ensure exclusive lock acquired when removing sync point snapshots

Fixes: http://tracker.ceph.com/issues/24898
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 5bb823098ed0728cf9b88c654ba4528f3e42d695)

librbd: journaling unable request can not be sent to remote lock owner

Fixes: http://tracker.ceph.com/issues/26939
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 985535bae74106aec1938813bdb18661967f468f)

Merge pull request #24091 from ceph/wip-ansi-version-mimic

mimic: qa/tests: update ansible version to 2.5

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/tests: Set ansible-version: 2.5

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit f40ea9709fedd5ff14a4267961d4a4061b84181e)

Merge pull request #24008 from SUSE/wip-fix-35906

mimic: ceph-disk: compatibility fix for python 3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24081 from alfredodeza/mimic-rm35970

mimic: ceph-volume fix zap not working with LVs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24079 from alfredodeza/mimic-wip-rm35913

mimic: ceph-volume batch: allow --osds-per-device, default it to 1

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.functional comment why not use --destroy when zapping lvs in filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 11858c74a1e704d63185f0ef4b6cd6aa02f17e52)

ceph-volume tests.functional use --destroy when zapping lvs in bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 87af815691a7f43d14067c24e308d60f39d58051)

ceph-volume lvm.zap use remove_pv only for physical devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 036683fde799ab1cfad7d6192981515d6b3419cc)

Merge pull request #23105 from ukernel/mimic-24914

mimic: mds: introduce cephfs' own feature bits

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests.lvm.strategies initial filestore validation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d1d68eade56ba3f898cd3bfd29aa1ee3bf5460ec)

ceph-volume lvm.tests conftest fixtures for custom vgs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9f14db432317c15b29b187d1e1a4bd3a729a7043)

ceph-volume tests.devices.lvm add bluestore strategy cases

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2ced05a2bc6ba01cf8946f43d5839890a28cab20)

ceph-volume tests verify strategy validators for minimum size

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9167f76c2a7107c82b781cbe5f51e24ee72e4db4)

ceph-volume lvm.batch.filestore allow osds-per-device

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 66947537b542da82185ea4d79bfaedaae3117918)

ceph-volume lvm.batch.validators account for osds per device, include collocation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 04a40eda0ac089c07654d9ab07958126f0fc8f84)

ceph-volume lvm.batch.bluestore allow osds-per-device with MixedType

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 028ef60df768c24a9573a044a451164dcb037677)

ceph-volume lvm.batch.bluestore allow osds-per-device with SingleType

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8c0e707110e284626fe4161097a3121170a3f09d)

ceph-volume lvm.batch add --osds-per-device flag, defaulting to 1

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fe7c5e0129172a4b2a216e7c49e67d8ead570b39)

Merge pull request #24058 from theanalyst/civetweb-mimic-poll

mimic: rgw: civetweb: use poll instead of select while waiting on sockets

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24041 from smithfarm/wip-35942-mimic

mimic: ceph tell osd.x bench writes resulting JSON to stderr instead of stdout.

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24074 from alfredodeza/mimic-rm34535

mimic: ceph-volume batch carve out lvs for bluestore

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.util verify Disk objects don't change state with divisions

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9ca526336221ea2ad32e7daf3e5e46b07291a1d8)

ceph-volume util.disk fix an issue where Disk objects would mutate on div operations

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7d1a3c74240e67621e993a304eaf566d4e2789d1)

ceph-volume util.prepare add a helper to get block.db sizes from ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 860068ec5daca6d7087cf1ce0594a2fe36e91193)

ceph-volume lvm.batch.bluestore add TODOs for custom fast/slow devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0eb0bdfc55d8ceb27cd12bc3a7fb807a7e65c788)

-f  ceph-volume lvm.batch.bluestore validation and reporting with VG reuse

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d117d66903fa740108c9948b499150b02523fdf4)

ceph-volume lvm.batch fix error reporting, Device objects aren't strings

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 267a386cc270081b2cd284bf78bf94b52be113dc)

ceph-volume lvm.batch.bluestore validation and reporting with VG reuse

Reworks the bluestore validation and reporting to account for reusable
VGs from fast devices, and adds validation calls to ensure the new way
to calculate this process will work.

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 27f47565a0d43ecee03dea4f6addc739e3e78655)