mgr/volumes: Fix subvolumegroup ls

The subvolumegroup ls listed '_deleting' directory which is
internal to 'mgr/volumes' and should not be listed as a
subvolumegroup. This patch fixes the same by filtering it.

Fixes: https://tracker.ceph.com/issues/54099
Signed-off-by: Kotresh HR <khiremat@redhat.com>

qa: Add test for subvolumegroup ls filter

Fixes: https://tracker.ceph.com/issues/54099
Signed-off-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #44748 from cyx1231st/wip-crimson-improve-log-tm

crimson/os/seastore/transaction_manager: cleanups and logs

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44760 from mgfritch/fixup-asyncssh-mypy

mgr/cephadm: improve asyncssh type checking

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Melissa Li <melissali@redhat.com>

Merge pull request #44792 from Matan-B/wip-matanb-email-contact

mailmap: Add Matan Breizman

Reviewed-by: Neha Ojha <nojha@redhat.com>

mailmap: Add Matan Breizman

Add Matan Breizman to mailmap.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #44304 from guits/guits-fix-cv-batch-mpath

ceph-volume: improve mpath devices support

Merge pull request #44782 from liu-chunmei/crimson-fix-content-corrupt

crimson: fix qa/RadosModel content corrupt

reviewed by: Samuel Just <sjust@redhat.com>

Merge pull request #44718 from liu-chunmei/crimson-fix-split_pin_right-assert

crimson: fix split_pin_left assert_aligned

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson: fix qa/RadosModel content corrupt

RadosModel test read more length than object size, if truncate
not append zero to bufferlist, RadosMode will read more data and
it will check if there is content after the object size. So append
zero after truncate size to end of extent.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson: eliminate warning for unused variable

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson: fix split_pin_left assert_aligned

hint passed to lba alloc_extent may not be block size aligned
in a certain case, this hint insert lba tree as key, so make
laddr in lba tree is not aligned.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

ceph-volume: fix typo in tests

This fixes 2 typo in ceph-volume tests.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

doc/ceph-volume: fix a typo

This fixes a typo in ceph-volume documentation.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: add a test `test_mpath_device_is_device`

This test checks that Device.is_device() returns True for a mpath device.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: improve mpath devices support

ee8887f4c0ff4f91117f31b621b95c8d08019130 was intended for adding
mpath devices support in ceph-volume but it has missed the lvm batch scenario.
This also fixes the zapping of mpath devices prepared with `ceph-volume raw`

Fixes: https://tracker.ceph.com/issues/52908
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #44549 from melissa-kun-li/remove-iscsi

mgr/cephadm/iscsi: use `mon_command` in `post_remove` instead of `check_mon_command`

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #44682 from sebastian-philipp/cephadm-ipv6-parse-ipv6-addr-wo-route

cephadm: _parse_ipv6_route: Fix parsing ifs w/o route

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

mgr/cephadm: improve asyncssh type checking

asyncssh 2.9.0 introduces additional type hints

Fixes: https://tracker.ceph.com/issues/54003
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #44735 from neha-ojha/wip-reduce-cephadm

qa/suites/rados: reduce the number of cephadm tests

Reviewed-by: Adam King adking@redhat.com
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #44662 from guits/cv-shared_ceph_folder

cephadm: add shared_ceph_folder opt to ceph-volume subcommand

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44695 from liewegas/fix-cephadm-pretty-timedelta

pybind/mgr/mgr_util: fix to_pretty_timedelta()

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44459 from adk3798/misc-args

mgr/cephadm: allow miscellaneous container args at service level

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44192 from liuqinfei/docfix

doc: typo fixes in ceph-volume and install

cephadm: add shared_ceph_folder opt to ceph-volume subcommand

This commit adds the `--shared_ceph_folder` option to `ceph-volume`
subcommand, just like `shell` and `bootstrap` subcommands.

Fixes: https://tracker.ceph.com/issues/53931
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #44282 from orozery/qa-qemu-nbd-ide-interface

qa/tasks/qemu: switch nbd devices from virtio to ide

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #44066 from guits/guits-fix-rollback-osd

ceph-volume: zap osds in rollback_osd()

Merge pull request #44755 from ronen-fr/wip-rf-be2-clang1

osd/scrub: fix ceph::mutex usage

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #44740 from liu-chunmei/crimson-fix-remove_from_lru-assert

crimson: fix remove_from_lru assert

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #44653 from cyx1231st/wip-crimson-osd-empty-transaction

crimson/osd: ShardServices::dispatch_context_transaction to flush

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction_manager: consolidate logs in levels

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction_manager: abort when error

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction_manager: drop the extent metrics

These metrics won't distinguish extent operations from invalidated and
successful transactions.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #44761 from mgfritch/cephadm-asyncssh-2_8

mgr/cephadm: require asyncssh 2.8

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Melissa Li <melissali@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: require asyncssh 2.8

Fixes: https://tracker.ceph.com/issues/54003
Signed-off-by: Michael Fritch <mfritch@suse.com>

osd/scrub: fix clang compilation issues

Issues created by the scrub-backend refactoring PR.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #44737 from adamemerson/wip-wrong-modelines

cleanup: Change incorrect modelines

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson: fix remove_from_lru assert

RETIRED_PLACEHOLDER extent also put into retired set, when do
commit_retire_extent need check if it is RETIRED_PLACEHOLDER,
otherwise will cause remove_from_lru assert.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge PR #44574 into master

* refs/pull/44574/head:
qa: Add syslog ignore list

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #44714 from idryomov/wip-rbd-mirror-delprop-races

rbd-mirror: fix races in snapshot-based mirroring deletion propagation

Reviewed-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #42993 from ronen-fr/wip-ronenf-scrub-be2

osd/scrub: extracting scrub "backend" functionality

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #44744 from ronen-fr/wip-rf-lost-update

scrub/osd: add a missing 'publish stats to osd'

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>

crimson/os/seastore: suppress verbose onode info logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #44649 from cyx1231st/wip-crimson-improve-log-cache

crimson/os/seastore: consolidate seastore_t and seastore_cache logs

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/run_xfstests_qemu.sh: disable 251, 260 and 288

All three are skipped with virtio disks:

251  [not run] FITRIM not supported on /dev/vdc
260  [not run] FITRIM not supported on /dev/vdc
288  [not run] FITRIM not supported on /dev/vdc

But 260 and 288 fail with ide disks, where discard defaults to on.  The
ancient kernel in our ubuntu-12.04.qcow2 doesn't support virtio discard
anyway so let's just disable them for consistency.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

scrub/osd: add a missing 'publish stats to osd'

to publish the last scrub status report.
The change is needed following the merge of
PR #42735.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #44572 from 5cs/fix-spdk-cmake

cmake/modules/BuildSPDK.cmake: link whole-archive

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

qa/run_xfstests_qemu.sh: fall back to ide disks if needed

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

osd/scrub: change missing_digest fix-list type

...allowing for efficient creation and, more important,
direct consumption by the function performing the
actual I/O.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: scrub components embedded documentation

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: add to PgScrubber a local ref to next-scrub's details

The owner is still the PG.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
osd/scrub: removing PgScrubber's ref to the primary scrub map

as caching this reference (to an object that is owned by the
scrubber backend) creates a dangling-pointer risk.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #44669 from idryomov/wip-rbd-switch-arguments-fix

rbd: add missing switch arguments for recognition by get_command_spec()

Reviewed-by: Mykola Golub <mgolub@suse.com>

osd/scrub: minor fixes split from main "scrub backend" commit

Reformatting, minor compiler warnings, etc.
Split from "osd/scrub: extract scrub backend code into separate classes"

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: extract scrub backend code into separate classes

A refactoring of the scrub backend code (all Scrub related code
that checks/manipulates objects' (meta)data).

Part of the refactoring was breaking long chunks of code into
separate functions. In order to avoid passing endless lists
of state parameters from/to these new methods, the scrub backend
state is now maintained within its own objects.

a change to note:
Instead of two scrub-map collections - one that holds all
incoming maps, and one that holds pointers to all of those incoming
plus the one we (the Primary) manage - just use one collection
for both incoming and our own.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Split from "osd/scrub: minor fixes split from main "scrub backend" commit"

Split from "osd/scrub: scrub components embedded documentation"

fix

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
osd/scrub: fix the scrubber backend to include all PR #42735

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #44736 from adamemerson/wip-no-using-namespace-std

src: Remove some 'using namespace std' lines that crept back into headers

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

neorados: Remove using namespace std::literals from header

We don't actually need it.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mgr: Remove straggling `using namespace std` from headers

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Remove straggling `using namespace std` from headers

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

{osd,crimson}: Remove straggling `using namespace std` from headers

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove straggling `using namespace std` from headers

A few crept back in.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cleanup: Change incorrect modelines

Change C modelines to C++ for C++ files.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #44734 from anthonyeleven/anthonyeleven/cephfs-pool-recs

doc/cephfs: Clarify CephFS pool guidance

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #44509 from rzarzynski/wip-crimson-alienstore-drop-constant

crimson/alienstore: drop unused constant from AlienStore.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

qa/suites/rados: reduce the number of cephadm tests

Currently, every rados run of ~400 jobs is running ~150 cephadm tests,
which is unnecessary and redundant. With this change, we will run some
basic cephadm tests within the rados suite. The following seems to be
a good start.

qa/suites/rados/cephadm/osds
qa/suites/rados/cephadm/smoke
qa/suites/rados/cephadm/smoke-singlehost
qa/suites/rados/cephadm/workunits

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44646 from neha-ojha/wip-backport-q

script: backport updates for quincy

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #44265 from KellenRenshaw/WIP-replay_alloc

test/allocator_replay_test: Add replay_alloc option

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #44705 from ceph/wip-yuriw-build-script

script/build-integration-branch: add quincy to the list of releases

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/cephfs: Clarify CephFS pool guidance

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #44674 from adamemerson/wip-53941

rgw: Report empty endpoints as error instead of crashing

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #44706 from rzarzynski/wip-tests-bl-fix-rebuild

test/bufferlist: ensure rebuild_aligned_size_and_memory() always rebuilds.

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44717 from zdover23/wip-doc-2021-01-22-man-page-radosgw-admin-missing-apostrophe

doc/man: radosgw-admin.rst: missing apostrophe

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/man: radosgw-admin.rst: missing apostrophe

This commit adds two missing apostrophes to two
different strings.

Hat tip to IcePic.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #44694 from alimaredia/wip-rgw-kmip-teuthology-test

qa: move certificates for kmip task into /etc/ceph

Reviewed-by: Al Maredia <amaredia@redhat.com>

test/rbd_mirror: drop redundant MockJournaler instances

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd-mirror: fix races in snapshot-based mirroring deletion propagation

When remote image is deleted, rbd-mirror can encounter three cases:

  1) no remote image id
  2) no remote mirror metadata
  3) MIRROR_IMAGE_STATE_DISABLING in remote mirror metadata

Commit d4c66ac5c615 ("rbd-mirror: fix issue with snapshot-based
mirroring deletion propagation") fixed case 1.  Cases 2 and 3 remained
broken because for both of them finalize_snapshot_state_builder() would
populate not only remote_mirror_peer_uuid but also remote_image_id,
thus disabling ENOLINK logic in handle_prepare_remote_image() and
handle_bootstrap().  Commit ff60aec2d9ef ("rbd-mirror: fix bootstrap
sequence while the image is removed") touched on case 3, but it made
a difference only for journal-based mirroring.

Stop calling finalize_snapshot_state_builder() on errors.  Instead,
align with journal-based mirroring by filling remote_mirror_peer_uuid
together with remote_mirror_uuid.

Fixes: https://tracker.ceph.com/issues/53963
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #44112 from zaitcev/wip-py_ssize_t_clean

mgr: Define PY_SSIZE_T_CLEAN ahead of every Python.h

rbd-mirror: don't default replay_requires_remote_image() implementation

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd-mirror: untangle StateBuilder::is_linked() overloads

Make it clear that the local image non-primariness is asserted
independent of the mode; avoid the default implementation being
overridden but still relied on by both modes.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd-mirror: drop redundant initialization of StateBuilder members

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #44604 from mgfritch/c-v-skip-rbd-devs

ceph-volume: show RBD devices as not available

Merge pull request #44687 from sebastian-philipp/gov

doc/governance: Adam King

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os/seastore/journal: add logs to flush operation

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: consolidate logs in levels

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: introduce seastore_t logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: move transaction-id forward in log

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: misc cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: cleanup effort_t and reuse Transaction::io_stat_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: consolidate Cache::init_cached_extent with drop_from_cache

So Cache::drop_from_cache() can be removed, and make it possible to add
related logs within the Cache component.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/bufferlist: ensure rebuild_aligned_size_and_memory() always rebuilds.

Before the patch the test case was showing an unreliable behaviour
dependent on the underlying memory allocator. It was because
the bufferlist rebuild can be skipped, resulting in unchanged number
of buffers, if all of them begin at aligned addresses.

The commit fixes that by allocating a 4 KiB-aligned buffer and
offsetting it by a small constant (42) to ensure the memory added
to the bufferlist begins at non-4 KiB address.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tests: add quincy name to teh wip name

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa: move certificates for kmip task into /etc/ceph

On rhel/centos the ceph user does not have permission
to access these certs which leads to s3-test failures
in teuthology.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #44042 from myoungwon/wip-add-blk-paddr

seastore: add blk_paddr_t

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44685 from rhcs-dashboard/fix-53950-master

mgr/dashboard: remove feedback module as always-on

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44665 from guits/cv-sw-logger

ceph-volume: don't use MultiLogger in find_executable_on_host()

Merge pull request #43345 from trociny/wip-osdmaptool-down

tools/osdmaptool: fix possible segfaults when there are down osds

mgr/dashboard: fix openapi-check

Fixes: https://tracker.ceph.com/issues/53950
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

pybind/mgr/tests: fix style

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/mgr/mgr_util: fix to_pretty_timedelta

- total_seconds, not seconds
- int, not float

Added unit tests.

Fixes: https://tracker.ceph.com/issues/53953
Signed-off-by: Sage Weil <sage@newdream.net>

rgw: Report empty endpoints as error instead of crashing

Fixes: https://tracker.ceph.com/issues/53941
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph-volume: filter RBD devices from the device inventory

Avoid running `blkid` or deploying OSDs on RBD devices by ensuring they
do not appear in the `ceph-volume inventory`

Fixes: https://tracker.ceph.com/issues/53846
Signed-off-by: Michael Fritch <mfritch@suse.com>