nautilus: qa/ceph-ansible: ansible-version and ceph_ansible

Upgrade to 2.8.1 and stable-4.0 respectively

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #33278 from smithfarm/wip-44085-nautilus

nautilus: ceph-monstore-tool: correct the key for storing mgr_command_descs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33277 from smithfarm/wip-43722-nautilus

nautilus: common/bl: fix the dangling last_p issue.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33276 from smithfarm/wip-44082-nautilus

nautilus: qa/suites/rados/multimon/tasks/mon_clock_with_skews: disable ntpd etc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32844 from smithfarm/wip-43239-nautilus

nautilus: mgr/DaemonServer: fix 'osd ok-to-stop' for EC pools

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #33337 from jan--f/wip-44153-nautilus

nautilus: ceph-volume: don't remove vg twice when zapping filestore

Merge pull request #33334 from jan--f/wip-44152-nautilus

nautilus: ceph-volume: pass journal_size as Size not string

ceph-volume: don't remove vg twice when zapping filestore

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
Fixes: https://tracker.ceph.com/issues/44149
(cherry picked from commit bccdf6eafaf851d5092bb99d61edd44cd36d9dd2)

ceph-volume: pass journal_size as Size not string

Fixes: https://tracker.ceph.com/issues/44148
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 49f6e6d600aae6310f941c6635408d496b0ff2b9)

Merge pull request #33301 from jan--f/wip-43871-nautilus-failed-cp

nautilus: ceph-volume: batch bluestore fix create_lvs call

Merge pull request #33297 from jan--f/wip-44135-nautilus

nautilus: ceph-volume: avoid calling zap_lv with a LV-less VG

ceph-volume: batch bluestore fix create_lvs call

Fixes: https://tracker.ceph.com/issues/43844
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit df18497bc9eaf1922e5c885e8cc124e439c59364)

ceph-volume: avoid calling zap_lv with a LV-less VG

Fixes: https://tracker.ceph.com/issues/44125
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit ad0dea53b8585d8397bf3069b1b39f13b6e0a8ce)

Merge pull request #33151 from shyukri/wip-43877-nautilus

nautilus: rgw: fix one part of the bulk delete(RGWDeleteMultiObj_ObjStore_S3) fails but no error messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33149 from shyukri/wip-43874-nautilus

nautilus: rgw: maybe coredump when reload operator happened

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33008 from smithfarm/wip-43922-nautilus

nautilus: rgw_file: avoid string::front() on empty path

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33152 from shyukri/wip-43879-nautilus

nautilus: mon: Don't put session during feature change

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #33095 from k0ste/wip-43979-nautilus

nautilus: mgr/telemetry: check get_metadata return val

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #32908 from smithfarm/wip-43821-nautilus

nautilus: mon/Session: only index osd ids >= 0

Merge pull request #33170 from k0ste/wip-43727-nautilus

nautilus: mgr/pg_autoscaler: calculate pool_pg_target using pool size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33168 from k0ste/wip-44057-nautilus

nautilus: mgr/telemetry: split entity_name only once (handle ids with dots)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33157 from shyukri/wip-43924-nautilus

nautilus: mgr/prometheus: report per-pool pg states

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #33155 from shyukri/wip-43916-nautilus

nautilus: mon/ConfigMonitor: only propose if leader

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33147 from shyukri/wip-43989-nautilus

nautilus: osd: Allow 64-char hostname to be added as the "host" in CRUSH

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33142 from shyukri/wip-44000-nautilus

nautilus: mon/MgrMonitor.cc: warn about missing mgr in a cluster with osds

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33082 from k0ste/wip-43974-nautilus

nautilus: mgr/telemetry: anonymizing smartctl report itself

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32948 from yaarith/wip-telemetry-serial-nautilus-yh

nautilus: mgr/telemetry: fix device serial number anonymization

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33007 from smithfarm/wip-43928-nautilus

nautilus: mon: elector: return after triggering a new election

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #32931 from smithfarm/wip-43819-nautilus

nautilus: mgr/pg_autoscaler: default to pg_num[_min] = 32

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #32905 from smithfarm/wip-43731-nautilus

nautilus: crush/CrushWrapper: behave with empty weight vector

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32856 from zhengchengyao/nautilus_no_mon_update

nautilus: mon/ConfigMonitor: fix handling of NO_MON_UPDATE settings

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: update mondb recovery script

to note that we also need to add mgr's key to monitor's keyring

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 75f4765f2ffe795dba85540b8aa1675ba9de28e4)

ceph-monstore-tool: correct the key for storing mgr_command_descs

Fixes: https://tracker.ceph.com/issues/43582
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a5bfeca64f3204142fda5320b4fd403df4b5f532)

ceph-monstore-tool: rename mon-ids in initial monmap

when ceph-mon starts, it checks to see if it's listed in the monmap, if
not it complains
```
no public_addr or public_network specified, and mon.a not present in
monmap or ceph.conf.
```
then bails out. normally, the monitor will try to rename its name in
monmap when performing "mkfs", but in our case, we are merely using the
"mkfs" monmap for passing the monmap built by ceph-monstore-tools, and
we don't actually go through the "mkfs" process. so, ceph-mon won't
rename when booting up.

in this change, user is allowed to specify the mon-ids in command line
when rebuilding mondb, the default mon-ids would be a,b,c,... if not
specified.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 4b3df5a850db054928f9fcc6ef0a160a05a2ffa9)

common/bl: fix the dangling last_p issue.

Fixes: https://tracker.ceph.com/issues/43646
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 8198332f3afb5de748dfa6e3349fefbf7e9ed137)

Conflicts:
src/test/bufferlist.cc

qa/suites/rados/multimon/tasks/mon_clock_with_skews: whitelist MOST_DOWN

The skewed clock makes some mons miss elections.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 08b6a2bc008c006ff815dcb85bfb24b73072c7ab)

qa/suites/rados/multimon/tasks/mon_clock_with_skews: disable ntpd etc

Fixes: https://tracker.ceph.com/issues/43889
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 9f2a854b175f156d4ab7fba955aff515052c9d93)

Merge pull request #33254 from jan--f/wip-44112-nautilus

nautilus: ceph-volume: use get_device_vgs in has_common_vg

Merge pull request #33253 from jan--f/wip-44109-nautilus

nautilus: ceph-volume: fix is_ceph_device for lvm batch

Merge pull request #33240 from jan--f/wip-44035-nautilus

nautilus: ceph-volume: finer grained availability notion in inventory.

Merge pull request #33239 from jan--f/wip-43984-nautilus

nautilus: ceph-volume: fix has_bluestore_label() function

ceph-volume: use get_device_vgs in has_common_vg

Fixes: https://tracker.ceph.com/issues/44099
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 2c5a8c3b4066dd2aca47d719c9723850ce5f96fc)

ceph-volume: add is_ceph_device unit tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 60d80636e4708761287197c534347f82e307c603)

ceph-volume: fix is_ceph_device for lvm batch

This is a regression introduced by 634a709

The lvm batch command fails to prepare the OSDs on the created LV.
When using lvm batch, the LV/VG are created prior the OSD prepare.
During that creation, multiple tags are set with null value.

$ lvs -o lv_tags --noheadings
  ceph.cluster_fsid=null,ceph.osd_fsid=null,ceph.osd_id=null,ceph.type=null

Since we call is_ceph_device which returns True if the ceph.osd_id LVM
tag exists but doesn't test the value then we raise an execption.

When the tag value is set to 'null' then we can consider that the device
isn't part of the ceph cluster (because not yet prepared).

Closes: https://tracker.ceph.com/issues/44069
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit a82582364c7b65a4a5e2673e3886acd6d2066130)

Merge pull request #33242 from jan--f/wip-44047-nautilus

nautilus: ceph-volume: skip osd creation when already done

Merge pull request #32919 from smithfarm/wip-43780-nautilus

nautilus: cephfs: qa: ignore trimmed cache items for dead cache drop

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33183 from smithfarm/wip-43846-nautilus

nautilus: rgw: update the hash source for multipart entries during resharding

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #33115 from batrick/i43790

nautilus: RuntimeError: Files in flight high water is unexpectedly low (0 / 6)

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #32921 from smithfarm/wip-43784-nautilus

nautilus: mds/OpenFileTable: match MAX_ITEMS_PER_OBJ to osd_deep_scrub_large_omap_object_key_threshold

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32918 from smithfarm/wip-43777-nautilus

nautilus: cephfs: qa: save MDS epoch barrier

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32917 from smithfarm/wip-43733-nautilus

nautilus: cephfs: qa: ignore slow ops for ffsb workunit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32756 from batrick/i43347

nautilus: mds: fix assert(omap_num_objs <= MAX_OBJECTS) of OpenFileTable

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #31905 from batrick/i43046

nautilus: mgr: "mds metadata" to setup new DaemonState races with fsmap

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #32916 from smithfarm/wip-43729-nautilus

nautilus: cephfs: client: Add is_dir() check before changing directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #32807 from smithfarm/wip-43770-nautilus

nautilus: mount.ceph: remove arbitrary limit on size of name= option

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #32910 from smithfarm/wip-43503-nautilus

nautilus: mount.ceph: give a hint message when no mds is up or cluster is laggy

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #33122 from ajarr/wip-ajarr-mgr-volumes-nautilus

mgr/volumes: misc fix and feature enhancements

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa/standalone/misc/ok-to-stop: improve test

Make sure PGs peer (simply flushing state to mon isn't enough).

Fixes: https://tracker.ceph.com/issues/43721
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 76ea774c109cc565c0b85feed40f7f29105029d3)

qa/standalone/ceph-helpers: add wait_for_peered

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 78ec6aec90c7d0eb8b017c7f3d34e376b7f6713f)

mgr/DaemonServer: fix 'osd ok-to-stop' for EC pools

We need to pay attention to account for CRUSH_ITEM_NONE entries in the
EC PG acting set.

Fixes: https://tracker.ceph.com/issues/43151
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 66690ea3143ac5097f7c7f3118f2d00fed30cc4b)

Conflicts:
        qa/standalone/misc/ok-to-stop.sh
- nautilus "ceph osd pool create" CLI command takes a pg_num argument

ceph-volume: add unit test test_safe_prepare_osd_already_created

This commit adds a new unit test
`test_safe_prepare_osd_already_created()` in order to test when
`is_ceph_device()` returns `True` `RuntimeError` is well raised.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit ccf92d718d6f363f6ca3b7e8499b68beb8b30c06)

ceph-volume: skip osd creation when already done

When rerunning ceph-volume lvm create on a device already prepared and
activated, ceph-volume should skip the creation.

This is a regression introduced by bb4de1a3fc238eaf9f717dc59c6bdf338ef6d657

Fixes: https://tracker.ceph.com/issues/43981
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 634a709b9c6802c5b12e2d45c2f43181b297adfb)

ceph-volume: add available property in target specific flavors

This adds two properties available_[lvm,raw] to device (and thus inventory).
The goal is to have different notions of availability based on the
intended use case. For example finding LVM structures make a drive
unavailable for the raw mode, but might be available for the lvm mode.

Fixes: https://tracker.ceph.com/issues/43400
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 233ccff24006082766b52a94b7c46cdf5b7cd929)

ceph-volume: remove stderr in has_bluestore_label()

We don't want to generate this log when a call to
`has_bluestore_label()` fails.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 7f8371c32b3f613b0d094f6f26ffbeb19ea0d25b)

ceph-volume: fix has_bluestore_label() function

When using vg/lv, this function throws an error like following:

```
 stderr: unable to read label for test_group/data-lv2: (2) No such file or directory
 stderr: 2020-02-04T21:03:32.153+0000 7fe091af4200 -1 bluestore(test_group/data-lv2) _read_bdev_label failed to open test_group/data-lv2: (2) No such file or directory
```

using `self.abspath` fixes this error.

Fixes: https://tracker.ceph.com/issues/43970
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 148069a20fef79ef8fe510f06879a0de02987eac)

Merge pull request #33238 from jan--f/wip-31700-notracker-nautilus

nautilus: ceph-volume: refactor listing.py + fixes

ceph-volume: fix various lvm list issues

A single report on a non-lvm device now works.
Format was cleaned up, report lvm journal,wal, db only once.

Fixes: https://tracker.ceph.com/issues/44009
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 000bf2ffff57701952e2aa1a67a04e519c4d07a6)

ceph-volume: add get_device_lvs to easily retrieve all lvs per device

Also drop the sep argument from get_lvs and siblings, unused.
Introduce LV_CMD_OPTIONS to unify options to lvs.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit ffe5b5732a46bd5ff888696adbfe53a38c982448)

ceph-volume: fix lvm list

17957d9beb42a04b8f180ccb7ba07d43179a41d3 introduced a regression in `lvm
list`.

When passing a vg/lv path for generating a single report, it fails
because the filter used in the `lvs` command isn't right. It uses the lv
name instead of the vg name because `os.path.basename(device)` is used
while it should be `os.path.dirname(device)`

Fixes: https://tracker.ceph.com/issues/43969
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 0179fed3ab624830ba77349531763c3e116c82e5)

ceph-volume: delete test_lvs_list_is_created_just_once

lisitng.py doesn't call api.Volumes anymore. Therefore, this test is
redundant.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 665ed2424b7bef4160289f0135acc015f8ea9980)

ceph-volume: update tests since listing.py got heavily modified

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit d1ae6d1a8b495adfc0c512f08359e0db1590272d)

ceph-volume: refactor devices/lvm/listing.py

Get rid of duplicate and redundant code and use get_lvs, get_vgs and
get_pvs to simplify the module as much as possible.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit d02bd7dd581a4bd4041eb397fae540a18f16a88b)

ceph-volume: add new method in api/lvm.py

The method determines whether given LV is managed by Ceph or not.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 876244b6ab2cf1fbd724fd33966501a3366c6d3f)

ceph-volume: add helper methods to get only first LVM devs

These convenience methods shortens following phrase to
"lv = get_first_lv()" -

lvs = get_lvs()
if len(lvs) >= 1:
lvs = lv[0]

These methods do the same things as above phrase internall. Rewrite
listing.py to use these new helper methods.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 17957d9beb42a04b8f180ccb7ba07d43179a41d3)

ceph-volume: filter based on tags for api.lvm.get_* methods

get_pvs, get_vgs and get_lvs must accept tags and filter volumes based
on tags.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit fb1390964fdfd10815ab4a4302ad454169bc0d5f)

Merge pull request #33232 from jan--f/wip-43871-nautilus

nautilus: ceph-volume: batch bluestore fix create_lvs call

Merge pull request #33231 from jan--f/wip-43849-nautilus

nautilus: ceph-volume: add sizing arguments to prepare

mgr/volumes: fix py2 compat issue

Fix the following issue seen while upstream teuthology testing,
 File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 98, in load_config
   self.metadata_mgr = MetadataManager(self.fs, self.legacy_config_path, 0o640)
 File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 73, in legacy_config_path
   meta_config = "{0}.meta".format(m.digest().hex())
 AttributeError: 'str' object has no attribute 'hex'

This issue is not observed in master/octopus, as it only supports
py3.

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/volumes: type convert uid and gid to int

This fix is only needed in nautilus, and the issue
was observed during upstream teuthology testing.

   File "/usr/share/ceph/mgr/volumes/fs/async_cloner.py", line 114, in cptree
     copy_file(fs_handle, d_full_src, d_full_dst, mo, st.st_uid, st.st_gid)
   File "/usr/share/ceph/mgr/volumes/fs/fs_util.py", line 97, in copy_file
     fs.chown(dst, uid, gid)
   File "cephfs.pyx", line 855, in cephfs.LibCephFS.chown
      TypeError: uid must be an int

The issue wasn't observed in master/octopus teuthology
testing.

Signed-off-by: Ramana Raja <rraja@redhat.com>

test: add subvolume clone tests

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit b5970ff80d9d203e6289770cbb8f51cc6a2c5938)

mgr/volumes: allow force removal of incomplete failed clones

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 451be112603f970de3cb0ca9048c88a98c034188)

mgr/volumes: asynchronous cloner module

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 4f09568b012cef24d2075733f9d81064790fe4e6)

mgr/volumes: purge thread uses new async interface

This also makes `_cancel_jobs()` thread safe, which was not the
case earlier (with `_cancel_purge_job()`) -- this also makes the
code simpler by sharing the lock betweent two condition variables.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit f16cc1e8eebb0868a1e07c25f8d8e4656b11b7bf)

mgr/volumes: fetch oldest clone entry

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 7ad14cf09b447fcdbfe76ac739971be8609aa862)

mgr/volumes: add clone specific commands

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit b2145b73d7640851c27ac4bb09fd98e146de953d)

mgr/volumes: interface for fetching cloned subvolume status

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit fa3c56f55273eca5be0a4a9d9fc93aab3dacda2b)

mgr/volumes: add protect/unprotect and snap clone interface

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 8d68f1af3a12ce096e52024aa710e03e0006ba0c)

mgr/volumes: handle transient subvolume states

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 461909be2edd2f691263785469e5fc90a82448e8)

mgr/volumes: interface for creating a cloned subvolume

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 7089808bf82321712676803dd4bfdd62cf5fc00e)

mgr/volumes: get/set property for subvolume mode/uid/gid

This will be required when creating a clone as the clone would
inherit source subvolumes creation mode and uid/gid.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit f02b1e7e546609a4f4287a27b3495a09e287888b)

mgr/volumes: module to track pending clone operations

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 53187795838ccee4b470f3a08aeaf6e474975408)

mgr/volumes: add operation state machine table

... and fetch creation state from state machine table.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 46f29bf3b4fbb86aeefc940d47eadd6634b5a1ec)

mgr/volumes: fail removing subvolume with snapshots

Fixes: http://tracker.ceph.com/issues/43645
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit c158a1334207966c9727eb1e9e2eaab756e5e08b)

mgr/volumes: remove stale subvolume module

this was lying around post versioning changes.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 5595476af7fe4dc2dfdeae17d5507d1d8eeeb3f5)

test: auto-upgrade subvolume test

Fixes: https://tracker.ceph.com/issues/43349
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 03ee966b6c92b43a4f38f35351baf9ebc0126bbc)

mgr/volumes: tie everything together to implement versioned subvolumes

apart from the new way of provisioning subvolumes, this makes heavy
use of context manager for volumes, groups and subvolumes.

this change classifies volumes, groups and subvolumes to be treated
as filesystem dentries and inodes. a "volume" can be thought as a
dentry with "groups" as it's entries (inodes). likewise, a "group"
is a dentry again with "subvolumes" as entries (inodes). this is
built into the access mechanism as follows:

      with open_volume(...) as fs_handle:
          with open_gorup(fs_handle, ...) as group:
              with open_subvolume(group, ...) as subvolume:
                  # call subvolume object API
                  path = subvolume.getpath()

this way, lot of redundant checks such as verifying if a volume or
group exist before accessing a subvolume is built right into the
access mechanism, plus, an added bonus of simple error handling.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 9b87bd72093eaecda5e2ffc3ed32e482431f9842)

mgr/volumes: provide subvolume create/remove/open APIs

create_subvolume() creates a subvolume with the max version known
to the plugin. open_subvolume() performs version discovery by
using loader stub and returns a subvoule object.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 8a649148025620991a67ed761153f18594f5268c)

mgr/volumes: implement subvolume based on subvolume template

subvolume base class implements common routines/helpers and
initializes a metadata manager. later, when v2 subvolume version
is implemented, the metadata manager would be used to persist
subvolume metadata in ceph filesystem. this would allow flexible
metadata management when complex subvolume features are added.

typically, a subvolume would be implemented by subclassing the
subvolume base class and the subvolume template -- instantiating
this would be called a "subvolume object".

with this commit, current subvolume topology is maintained. but
we introduce the concept of subvolume versions. a loader stub
loads available "versions" of subvolumes. right now, the only
available version is v1. since backward compatibility needs to
be maintained for existing subvolumes, the loader API allows
version discovery w/ auto upgradation to the most recent version.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 97170d72b764ed26e93b45e8e2e7ecc2368a2a04)

mgr/volumes: implement subvolume group based on group template

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 3eccd618ecce13052e31d07437ee2d09ae98e5c4)

mgr/volumes: implement trash as a subvolume group

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 0e3c48e39b2f366877fb38aa79438b0f7c5ed075)