git.apps.os.sepia.ceph.com Git

mgr: helper function to check if a service is a normal ceph service

This would be widely required since ceph metadata server entries are
maintained in service map (DaemonServer::pending_service_map). Such
normal ceph services would need to filtered when processing the service
map to avoid extraneous entries getting processed.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 79503fc16749ed0cfe8a89ea3b3c8c792d6b8809)

mds: establish session with mgr only after added to FSMap

Note that we now sub to the mgrmap after init because the MgrClient
connection to the mgr is driven by receipt of the MgrMap.

This is important so that the MDS does not have metadata with the mgr
when the mons are ignoring the MDS otherwise due to CompatSet
incompatibilities.

Fixes: https://tracker.ceph.com/issues/41538
Fixes: https://tracker.ceph.com/issues/42635
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit e765f2d533440cfc4189f36fcaba24617a302e84)

Conflicts:
src/mds/MDSDaemon.cc

Nautilus uses Lock/Unlock rather than lock/unlock in master.

mds: do not register as a service daemon

This commit undoes the service daemon registration for the MDS. It doesn't look
absolutely necessary and it causes the MDS to be listed twice in the `ceph
versions` output:

    $ ceph versions
        ...
        "mds": {
            "ceph version v15.0.0-6915-g0143b904676 (0143b9046763ea1801efa8358a0c033ec862cea9) octopus (dev)": 3
        },
        "mds": {
            "unknown": 3
        },
        "overall": {
            "ceph version v15.0.0-6915-g0143b904676 (0143b9046763ea1801efa8358a0c033ec862cea9) octopus (dev)": 10,
            "unknown": 3
        }

Fixing that requires looking for duplicates or ignoring MDSs in the
service daemons when the mon processes `ceph versions`. I have a feeling
that it wasn't actually designed to be used by the MDS this way however.
Additionally, the reason for "unknown" version is because the metadata
sent to the mgr does not include "ceph_version".

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 046137f819aae72f1423e3feb213f0e46c97c9ce)

mds: do not try to diagnose cause of MDSMap removal

The Monitors send an empty MDSMap to an MDS it is removing. The MDS
can't diagnose the cause. Instead suggest looking at the cluster/monitor
logs.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit c385178b0c04d09bc6774acf5bdb4537fa6803ad)

mds: fix handling of initial MDS states

Few things here:

- Make explicit the check for getting removed from the MDSMap. This was
only done before by checking if MDS held a rank which does not check the
case where a standby is removed from the FSMap.

- Use mds_info_t::dump to simplify various debug output.

- Add a few sanity asserts for invalid state transitions.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 26a08df2adc7495fc660113cd26c33d5debd3ee6)

Conflicts:
src/mds/MDSDaemon.h

Nautilus still uses "const_ref" rather than "cref_t<>" in master.

mds: remove unnecessary const qualifier

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 3355581a298963711ad058c6e47d21b7eab887a4)

mds: cleanup type decl and map iteration

This is a trivial refactor.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit bafa3b731ff93a356fcf60ebbaf0e5f299182992)

mds: define stream operator for mds_info_t

This also cleans up the output to be more readable/useful in debug
output.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 1c56632e88a126d0eac75235d9a9716833cba6b7)

Conflicts:
PendingReleaseNotes

Fix minor conflict in PendingReleaseNotes.

mds/MDSRank: report state to mgr as mds id, not rank

Introduced by 625dffe65c0f8001b3b6ca6d0b12732a1a103849, which added
periodic scrub stats reporting.

Fixes: https://tracker.ceph.com/issues/42494
Fixes: https://tracker.ceph.com/issues/41525
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a1a220d137f7fa128d08f96d1eff83865388e5fc)

mgr: missing lock release in DaemonServer::handle_report()

Fixes: http://tracker.ceph.com/issues/42169
Introduced-by: 5c25a01864
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit cda3eadbfbd3e3a8a057cc7ed042ba0a6d7fef11)

mgr/ServiceMap: use plain gid

* partially revert 5c25a018. which is not backward compatible.
* change `ServiceMap::get_daemon()` so it returns a
`pair<Daemon*,bool>`.

git is an `optional<uint64_t>`, so we cannot dump it without checking.

Fixes: http://tracker.ceph.com/issues/41424
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 3e65551d0ab4acafaf4012601987b3391b2411ed)

doc: document scrub summary in ceph status output

Fixes: http://tracker.ceph.com/issues/36370
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 05d17994a879995b56bda5d770a938d0aabaaed9)

test: extend scrub control test to validate mds task status

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 465a3adc6c31fd9b8359920ab47adf8e1f45d5f1)

mds: send scrub state changes to cluster log.

... also log new and completed scrubs.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 49520156414cb0667e8edd6ea4b31462e7cb7752)

mds: periodically sent mds scrub status to ceph manager

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 625dffe65c0f8001b3b6ca6d0b12732a1a103849)

Conflicts:
src/mds/MDSRank.cc

mgr, mon: allow normal ceph services to register with manager

Additionally, introduce `task status` field in manager report
messages to forward status of executing tasks in daemons (e.g.,
status of executing scrubs in ceph metadata servers).

`task status` makes its way upto service map which is then used
to display the relevant information in ceph status.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 5c25a018643b10aa78db8270cae1476f71d8f4f4)

Conflicts:
src/messages/MMgrReport.h
src/mgr/DaemonServer.cc
src/mgr/ServiceMap.h

Merge pull request #33684 from k0ste/wip-44367-nautilus

nautilus: doc/mgr/telemetry: added device channel details

doc/mgr/telemetry: added device channel details

Plus details about license agreement.

Fixes: https://tracker.ceph.com/issues/43648
Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
(cherry picked from commit 66c41f3d0e516e217dc89ab20c53563ea10f97f7)

Merge pull request #34038 from rhcs-dashboard/wip-44334-nautilus

nautilus: mgr/dashboard: UI fixes

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #33764 from shyukri/wip-44327-nautilus

nautilus: mgr/dashboard: 'destroyed' view in CRUSH map viewer

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #33763 from shyukri/wip-43258-nautilus

nautilus: mgr/dashboard: Disable cache for static files

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33739 from rhcs-dashboard/wip-44435-nautilus

nautilus: mgr/dashboard: remove 'config-opt: read' perm. from system roles.

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #33738 from rhcs-dashboard/wip-44434-nautilus

nautilus: vstart.sh: set prometheus port for each mgr.

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #33530 from thmour/nautilus_test

nautilus: osd: fix racy accesses to OSD::osdmap.

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #34064 from smithfarm/wip-44145-nautilus

nautilus: rgw: set correct storage class for append

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33913 from shyukri/wip-42324-nautilus

nautilus: qa: get rid of iterkeys for py3 compatibility

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33649 from k0ste/wip-44364-nautilus

nautilus: mgr/telemetry: fix and document proxy usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33421 from k0ste/wip-44218-nautilus

nautilus: common/blkdev: fix some problems with smart scraping

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33420 from k0ste/wip-44219-nautilus

nautilus: mgr/pg_autoscaler: fix division by zero

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33409 from k0ste/wip-44199-nautilus

nautilus: mgr/telemetry: add 'last_upload' to status

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33355 from shyukri/wip-44141-nautilus

nautilus: rgw: sts: add all http args to req_info

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #33893 from rhcs-dashboard/wip-44200-nautilus

nautilus: mgr: Add get_rates_from_data to mgr_util.py

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33275 from smithfarm/wip-44081-nautilus

nautilus: mon/PGMap: fix summary display of >32bit pg states

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33261 from smithfarm/wip-43997-nautilus

nautilus: global: ensure CEPH_ARGS is decoded before early arg processing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33032 from shyukri/wip-42164-nautilus

nautilus: mgr/dashboard: REST API: OpenAPI docs require internet connection

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #32891 from shyukri/wip-42673-nautilus

nautilus: mgr/dashboard: Searchable objects for table

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34087 from jdurgin/wip-target-ratio-nautilus

nautilus: mgr/pg_autoscaler: treat target ratios as weights

Reviewed-by: Neha Ojha <nojha@redhat.com>

PendingReleaseNotes: mention target_size_ratio change

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 9118065c4c3803f8a2c26c28c69ffc9f14ff260a)

Merge pull request #34036 from ajarr/nautilus-volumes-cancel-clone

nautilus: fs subvolume clone cancel

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: update autoscaler docs for target ratio

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 0a29cf7d6c9076c79416ef7868be3635921fd140)

mgr/pg_autoscaler: include effective target ratio in status

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 9758197afa856f3680d5b5d599c79353816e9927)

mgr/pg_autoscaler: add warning when target bytes and ratio are both set

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 772d7c1d3c1adf49821670da0f6cdb06e293dc0d)

mgr/pg_autoscaler: remove target ratio warning

Since the ratios are normalized, they cannot exceed 1.0 or overcommit
combined with target_bytes.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit d62c121ee3839a2fd4243a9b1c33aded1cc3db13)

qa: use ratios >1 in pg_autoscaler test

Also check for pg_num_target being set correctly, rather than pg_num,
so the test doesn't depend on merging/splitting speed.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 1ea8ce3480e96aa87b4d4f570022f9844bc54c17)

mgr/pg_autoscaler: treat target ratios as weights

Normalize across pools so that it's simpler to use - this way you
don't have to adjust every other pool when you add one.

Handle pools with target_bytes by taking their capacity off the top,
and dividing the rest into the pools with a target_ratio.

If both target bytes and ratio are specified, ignore bytes. This
matches the docs and makes accounting simpler.

Fixes: https://tracker.ceph.com/issues/43947
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 435c56a03b7424b23fe977a13c4704304189e829)

Merge pull request #33978 from yuvalif/wip-yuval-fix-44614

nautilus: rgw/notifications: fix topic action fail with "MethodNotAllowed"

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #33965 from smithfarm/wip-44466-nautilus

nautilus: rgw: get barbican secret key request maybe return error code

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33666 from yaarith/nautilus-fix-uuid-str-concat

nautilus: mgr/telemetry: fix UUID and STR concat

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33560 from liewegas/workaround-43914-nautilus

nautilus: qa/tasks/ceph_manager: increase CLI command timeout

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: set correct storage class for append

fix https://tracker.ceph.com/issues/42444

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>
(cherry picked from commit ef3ec2e56236fb031ddd4404121449c656a62481)

Conflicts:
src/rgw/rgw_putobj_processor.cc
- cur_manifest assignment looks different in nautilus

doc: document `clone cancel` command

Fixes: http://tracker.ceph.com/issues/44208
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 60105f6101d2370f34c2d758e1452acddf962075)

test: add `clone cancel` tests

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit ba4b9358e125eb44f621a4087f46d13636d73d20)

mgr/volumes: introduce "clone cancel" volume command

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit a427c7074ce92e6e832566e0afd43c8d7d8e2770)

mgr/volumes: allow canceling a single asynchronous job for a volume

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 89a16c7867415085dd20c06d87e97bfa7de34fb2)

mgr/volumes: helper for looking up a clone entry index

... this would be used when canceling a clone operation to
find the clone index for a given clone.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 2dc41ea2db423b1e8178d41ce2fef6ca0eeccf32)

mgr/volumes: periodically check if clone operations should be canceled

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 2f93b3beb421f6be4611ffab8d1fb7fbe3fb856c)

mgr/volumes: periodically check if copy operations should be canceled

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 9a0507acb62249e904d54c5ed11ef106f3239bd8)

mgr/volumes: introduce 'canceled' state in clone op state machine

When fetching the next execution state, -EINTR jumps to 'canceled'
state signifying a canceled (interrupted) operation. Also include
a helper routine to check if a given state machine is in initial
state.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit ac9d697b1cdb93b4c7e532727b482cece3674155)

mgr/volumes: OpSmException.error -> OpSmException.errno

fixes mypy errors:

volumes/fs/async_cloner.py: note: In function "handle_clone_pending":
volumes/fs/async_cloner.py:71: error: "OpSmException" has no attribute "error"; maybe "errno"?

volumes/fs/async_cloner.py: note: In function "handle_clone_in_progress":
volumes/fs/async_cloner.py:139: error: "OpSmException" has no attribute "error"; maybe "errno"?

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 7fa4da445650fd4a9f799c48ba513cb6f1a0d26c)

mgr/volumes: remove unneeded assignment to `NoneType`

fixes mypy error:

volumes/fs/operations/versions/__init__.py: note: In member "get_subvolume_object" of class "SubvolumeLoader":
volumes/fs/operations/versions/__init__.py:70: error: Incompatible types in assignment (expression has type "None", variable has type "SubvolumeBase")

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit a09af07cb69fbca68f08b5740036cf52aa6f3c27)

mgr/volumes: add missing OpSmException import

fixes mypy error:

volumes/fs/operations/versions/__init__.py: note: In member "upgrade_legacy_subvolume" of class "SubvolumeLoader":
volumes/fs/operations/versions/__init__.py:56: error: Name 'OpSmException' is not defined

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit d5c1a7cbd5e800950495b4e70a83a8becf2f0abd)

mgr/volumes: add missing error code

fixes mypy error:

volumes/fs/operations/versions/__init__.py: note: In member "_load_supported_versions" of class "SubvolumeLoader":
volumes/fs/operations/versions/__init__.py:35: error: Too few arguments for "VolumeException"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 22b4cb9405051b6ea17f76dac85a117b1ab34a41)

mgr/volumes: remove stray return

fixes mypy errors:

volumes/fs/purge_queue.py: note: In function "get_trash_entry_for_volume":
volumes/fs/purge_queue.py:28: error: Name 'ret' is not defined

volumes/fs/async_cloner.py: note: In function "get_next_clone_entry":
volumes/fs/async_cloner.py:39: error: Name 'ret' is not defined

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 760a5918398626c956f22017a6513c7a70ac359a)

mgr/volumes: fixup format string args

fixes mypy errors:

volumes/fs/purge_queue.py:26: error: Cannot find replacement for positional format specifier 1
volumes/fs/async_cloner.py:37: error: Cannot find replacement for positional format specifier 1

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 85e57a64cbdb1e5a072e16f3f0f8015787a5abd3)

mgr/volumes: add missing `mgr` param

fixes mypy errors:

volumes/fs/operations/volume.py: note: In function "create_volume":
volumes/fs/operations/volume.py:216: error: Too few arguments for "remove_pool"
volumes/fs/operations/volume.py:223: error: Too few arguments for "remove_pool"
volumes/fs/operations/volume.py:224: error: Too few arguments for "remove_pool"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit a3b61a357730e0834ded4e6c47be0998bc4ea472)

mgr/volumes: assert self.fs

fixes mypy errors:

volumes/fs/operations/volume.py: note: In member "disconnect" of class "Connection":
volumes/fs/operations/volume.py:94: error: Item "None" of "Optional[Any]" has no attribute "get_addrs"
volumes/fs/operations/volume.py:95: error: Item "None" of "Optional[Any]" has no attribute "shutdown"

volumes/fs/operations/volume.py: note: In member "abort" of class "Connection":
volumes/fs/operations/volume.py:105: error: Item "None" of "Optional[Any]" has no attribute "abort_conn"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit cd97420965b4fb33f2048826c708283b5e116a97)

mgr/volumes: fixup threading.Timer import

Python 3.3 changed `Timer` from a factory function to a class
https://docs.python.org/3.8/library/threading.html#timer-objects

fixes mypy error:

volumes/fs/operations/volume.py:9: error: Module 'threading' has no attribute '_Timer'; maybe "Timer"?

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit a611864dae86fcee9ea9cb9725aa26318396b55e)

mgr/volumes: fix positional str formatting

fixes mypy error:

volumes/fs/operations/group.py: note: In function "create_group":
volumes/fs/operations/group.py:135: error: Not all arguments converted during string formatting

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit f46b5059e635228f07dd5fe593f2d7454c3aa67b)

mgr/volumes: place getters and setters next to each other

workaround for mypy issue:
https://github.com/python/mypy/issues/1465

fixes mypy errors:

volumes/fs/operations/group.py: note: In class "Group":
volumes/fs/operations/group.py:44: error: Name 'uid' already defined on line 36
volumes/fs/operations/group.py:44: error: "Callable[[Group], Any]" has no attribute "setter"
volumes/fs/operations/group.py:48: error: Name 'gid' already defined on line 40
volumes/fs/operations/group.py:48: error: "Callable[[Group], Any]" has no attribute "setter"

volumes/fs/operations/group.py: note: In function "open_group":
volumes/fs/operations/group.py:170: error: Property "uid" defined in "Group" is read-only
volumes/fs/operations/group.py:171: error: Property "gid" defined in "Group" is read-only

volumes/fs/operations/versions/subvolume_base.py: note: In class "SubvolumeBase":
volumes/fs/operations/versions/subvolume_base.py:45: error: Name 'uid' already defined on line 33
volumes/fs/operations/versions/subvolume_base.py:45: error: "Callable[[SubvolumeBase], Any]" has no attribute "setter"
volumes/fs/operations/versions/subvolume_base.py:49: error: Name 'gid' already defined on line 37
volumes/fs/operations/versions/subvolume_base.py:49: error: "Callable[[SubvolumeBase], Any]" has no attribute "setter"
volumes/fs/operations/versions/subvolume_base.py:53: error: Name 'mode' already defined on line 41
volumes/fs/operations/versions/subvolume_base.py:53: error: "Callable[[SubvolumeBase], Any]" has no attribute "setter"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 6a04ecdc9cb76f87a8e2c54a17d590b20037ef3f)

mgr/volumes: reverse params passed to `isinstace()`

fixes mypy error:

volumes/fs/operations/clone_index.py: note: In member "track" of class "CloneIndex":
volumes/fs/operations/clone_index.py:38: error: Argument 2 to "isinstance" has incompatible type "Union[VolumeException, Any]"; expected "Union[type, Tuple[Union[type, Tuple[Any, ...]], ...]]"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 896448324f35bb1c307baff46e67a05d5d23e83b)

mgr/volumes: fix arg typo '.' -> ','

fixes mypy error:

volumes/fs/fs_util.py: note: In function "get_ancestor_xattr":
volumes/fs/fs_util.py:129: error: Too few arguments for "VolumeException"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit e2516a480679090a27de3401410d4f9a85cae8d2)

mgr/volumes: `threading.Event.isSet` no longer exists in py3

fixes mypy errors:

volumes/fs/async_job.py: note: In member "should_cancel" of class "JobThread":
volumes/fs/async_job.py:70: error: "Event" has no attribute "isSet"

volumes/fs/volume.py: note: In member "is_stopping" of class "VolumeClient":
volumes/fs/volume.py:59: error: "Event" has no attribute "isSet"

volumes/fs/volume.py: note: In member "list_fs_volumes" of class "VolumeClient":
volumes/fs/volume.py:106: error: "Event" has no attribute "isSet"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit aaed49a142722089f8a24ceb8b846b7c1c51d5f6)

mgr/volumes: assert JobThread class

fixes mypy error:

volumes/fs/async_job.py: note: In member "run" of class "JobThread":
volumes/fs/async_job.py:43: error: "Thread" has no attribute "should_cancel"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 6907c440e1350e4088c0562bffcc08439633298f)

mgr/volumes: import VolumeException

fixes mypy errors:

volumes/fs/operations/index.py: note: In member "track" of class "Index":
volumes/fs/operations/index.py:19: error: Name 'VolumeException' is not defined

volumes/fs/operations/index.py: note: In member "untrack" of class "Index":
volumes/fs/operations/index.py:22: error: Name 'VolumeException' is not defined

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 5b9d5e2e43e3d2e2d391e9bce2328868a6159e53)

mgr/volumes: use `SafeConfigParser` with py2

Python3.2 renamed `SafeConfigParser` -> `ConfigParser`

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 23efb9240ea6187386677e262dab7f87e1dbac42)

mgr/volumes: use python version during configparser import

fixes mypy error:

volumes/fs/operations/versions/metadata_manager.py:8: error: Name 'configparser' already defined (by an import)

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit f213b646db883e4d9899d5ecb51f7ac591237901)

Merge pull request #33927 from rhcs-dashboard/wip-44574-nautilus

nautilus: mgr/dashboard: Dashboard does not allow you to set norebalance OSD flag

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #33751 from batrick/i44291

nautilus: mds: SIGSEGV in Migrator::export_sessions_flushed

Reviewed-by: Zheng Yan <zyan@redhat.com>

mgr/dashboard: UI fixes
Fixes: https://tracker.ceph.com/issues/42914
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit dd8cb9d2d6b69f57e5f6f3ecd3ca0943411d6638)

mgr/volumes: import errno

fixes mypy errors:

volumes/fs/operations/op_sm.py:36: error: Name 'errno' is not defined
volumes/fs/operations/op_sm.py:39: error: Name 'errno' is not defined
volumes/fs/operations/op_sm.py:46: error: Name 'errno' is not defined
volumes/fs/operations/op_sm.py:49: error: Name 'errno' is not defined
volumes/fs/operations/template.py:5: error: Name 'errno' is not defined
volumes/fs/operations/template.py:14: error: Name 'errno' is not defined
volumes/fs/operations/template.py:23: error: Name 'errno' is not defined
volumes/fs/operations/template.py:32: error: Name 'errno' is not defined
volumes/fs/operations/template.py:42: error: Name 'errno' is not defined
volumes/fs/operations/template.py:45: error: Name 'errno' is not defined
volumes/fs/operations/template.py:62: error: Name 'errno' is not defined
volumes/fs/operations/template.py:74: error: Name 'errno' is not defined
volumes/fs/operations/template.py:85: error: Name 'errno' is not defined
volumes/fs/operations/template.py:94: error: Name 'errno' is not defined
volumes/fs/operations/template.py:103: error: Name 'errno' is not defined
volumes/fs/operations/template.py:112: error: Name 'errno' is not defined
volumes/fs/operations/template.py:121: error: Name 'errno' is not defined
volumes/fs/operations/template.py:130: error: Name 'errno' is not defined
volumes/fs/operations/template.py:139: error: Name 'errno' is not defined
volumes/fs/operations/template.py:148: error: Name 'errno' is not defined
volumes/fs/operations/template.py:158: error: Name 'errno' is not defined
volumes/fs/operations/template.py:169: error: Name 'errno' is not defined
volumes/fs/operations/template.py:180: error: Name 'errno' is not defined
volumes/fs/operations/index.py:18: error: Name 'errno' is not defined
volumes/fs/operations/index.py:21: error: Name 'errno' is not defined

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 8478b72dc3706190993bc2dcf29667805b791e26)

Merge pull request #33722 from jan--f/wip-44331-nautilus

nautilus: ceph-volume: strip _dmcrypt suffix in simple scan json output

ceph-volume: strip _dmcrypt suffix in simple scan json output

LUKS encrypted OSDs name their block* files with a _dmcrypt suffix.
activate fails on json files like this. Stripping this suffix in scan
fixes this.

Fixes: https://tracker.ceph.com/issues/43966
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 2ddf76d118d77659c590ea076d34ce9a8e351a86)

Merge pull request #33391 from jan--f/c-v-nautilus-func-tests

nautilus: ceph-volume: fix nautilus functional tests

rgw/notifications: fix topic action fail with "MethodNotAllowed"

Note that this ommit cannot be cherry-picked from master because it's fixing a
regression introduced by a backport which went into 14.2.8.

Fixes: 33bb1bad74234b913a82bb3773022113b02ffa75
Fixes: http://tracker.ceph.com/issues/44614
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #34008 from smithfarm/wip-44641-nautilus

nautilus: do_cmake.sh: fix application of -DWITH_RADOSGW_KAFKA_ENDPOINT=OFF

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #33877 from ajarr/wip-sync-attrs-nautilus

nautilus: mgr/volumes: synchronize ownership (for symlinks) and inode timestamps for cloned subvolumes

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #33820 from rzarzynski/wip-43808-nautilus

nautilus: msgr: backport the EventCenter-related fixes

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33834 from tchaikov/nautilus-44524

nautilus: mgr: update "hostname" when we already have the daemon state from that entity

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33716 from tchaikov/nautilus-43747

nautilus: cmake: Improve test for 16-byte atomic support on IBM Z

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #33234 from rhcs-dashboard/wip-44095-nautilus

nautilus: mgr/dashboard: Use booleanText pipe

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #33749 from cbodley/wip-44444

nautilus: rgw: making implicit_tenants backwards compatible.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33503 from sseshasa/wip-44259-nautilus

nautilus: osd/OSD: Log slow ops/types to cluster logs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge pull request #33295 from mchangir/wip-42441-nautilus

nautilus: mds: add configurable snapshot limit

Reviewed-by: Raman Raja <rraja@redhat.com>

do_cmake.sh: fix application of -DWITH_RADOSGW_KAFKA_ENDPOINT=OFF

50ba25b2aa4db34319a7b95e789065540347f895 broke do_cmake.sh in nautilus
and the backport of 46b3d671b4408a766aaed379274242ad4b38afb3 (which
was included in the same PR) did not fix it enough.

This commit cannot be backported from master. It is needed to fix
a partially broken backport.

After this commit is applied, the relevant part of do_cmake.sh in
nautilus will look exactly like it does now in master.

Fixes: 50ba25b2aa4db34319a7b95e789065540347f895
Fixes: https://tracker.ceph.com/issues/44641
Signed-off-by: Nathan Cutler <ncutler@suse.com>

do_cmake.sh: set WITH_RADOSGW_AMQP_ENDPOINT using ARGS

so we can extend ARGS in a more unified way

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 612376a74964c79af5b4029c650c8add5fd50120)

Conflicts:
do_cmake.sh
- nautilus does not have 9a47b328b80e5e502bd67cb94f925efe0c53c755

qa: get rid of iterkeys for py3 compatibility

Fixes: https://tracker.ceph.com/issues/42287
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 5f95b532aa26249cecea4586166399a0bf10ad47)

Conflicts:
qa/tasks/barbican.py
qa/tasks/keystone.py
qa/tasks/vault.py

- ensured that all instances of iterkeys() under qa/ in luminous are changed to keys()
- removed barbican.py and vault.py; install_packages
method per PR comments.

mgr/volumes: type convert uid and gid to int

This fix is only needed in nautilus. The issue
was observed during upstream teuthology testing in CentOS 7.5
machines.

  File "/usr/share/ceph/mgr/volumes/fs/async_cloner.py", line 153, in do_clone
    bulk_copy(fs_handle, src_path, dst_path, should_cancel)
  File "/usr/share/ceph/mgr/volumes/fs/async_cloner.py", line 144, in bulk_copy
    cptree(source_path, dst_path)
  File "/usr/share/ceph/mgr/volumes/fs/async_cloner.py", line 134, in cptree
    sync_attrs(fs_handle, d_full_dst, stx)
  File "/usr/share/ceph/mgr/volumes/fs/async_cloner.py", line 77, in sync_attrs
    fs_handle.lchown(target_path, source_statx["uid"], source_statx["gid"])

The issue wasn't observed in master/octopus teuthology
testing.

Signed-off-by: Ramana Raja <rraja@redhat.com>

qa: allow keeping sudo in command arguments

sudo cannot be omitted from the given command's arguments, when running
passwd, chown and, specially, when sudo is used for running the given
command as different user.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 3e0a1361f78452d79c283f70b5dc2b865d62cb61)

qa: make execution abortion optional for command ran through run_shell()

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 5df2b82c4f9ce71cd82889531333be184c84707c)

test: add test_statx for new statx()

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>
(cherry picked from commit ea8527b5e8a9695050275a6715ebce2798579de2)