python-common/rgw: fix style issues

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc/rgw: fix docs build

Workaround rgw modules conflict, as there are two separate modules named
rgw: src/pybind/rgw, src/pybind/mgr/rgw

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: ignore mypy errors

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: use tool_exec instead of directly spawning commands

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

docs: document mgr/rgw module

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: change zone-creds cli commands

 - ceph rgw zone-creds create
 - ceph rgw zone-creds remove

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: reorganize code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: start_radosgw is true by default

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: pass realm_id via token

Add realm_id to token.
Also, use realm_id from token instead of requiring realm_name
for command that uses realm token.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: realm remove zone-creds

add command to remove zone creds. Either removes the access key
or the entire user if was the only access key for that user.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

pybind/argparse: handle None sequence

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: add standalone cli util

for external testing

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: add realm reconcile command

reconcile endpoints defined in orchestrator vs realm period, and optionally
update realm period

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: use entity ids instead of names when possible

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: don't specify default realm/zg/zone

Access entities explicitly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: apply rgw through orchestrator interface

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: add realm create zone-creds command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: add zone create command (based on rgwam)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: add realm bootstrap command based on rgwam

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: initial work

Basic catch all command that passes through params to radosgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: can pass keyring param

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: can pass ceph user to radosgw/radosgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: modify command line parsing to support generic attributes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: optionally run radosgw

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: info logging

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: code cleanups

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: realm new-zone-creds

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: rename file

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: zone create

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: create users, zone run command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: more work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: rgw admin for multisite tool

initial work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #43836 from ronen-fr/wip-rf-atomic-enums

crimson/common: disable arithmetic operators for atomic enums

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43525 from ceph/wip-yuriw-release-15.2.15-master

doc: 15.2.15 Release Notes

Merge pull request #42479 from ronen-fr/wip-ronenf-scrub-prefix

osd/scrub: remove Scrubber sub-objects reliance on PG::gen_prefix()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #43730 from nmshelke/newline-vstart-help

vstart.sh: print newline character after vstart.sh help text

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43414 from joscollin/wip-update-get-involved-cephfs

doc: update get-involved for cephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

vstart.sh: print newline character after vstart.sh help text

Without this newline character the help text messes up the
expected position of the prompt string.

Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>

doc: update dev list and kernel client

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc: update #cephfs channel

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #43835 from xxhdx1985126/wip-segment-avail-bytes

crimson/os/seastore/segment_cleaner: initialize segments' avail_bytes…

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43754 from cyx1231st/wip-seastore-fix-journal-committed-to

crimson/os/seastore: fix ordered updates to JournalSegmentManager::committed_to

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge PR #43827 into master

* refs/pull/43827/head:
qa/suites/orch/cephadm: add repave-all test case
mgr/cephadm/services/osd: less noisy
mgr/cephadm/services/osd: do not log ok-to-stop/safe-to-destroy failures
mgr/orchestrator: clean up 'orch osd rm status'

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #43699 from sebastian-philipp/qa-rados-mgr-random-objectstore

qa/suites/rados/mgr: use only one objectstore instead of all

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43621 from ifed01/wip-ifed-fix-53011

os/bluestore: use proper prefix when removing undecodable Share Blob.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #43635 from adk3798/agent-responsiveness

mgr/cephadm: improve agent responsiveness

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43574 from sabzco/ceph-volume-fix

ceph-volume: fix a typo causing AttributeError

Merge pull request #43679 from guits/cv_quick_update_tests

ceph-volume/tests: update setup_mixed_type playbook

crimson/common: disable arithmetic operators for atomic enums

While class enums are 'integral types', they do not support
all that an int does.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os/seastore/segment_cleaner: initialize segments' avail_bytes with segments' sizes

Currently, we initialize segments' avail_bytes with "segment_size * num_segments". Both segment_size
and num_segments are 32 bits long, multiplying them would lead to overflow.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge PR #43826 into master

* refs/pull/43826/head:
mgr/cephadm: allow zapping devices from other clusters

Reviewed-by: Adam King <adking@redhat.com>

qa/suites/orch/cephadm: add repave-all test case

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/services/osd: less noisy

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/services/osd: do not log ok-to-stop/safe-to-destroy failures

These failures are normal and expected; they should not pollute the log.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: allow zapping devices from other clusters

This is the 99% of the devices that ever get zapped.

Fixes: b7782084ac9657be9b2da6ebd56b5029cf859225
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43814 from neha-ojha/wip-more-cv

qa/suites/upgrade/octopus-x/stress-split-no-cephadm: exclude ceph-volume

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/orchestrator: clean up 'orch osd rm status'

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43808 from cbodley/wip-qa-rgw-java-master

qa/rgw: master branch targets ceph-master branch of java_s3tests

Merge pull request #43807 from sebastian-philipp/osd_memory_target_autotune-true

doc/cephadm: Recommend osd_memory_target_autotune

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@newdream.net>

doc/cephadm: Recommend osd_memory_target_autotune

In case the cluster runs on hardware that is used exclusively for
Ceph, let's recommend `osd_memory_target_autotune`

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

qa/suites/upgrade/octopus-x/stress-split-no-cephadm: exclude ceph-volume

To address failures like

```
Command failed on smithi096 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.15-11-g5f8f263c-1focal ceph-mds=15.2.15-11-g5f8f263c-1focal ceph-mgr=15.2.15-11-g5f8f263c-1focal ceph-common=15.2.15-11-g5f8f263c-1focal ceph-fuse=15.2.15-11-g5f8f263c-1focal ceph-test=15.2.15-11-g5f8f263c-1focal ceph-volume=15.2.15-11-g5f8f263c-1focal radosgw=15.2.15-11-g5f8f263c-1focal python3-rados=15.2.15-11-g5f8f263c-1focal python3-rgw=15.2.15-11-g5f8f263c-1focal python3-cephfs=15.2.15-11-g5f8f263c-1focal python3-rbd=15.2.15-11-g5f8f263c-1focal libcephfs2=15.2.15-11-g5f8f263c-1focal librados2=15.2.15-11-g5f8f263c-1focal librbd1=15.2.15-11-g5f8f263c-1focal rbd-fuse=15.2.15-11-g5f8f263c-1focal'
```

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43406 from ljflores/wip-telemetry-perf-improvements

mgr/telemetry: add mempool stats to telemetry perf report

Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

Merge PR #42727 into master

* refs/pull/42727/head:
mgr/orchestrator: improve usage string for 'orch daemon add osd'
ceph-volume: activate: try simple mode too
mgr/cephadm: identify and instantiate raw osds post-create
mgr/orchestrator: accept --method arg to 'orch daemon add osd'
python-common: drivegroup: add 'method' property
cephadm: use generic ceph-volume activate
ceph-volume: top-level 'activate' command
ceph-volume: lvm activate: add --no-tmpfs
ceph-volume: lvm activate: infer bluestore or filestore
ceph-volume: raw activate: accept --osd-id and/or --osd-uuid instead of device

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #43705 from tchaikov/wip-no-more-python2

mgr: do not handle Python2

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #43700 from liewegas/fix-24990

ceph_test_rados_api_watch_notify: extend Watch3Timeout test

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43664 from NUABO/tanchangzhi

osd: fix 'ceph osd stop <osd.nnn>' doesn't take effect

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #43752 into master

* refs/pull/43752/head:
client: remove usless _openat()
client: remove optional for dirfd parameter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #43666 into master

* refs/pull/43666/head:
qa/vstart_runner: add "managers" to LocalContext instances

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #43638 into master

* refs/pull/43638/head:
qa: pass subdir arg when executing workunit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #43613 into master

* refs/pull/43613/head:
qa: lengthen health warning wait

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #41667 into master

* refs/pull/41667/head:
mds: do not trim cache when creating system file
mds: fix the comment in add_inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #43611 into master

* refs/pull/43611/head:
doc/mgr/nfs: document rgw user and bucket exports
PendingReleaseNotes: add note about nfs CLI change(s)
qa/suites/orch/cephadm/smoke-roleless: add rgw user nfs export case
mgr/nfs: take user-id and/or bucket for 'nfs export create rgw'
mgr/nfs: reorder 'nfs export creage rgw' arguments
mgr/nfs: reorder 'nfs export create cephfs' arguments
mgr/nfs: use keyword args for 'nfs export create rgw'
mgr/nfs: document and use keyword args for 'nfs export create cephfs'
qa/tasks/cephfs/test_nfs: use keyword args
pybind/ceph_argparse: handle misordered keyword arguments

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #43768 from Huber-ming/admin_mdlog_fetch

radosgw-admin: supplement help documents with 'mdlog autotrim'

Merge pull request #43778 from adamemerson/wip-53132

rgw: Ensure buckets too old to decode a layout have layout logs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: master branch targets ceph-master branch of java_s3tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37184 from ybwang0211/KMSMSMSMS_return_error_message

rgw:When KMS encryption is used and the key does not exist, we should…

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #43725 from rhcs-dashboard/nfs-export-form-fix

mgr/dashboard: NFS 'create export' form: fixes

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

osd/scrub: remove reliance of Scrubber objects' logging on the PG

Modify the Scrubber's sub-objects to use their own gen_prefix()
functions, instead of using PG::gen_prefix().

Fixes: https://tracker.ceph.com/issues/51843
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

doc/mgr/nfs: document rgw user and bucket exports

Signed-off-by: Sage Weil <sage@newdream.net>

PendingReleaseNotes: add note about nfs CLI change(s)

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/orch/cephadm/smoke-roleless: add rgw user nfs export case

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: take user-id and/or bucket for 'nfs export create rgw'

- move the bucket / user position after the cluster_id and pseudo_path
(since they are optional)
- require bucket or user or both
- if bucket, use the bucket owner
- if bucket+user, use that user
- if user only, then export at top-level (all users buckets)

Fixes: https://tracker.ceph.com/issues/53134
Signed-off-by: Sage Weil <sage@newdream.net>

mgr/orchestrator: improve usage string for 'orch daemon add osd'

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/dashboard: NFS 'create export' form: fixes

* Do not allow a pseudo that is already in use by another export.
* Create mode form: prefill dropdown selectors if options > 0.
* Edit mode form: do not reset the field values that depend on other values that are being edited (unlike Create mode).
* Fix broken link: cluster service.
* Fix error message style for non-existent cephfs path.
* nfs-service.ts: lsDir: thow error if volume is not provided.
* File renaming: nfsganesha.py => nfs.py; test_ganesha.py => test_nfs.py

Fixes: https://tracker.ceph.com/issues/53083
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #43737 from AndrewSharapov/master

mgr/cephadm: Fixed spawning ip addresses list for public network interface

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43797 from rhcs-dashboard/fix-53144-master

mgr/dashboard: fix missing alert rule details

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43461 from CongMinYin/fix-flush-advance

librbd/cache/pwl: fix external flush dispatch in advance

Reviewed-by: Mykola Golub <mgolub@suse.com>

crimson/os/seastore: fix ordered updates to JournalSegmentManager::committed_to

Journal segment should not update committed_to during rolling as there
might be still pending writes from the previous segment.

A side-effect here is that committed_to now needs to include
segment_seq_t to point to a previous segment.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43617 from cyx1231st/wip-seastore-batch-journal-records

crimson/os/seastore/journal: support both batching and concurrent writes

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43781 from liu-chunmei/osd_uuid_zero

crimson/osd: randomize the osd_uuid if not specified

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: randomize the osd_uuid if not specified

address the failure spotted in teuthology based test:
sudo ceph --cluster ceph osd new 00000000-0000-0000-0000-000000000000 0

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

radosgw-admin: supplement help documents with 'mdlog autotrim'

Signed-off-by: Huber-ming <zhangsm01@inspur.com>

Merge pull request #43456 from ljflores/wip-separate-data

mgr/telemetry: provide option for separated data in the telemetry perf channel

Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #43591 from cbodley/wip-52976

radosgw-admin: allow 'bi purge' to delete index if entrypoint doesn't exist

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #43710 from cbodley/wip-53003

rgw: fix self-comparison for RGWCopyObj optimization

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #43715 from cbodley/wip-52716

rgw: ListMultipartUploads returns the real upload Owners

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #43779 from cbodley/wip-47527

rgw: fix ListBucketMultiparts response with common prefixes

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

rgw: Ensure buckets too old to decode a layout have layout logs

When decoding `RGWBucketInfo` data from before Pacific, we won't call
`rgw::BucketLayout::decode`, but will instead synthesize the layout
information. This leaves the `rgw::BucketLayout::logs` empty, as the
fallback to populate it only applies to old versions of
`rgw::BucketLayout`.

Add a check at the end of `RGWBUcketInfo::decode` to populate it if
empty.

Fixes: https://tracker.ceph.com/issues/53132
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #42950 from CongMinYin/fix-dead-lock-during-shutdown

librbd/cache/pwl/ssd: fix dead lock and assert during shutdown

Reviewed-by: Mykola Golub <mykola.golub@clyso.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

mgr/dashboard: fix missing alert rule details

Fixes: https://tracker.ceph.com/issues/53144
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>