mgr/rgw: use entity ids instead of names when possible

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: don't specify default realm/zg/zone

Access entities explicitly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: apply rgw through orchestrator interface

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: add realm create zone-creds command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: add zone create command (based on rgwam)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: add realm bootstrap command based on rgwam

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/rgw: initial work

Basic catch all command that passes through params to radosgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: can pass keyring param

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: can pass ceph user to radosgw/radosgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: modify command line parsing to support generic attributes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: optionally run radosgw

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: info logging

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: code cleanups

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: realm new-zone-creds

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: rename file

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: zone create

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: create users, zone run command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: more work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgwam: rgw admin for multisite tool

initial work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #43836 from ronen-fr/wip-rf-atomic-enums

crimson/common: disable arithmetic operators for atomic enums

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43525 from ceph/wip-yuriw-release-15.2.15-master

doc: 15.2.15 Release Notes

Merge pull request #42479 from ronen-fr/wip-ronenf-scrub-prefix

osd/scrub: remove Scrubber sub-objects reliance on PG::gen_prefix()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #43730 from nmshelke/newline-vstart-help

vstart.sh: print newline character after vstart.sh help text

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43414 from joscollin/wip-update-get-involved-cephfs

doc: update get-involved for cephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

vstart.sh: print newline character after vstart.sh help text

Without this newline character the help text messes up the
expected position of the prompt string.

Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>

doc: update dev list and kernel client

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc: update #cephfs channel

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #43835 from xxhdx1985126/wip-segment-avail-bytes

crimson/os/seastore/segment_cleaner: initialize segments' avail_bytes…

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43754 from cyx1231st/wip-seastore-fix-journal-committed-to

crimson/os/seastore: fix ordered updates to JournalSegmentManager::committed_to

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge PR #43827 into master

* refs/pull/43827/head:
qa/suites/orch/cephadm: add repave-all test case
mgr/cephadm/services/osd: less noisy
mgr/cephadm/services/osd: do not log ok-to-stop/safe-to-destroy failures
mgr/orchestrator: clean up 'orch osd rm status'

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #43699 from sebastian-philipp/qa-rados-mgr-random-objectstore

qa/suites/rados/mgr: use only one objectstore instead of all

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43621 from ifed01/wip-ifed-fix-53011

os/bluestore: use proper prefix when removing undecodable Share Blob.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #43635 from adk3798/agent-responsiveness

mgr/cephadm: improve agent responsiveness

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43574 from sabzco/ceph-volume-fix

ceph-volume: fix a typo causing AttributeError

Merge pull request #43679 from guits/cv_quick_update_tests

ceph-volume/tests: update setup_mixed_type playbook

crimson/common: disable arithmetic operators for atomic enums

While class enums are 'integral types', they do not support
all that an int does.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os/seastore/segment_cleaner: initialize segments' avail_bytes with segments' sizes

Currently, we initialize segments' avail_bytes with "segment_size * num_segments". Both segment_size
and num_segments are 32 bits long, multiplying them would lead to overflow.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge PR #43826 into master

* refs/pull/43826/head:
mgr/cephadm: allow zapping devices from other clusters

Reviewed-by: Adam King <adking@redhat.com>

qa/suites/orch/cephadm: add repave-all test case

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/services/osd: less noisy

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/services/osd: do not log ok-to-stop/safe-to-destroy failures

These failures are normal and expected; they should not pollute the log.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: allow zapping devices from other clusters

This is the 99% of the devices that ever get zapped.

Fixes: b7782084ac9657be9b2da6ebd56b5029cf859225
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43814 from neha-ojha/wip-more-cv

qa/suites/upgrade/octopus-x/stress-split-no-cephadm: exclude ceph-volume

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/orchestrator: clean up 'orch osd rm status'

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43808 from cbodley/wip-qa-rgw-java-master

qa/rgw: master branch targets ceph-master branch of java_s3tests

Merge pull request #43807 from sebastian-philipp/osd_memory_target_autotune-true

doc/cephadm: Recommend osd_memory_target_autotune

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@newdream.net>

doc/cephadm: Recommend osd_memory_target_autotune

In case the cluster runs on hardware that is used exclusively for
Ceph, let's recommend `osd_memory_target_autotune`

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

qa/suites/upgrade/octopus-x/stress-split-no-cephadm: exclude ceph-volume

To address failures like

```
Command failed on smithi096 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.15-11-g5f8f263c-1focal ceph-mds=15.2.15-11-g5f8f263c-1focal ceph-mgr=15.2.15-11-g5f8f263c-1focal ceph-common=15.2.15-11-g5f8f263c-1focal ceph-fuse=15.2.15-11-g5f8f263c-1focal ceph-test=15.2.15-11-g5f8f263c-1focal ceph-volume=15.2.15-11-g5f8f263c-1focal radosgw=15.2.15-11-g5f8f263c-1focal python3-rados=15.2.15-11-g5f8f263c-1focal python3-rgw=15.2.15-11-g5f8f263c-1focal python3-cephfs=15.2.15-11-g5f8f263c-1focal python3-rbd=15.2.15-11-g5f8f263c-1focal libcephfs2=15.2.15-11-g5f8f263c-1focal librados2=15.2.15-11-g5f8f263c-1focal librbd1=15.2.15-11-g5f8f263c-1focal rbd-fuse=15.2.15-11-g5f8f263c-1focal'
```

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43406 from ljflores/wip-telemetry-perf-improvements

mgr/telemetry: add mempool stats to telemetry perf report

Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

Merge PR #42727 into master

* refs/pull/42727/head:
mgr/orchestrator: improve usage string for 'orch daemon add osd'
ceph-volume: activate: try simple mode too
mgr/cephadm: identify and instantiate raw osds post-create
mgr/orchestrator: accept --method arg to 'orch daemon add osd'
python-common: drivegroup: add 'method' property
cephadm: use generic ceph-volume activate
ceph-volume: top-level 'activate' command
ceph-volume: lvm activate: add --no-tmpfs
ceph-volume: lvm activate: infer bluestore or filestore
ceph-volume: raw activate: accept --osd-id and/or --osd-uuid instead of device

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #43705 from tchaikov/wip-no-more-python2

mgr: do not handle Python2

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #43700 from liewegas/fix-24990

ceph_test_rados_api_watch_notify: extend Watch3Timeout test

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43664 from NUABO/tanchangzhi

osd: fix 'ceph osd stop <osd.nnn>' doesn't take effect

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #43752 into master

* refs/pull/43752/head:
client: remove usless _openat()
client: remove optional for dirfd parameter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #43666 into master

* refs/pull/43666/head:
qa/vstart_runner: add "managers" to LocalContext instances

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #43638 into master

* refs/pull/43638/head:
qa: pass subdir arg when executing workunit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #43613 into master

* refs/pull/43613/head:
qa: lengthen health warning wait

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #41667 into master

* refs/pull/41667/head:
mds: do not trim cache when creating system file
mds: fix the comment in add_inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #43611 into master

* refs/pull/43611/head:
doc/mgr/nfs: document rgw user and bucket exports
PendingReleaseNotes: add note about nfs CLI change(s)
qa/suites/orch/cephadm/smoke-roleless: add rgw user nfs export case
mgr/nfs: take user-id and/or bucket for 'nfs export create rgw'
mgr/nfs: reorder 'nfs export creage rgw' arguments
mgr/nfs: reorder 'nfs export create cephfs' arguments
mgr/nfs: use keyword args for 'nfs export create rgw'
mgr/nfs: document and use keyword args for 'nfs export create cephfs'
qa/tasks/cephfs/test_nfs: use keyword args
pybind/ceph_argparse: handle misordered keyword arguments

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #43768 from Huber-ming/admin_mdlog_fetch

radosgw-admin: supplement help documents with 'mdlog autotrim'

Merge pull request #43778 from adamemerson/wip-53132

rgw: Ensure buckets too old to decode a layout have layout logs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: master branch targets ceph-master branch of java_s3tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37184 from ybwang0211/KMSMSMSMS_return_error_message

rgw:When KMS encryption is used and the key does not exist, we should…

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #43725 from rhcs-dashboard/nfs-export-form-fix

mgr/dashboard: NFS 'create export' form: fixes

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

osd/scrub: remove reliance of Scrubber objects' logging on the PG

Modify the Scrubber's sub-objects to use their own gen_prefix()
functions, instead of using PG::gen_prefix().

Fixes: https://tracker.ceph.com/issues/51843
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

doc/mgr/nfs: document rgw user and bucket exports

Signed-off-by: Sage Weil <sage@newdream.net>

PendingReleaseNotes: add note about nfs CLI change(s)

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/orch/cephadm/smoke-roleless: add rgw user nfs export case

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: take user-id and/or bucket for 'nfs export create rgw'

- move the bucket / user position after the cluster_id and pseudo_path
(since they are optional)
- require bucket or user or both
- if bucket, use the bucket owner
- if bucket+user, use that user
- if user only, then export at top-level (all users buckets)

Fixes: https://tracker.ceph.com/issues/53134
Signed-off-by: Sage Weil <sage@newdream.net>

mgr/orchestrator: improve usage string for 'orch daemon add osd'

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/dashboard: NFS 'create export' form: fixes

* Do not allow a pseudo that is already in use by another export.
* Create mode form: prefill dropdown selectors if options > 0.
* Edit mode form: do not reset the field values that depend on other values that are being edited (unlike Create mode).
* Fix broken link: cluster service.
* Fix error message style for non-existent cephfs path.
* nfs-service.ts: lsDir: thow error if volume is not provided.
* File renaming: nfsganesha.py => nfs.py; test_ganesha.py => test_nfs.py

Fixes: https://tracker.ceph.com/issues/53083
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #43737 from AndrewSharapov/master

mgr/cephadm: Fixed spawning ip addresses list for public network interface

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43797 from rhcs-dashboard/fix-53144-master

mgr/dashboard: fix missing alert rule details

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43461 from CongMinYin/fix-flush-advance

librbd/cache/pwl: fix external flush dispatch in advance

Reviewed-by: Mykola Golub <mgolub@suse.com>

crimson/os/seastore: fix ordered updates to JournalSegmentManager::committed_to

Journal segment should not update committed_to during rolling as there
might be still pending writes from the previous segment.

A side-effect here is that committed_to now needs to include
segment_seq_t to point to a previous segment.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43617 from cyx1231st/wip-seastore-batch-journal-records

crimson/os/seastore/journal: support both batching and concurrent writes

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43781 from liu-chunmei/osd_uuid_zero

crimson/osd: randomize the osd_uuid if not specified

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: randomize the osd_uuid if not specified

address the failure spotted in teuthology based test:
sudo ceph --cluster ceph osd new 00000000-0000-0000-0000-000000000000 0

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

radosgw-admin: supplement help documents with 'mdlog autotrim'

Signed-off-by: Huber-ming <zhangsm01@inspur.com>

Merge pull request #43456 from ljflores/wip-separate-data

mgr/telemetry: provide option for separated data in the telemetry perf channel

Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #43591 from cbodley/wip-52976

radosgw-admin: allow 'bi purge' to delete index if entrypoint doesn't exist

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #43710 from cbodley/wip-53003

rgw: fix self-comparison for RGWCopyObj optimization

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #43715 from cbodley/wip-52716

rgw: ListMultipartUploads returns the real upload Owners

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #43779 from cbodley/wip-47527

rgw: fix ListBucketMultiparts response with common prefixes

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

rgw: Ensure buckets too old to decode a layout have layout logs

When decoding `RGWBucketInfo` data from before Pacific, we won't call
`rgw::BucketLayout::decode`, but will instead synthesize the layout
information. This leaves the `rgw::BucketLayout::logs` empty, as the
fallback to populate it only applies to old versions of
`rgw::BucketLayout`.

Add a check at the end of `RGWBUcketInfo::decode` to populate it if
empty.

Fixes: https://tracker.ceph.com/issues/53132
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #42950 from CongMinYin/fix-dead-lock-during-shutdown

librbd/cache/pwl/ssd: fix dead lock and assert during shutdown

Reviewed-by: Mykola Golub <mykola.golub@clyso.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

mgr/dashboard: fix missing alert rule details

Fixes: https://tracker.ceph.com/issues/53144
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #43753 from soumyakoduri/wip-skoduri-dblock

rgw/dbstore: No need for explicit LOCK in DBStore

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #43790 from sebastian-philipp/doc-cephadm-purge

doc/cephadm: purge

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43659 from majianpeng/send-internal-flush-for-rbd-copy

librbd: send FLUSH_SOURCE_INTERNAL when do copy/deep_copy.

Reviewed-by: Mykola Golub <mykola.golub@clyso.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

doc/cephadm: purge

Fixes: https://tracker.ceph.com/issues/50534
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge PR #43786 into master

* refs/pull/43786/head:
mds: fix typo in MDSRank.cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #43690 from rhcs-dashboard/improve-error-handling-get-facts-backend

mgr/dashboard: improve error handling for gather_facts

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: python unit tests refactoring

* Controller tests: cherrypy config: authentication disabled by default; ability to pass custom config (e.g. enable authentication).
* Auth controller: add tests; test that unauthorized request fails when authentication is enabled.
* DocsTest: clear ENDPOINT_MAP so the test_gen_tags test becomes deterministic.
* pylint: disable=no-name-in-module: fix imports in tests.

Fixes: https://tracker.ceph.com/issues/53083
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

crimson/os/seastore/journal: measure io-depth and batching

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/journal: implement RecordSubmitter and RecordBatch

To be able to batch records for write, meanwhile, still allows
concurrent writes.

The current change doesn't seem to impact write performance either to
prefer batching or concurrent writes, so set the io_depth to 2 for
demonstration.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

mds: fix typo in MDSRank.cc

Signed-off-by: Yongseok Oh <yongseok.oh@linecorp.com>

Merge pull request #43750 from zdover23/wip-doc-2021-10-30-omap-format-conversion-data-corruption-bug-admonition

doc: add admonition for tracker 53062

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #43769 from ifed01/wip-ifed-omap-upgrade-fix-notes

PendingReleaseNotes: document OMAP upgrade bug.

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/nfs: reorder 'nfs export creage rgw' arguments

Put bucket name last so that it paves the way for an optional 'user' arg
to go along with it.

Signed-off-by: Sage Weil <sage@newdream.net>