Merge pull request #43411 from ljflores/wip-mgr-command-cleanup

mon: simplify 'mgr module ls' output

Merge PR #43767 into master

* refs/pull/43767/head:
qa: increase the timeout value to wait a litte longer

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #42520 into master

* refs/pull/42520/head:
test: add cephfs-mirror HA active/active workunit and test yamls
test: add cephfs_mirror thrasher
tasks/cephfs_mirror: optionally run in foreground
mgr/mirroring: throttle directory reassigment to mirror daemons

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #43775 from liewegas/wip-mgr-rook-osd-creation

mgr/rook: persist drive groups

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43746 from kamoltat/wip-fix-autoscale-typo

pybind/mgr/pg_autoscaler: typo default option scale-up to scale-down

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43847 from cbodley/wip-53095

qa/rgw: bump tempest version to resolve dependency issue

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #43536 from guits/lvm-wrapper

cephadm/ceph-volume: do not use lvm binary in containers

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43712 from adk3798/endpoint-port

mgr/cephadm: fix port handling for cephadm endpoint

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43789 from sebastian-philipp/could-not-locate-podmanb

cephadm: Avoid "Could not locate podman: podman not found"

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43825 from Daniel-Pivonka/cephadm_upgrade_name_already_inuse

cephadm: fix upgrade name already in use

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #43853 from cyx1231st/wip-seastore-fix-journal-write-boundary

crimson/os/seastore: fix journal updates to the write boundaries

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #43290 into master

* refs/pull/43290/head:
.github/pull_request_template: address review comments
.github/pull_request_template.md: update
.github/workflows/pr-checklist: add checklist action

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #43821 into master

* refs/pull/43821/head:
mgr/cephadm: allow osd spec removal
mgr/orchestrator: pass 'force' flag down for remove_service

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #43527 from sskaur/pybind-rados-omap-cmp

pybind: add wrapper for rados_write_op_omap_cmp

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43403 from ronen-fr/wip-rf-scrub-deep-v2

osd/scrub: expose PGs scrubbing schedule to the operator

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #43677 from majianpeng/remove-larger-debug-message

librbd/pwl: don't need print cache_bl contents.

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #43585 from anthonyeleven/anthonyeleven/fix-with-legacy

common/options: fix typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43844 from dang/wip-dang-zipper-doc

RGW Zipper - Cleanup and API doc pass

mgr/cephadm: allow osd spec removal

OSD specs/drivegroups are essentially templates for OSD creation but do
not map to the full lifecycle of the OSDs that they create.  When a spec
is removed, remove it immediately.

If no --force is provided, the error lists which OSDs will be left behind.
If --force is passed, the service is removed.

This leaves behind a few oddities:

- When you list services, OSDs that were created by the drivegroup may
  still exist, causing the drivegroup to appear in the list as
  unmanaged services.
- If you create a new drivegroup with the same name, the prior OSDs will
  appear to belong to the new spec instance, regardless of whether the
  spec/drivegroup parameters are the same.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/orchestrator: pass 'force' flag down for remove_service

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43836 from ronen-fr/wip-rf-atomic-enums

crimson/common: disable arithmetic operators for atomic enums

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

pybind/mgr/pg_autoscaler: typo default option scale-up to scale-down

Typo: `scale-up` should be `scale-down` in Module
Option.

This typo doesn't trigger a bug because we create
a key-value of `scale-down` profile in
the function `create_initial()` in `src/mon/KVMonitor.cc`.
This will override whatever is the default option
in pg_autoscaler/module.py when we start the cluster and
the monitor gets created.

The command: `ceph osd pool set autoscale-profile <option>`
is still the primary command to change the autoscale-profiler
after the pool is created.

Fixes: https://tracker.ceph.com/issues/53203
Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #43525 from ceph/wip-yuriw-release-15.2.15-master

doc: 15.2.15 Release Notes

RGW Zipper - Initial round of API docs

This is the initial round of API docs for Zipper, in the form of Doxygen
comments.  It's not detailed document, but it's a start.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

RGW Zipper - API Cleanups

During the documentation pass for the Zipper API, a number of cleanups
were found: APIs that should be slightly different, or that were unused
entirely.  This is a rollup commit of all those cleanups.

- move get_multipart_upload() to Bucket
- remove unused defer_gc
- move create_bucket() into User
- rename get_bucket_info() to load_bucket() to match load_user()
- Remove read_bucket_stats()
  The codepaths using read_bucket_stats() used CLS data types, and the
  function is confusingly named.  Load the ent in load_bucket(), and use
  an alternative data structure to get size stats for the bucket.
- rename get_bucket_stats to read_stats
- Remove remove_metadata() from API
- remove copy_obj_data() from API
- rename get_obj_layout to dump_obj_layout
- use SAL range_to_ofs

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42479 from ronen-fr/wip-ronenf-scrub-prefix

osd/scrub: remove Scrubber sub-objects reliance on PG::gen_prefix()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #43730 from nmshelke/newline-vstart-help

vstart.sh: print newline character after vstart.sh help text

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43414 from joscollin/wip-update-get-involved-cephfs

doc: update get-involved for cephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

vstart.sh: print newline character after vstart.sh help text

Without this newline character the help text messes up the
expected position of the prompt string.

Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>

doc: update dev list and kernel client

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc: update #cephfs channel

Signed-off-by: Jos Collin <jcollin@redhat.com>

test: add cephfs-mirror HA active/active workunit and test yamls

Fixes: http://tracker.ceph.com/issues/50372
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add cephfs_mirror thrasher

Signed-off-by: Venky Shankar <vshankar@redhat.com>

tasks/cephfs_mirror: optionally run in foreground

cephfs mirror damon thrasher needs to send SIGTERM to mirror
daemons. The mirror daemon needs to run in foreground for
it to receive signal via `daemon.signal`.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/mirroring: throttle directory reassigment to mirror daemons

This is to avoid over-shuffling directories when lots of mirror
daemons come and go.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/journal: fix updates to journal seq head and target

* update journal sequences at the write boundary;
* update journal head to the sequence of write end;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/journal: mark committed_to at the write boundary

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43835 from xxhdx1985126/wip-segment-avail-bytes

crimson/os/seastore/segment_cleaner: initialize segments' avail_bytes…

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43754 from cyx1231st/wip-seastore-fix-journal-committed-to

crimson/os/seastore: fix ordered updates to JournalSegmentManager::committed_to

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge PR #43827 into master

* refs/pull/43827/head:
qa/suites/orch/cephadm: add repave-all test case
mgr/cephadm/services/osd: less noisy
mgr/cephadm/services/osd: do not log ok-to-stop/safe-to-destroy failures
mgr/orchestrator: clean up 'orch osd rm status'

Reviewed-by: Adam King <adking@redhat.com>

mgr/rook: make 'osd' unmanaged service appear as unmanaged

This makes it match the cephadm result, where e.g., the placement
shows as '<unmanaged>' in 'orch ls'.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/rook: implement removal of drive groups

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/rook: include drivegroups in 'orch ls' result

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/rook/rook_cluster: ignore pods with no containers

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/rook: fix service_type filtering vs crash

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43699 from sebastian-philipp/qa-rados-mgr-random-objectstore

qa/suites/rados/mgr: use only one objectstore instead of all

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43621 from ifed01/wip-ifed-fix-53011

os/bluestore: use proper prefix when removing undecodable Share Blob.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #43635 from adk3798/agent-responsiveness

mgr/cephadm: improve agent responsiveness

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

qa/rgw: bump tempest version to resolve dependency issue

Fixes: https://tracker.ceph.com/issues/53095
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43574 from sabzco/ceph-volume-fix

ceph-volume: fix a typo causing AttributeError

Merge pull request #43679 from guits/cv_quick_update_tests

ceph-volume/tests: update setup_mixed_type playbook

crimson/common: disable arithmetic operators for atomic enums

While class enums are 'integral types', they do not support
all that an int does.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os/seastore/segment_cleaner: initialize segments' avail_bytes with segments' sizes

Currently, we initialize segments' avail_bytes with "segment_size * num_segments". Both segment_size
and num_segments are 32 bits long, multiplying them would lead to overflow.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge PR #43826 into master

* refs/pull/43826/head:
mgr/cephadm: allow zapping devices from other clusters

Reviewed-by: Adam King <adking@redhat.com>

qa/suites/orch/cephadm: add repave-all test case

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/services/osd: less noisy

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/services/osd: do not log ok-to-stop/safe-to-destroy failures

These failures are normal and expected; they should not pollute the log.

Signed-off-by: Sage Weil <sage@newdream.net>

.github/pull_request_template: address review comments

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: allow zapping devices from other clusters

This is the 99% of the devices that ever get zapped.

Fixes: b7782084ac9657be9b2da6ebd56b5029cf859225
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43814 from neha-ojha/wip-more-cv

qa/suites/upgrade/octopus-x/stress-split-no-cephadm: exclude ceph-volume

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/orchestrator: clean up 'orch osd rm status'

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: fix upgrade name already in use

Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>

Merge pull request #43808 from cbodley/wip-qa-rgw-java-master

qa/rgw: master branch targets ceph-master branch of java_s3tests

doc: document new scrub-related 'pg dump' columns

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: update the stand-alone tests to check 'scrub scheduling' entries

Analyzing and verifying the relevant entries in 'pg query' and
'pg dump' output.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: scrubbing schedule - minor related cleanups

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: expose scrubbing schedule to operator

Add a 'scrub scheduling info' column to pgs dump.
Modify the name and behavior of 'last-scrub-duration'.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Co-Authored-By: Aishwarya Mathuria <amathuri@redhat.com>

Merge pull request #43807 from sebastian-philipp/osd_memory_target_autotune-true

doc/cephadm: Recommend osd_memory_target_autotune

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@newdream.net>

doc/cephadm: Recommend osd_memory_target_autotune

In case the cluster runs on hardware that is used exclusively for
Ceph, let's recommend `osd_memory_target_autotune`

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

qa/suites/upgrade/octopus-x/stress-split-no-cephadm: exclude ceph-volume

To address failures like

```
Command failed on smithi096 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.15-11-g5f8f263c-1focal ceph-mds=15.2.15-11-g5f8f263c-1focal ceph-mgr=15.2.15-11-g5f8f263c-1focal ceph-common=15.2.15-11-g5f8f263c-1focal ceph-fuse=15.2.15-11-g5f8f263c-1focal ceph-test=15.2.15-11-g5f8f263c-1focal ceph-volume=15.2.15-11-g5f8f263c-1focal radosgw=15.2.15-11-g5f8f263c-1focal python3-rados=15.2.15-11-g5f8f263c-1focal python3-rgw=15.2.15-11-g5f8f263c-1focal python3-cephfs=15.2.15-11-g5f8f263c-1focal python3-rbd=15.2.15-11-g5f8f263c-1focal libcephfs2=15.2.15-11-g5f8f263c-1focal librados2=15.2.15-11-g5f8f263c-1focal librbd1=15.2.15-11-g5f8f263c-1focal rbd-fuse=15.2.15-11-g5f8f263c-1focal'
```

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43406 from ljflores/wip-telemetry-perf-improvements

mgr/telemetry: add mempool stats to telemetry perf report

Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

Merge PR #42727 into master

* refs/pull/42727/head:
mgr/orchestrator: improve usage string for 'orch daemon add osd'
ceph-volume: activate: try simple mode too
mgr/cephadm: identify and instantiate raw osds post-create
mgr/orchestrator: accept --method arg to 'orch daemon add osd'
python-common: drivegroup: add 'method' property
cephadm: use generic ceph-volume activate
ceph-volume: top-level 'activate' command
ceph-volume: lvm activate: add --no-tmpfs
ceph-volume: lvm activate: infer bluestore or filestore
ceph-volume: raw activate: accept --osd-id and/or --osd-uuid instead of device

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #43705 from tchaikov/wip-no-more-python2

mgr: do not handle Python2

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #43700 from liewegas/fix-24990

ceph_test_rados_api_watch_notify: extend Watch3Timeout test

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43664 from NUABO/tanchangzhi

osd: fix 'ceph osd stop <osd.nnn>' doesn't take effect

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #43752 into master

* refs/pull/43752/head:
client: remove usless _openat()
client: remove optional for dirfd parameter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #43666 into master

* refs/pull/43666/head:
qa/vstart_runner: add "managers" to LocalContext instances

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #43638 into master

* refs/pull/43638/head:
qa: pass subdir arg when executing workunit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #43613 into master

* refs/pull/43613/head:
qa: lengthen health warning wait

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #41667 into master

* refs/pull/41667/head:
mds: do not trim cache when creating system file
mds: fix the comment in add_inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #43611 into master

* refs/pull/43611/head:
doc/mgr/nfs: document rgw user and bucket exports
PendingReleaseNotes: add note about nfs CLI change(s)
qa/suites/orch/cephadm/smoke-roleless: add rgw user nfs export case
mgr/nfs: take user-id and/or bucket for 'nfs export create rgw'
mgr/nfs: reorder 'nfs export creage rgw' arguments
mgr/nfs: reorder 'nfs export create cephfs' arguments
mgr/nfs: use keyword args for 'nfs export create rgw'
mgr/nfs: document and use keyword args for 'nfs export create cephfs'
qa/tasks/cephfs/test_nfs: use keyword args
pybind/ceph_argparse: handle misordered keyword arguments

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #43768 from Huber-ming/admin_mdlog_fetch

radosgw-admin: supplement help documents with 'mdlog autotrim'

Merge pull request #43778 from adamemerson/wip-53132

rgw: Ensure buckets too old to decode a layout have layout logs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: master branch targets ceph-master branch of java_s3tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37184 from ybwang0211/KMSMSMSMS_return_error_message

rgw:When KMS encryption is used and the key does not exist, we should…

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #43725 from rhcs-dashboard/nfs-export-form-fix

mgr/dashboard: NFS 'create export' form: fixes

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

osd/scrub: remove reliance of Scrubber objects' logging on the PG

Modify the Scrubber's sub-objects to use their own gen_prefix()
functions, instead of using PG::gen_prefix().

Fixes: https://tracker.ceph.com/issues/51843
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

mgr/rook/rook_cluster: whitespace

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/rook/rook_cluster: remove drivegroup map and loop

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/rook/rook_cluster: do not duplicate StorageClassDeviceSets

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: persist applied drive groups and fix drive_group_loop

This commit changes OSD creation to store applied drive groups in a map
and persist that map in the mon's KV store in a map. This commit also
modifies the rook orchestrator's serve loop to periodically re-apply drive
groups, and creates a module option to specify the interval between
re-applies.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

doc/mgr/nfs: document rgw user and bucket exports

Signed-off-by: Sage Weil <sage@newdream.net>

PendingReleaseNotes: add note about nfs CLI change(s)

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/orch/cephadm/smoke-roleless: add rgw user nfs export case

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: take user-id and/or bucket for 'nfs export create rgw'

- move the bucket / user position after the cluster_id and pseudo_path
(since they are optional)
- require bucket or user or both
- if bucket, use the bucket owner
- if bucket+user, use that user
- if user only, then export at top-level (all users buckets)

Fixes: https://tracker.ceph.com/issues/53134
Signed-off-by: Sage Weil <sage@newdream.net>

mgr/orchestrator: improve usage string for 'orch daemon add osd'

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/dashboard: NFS 'create export' form: fixes

* Do not allow a pseudo that is already in use by another export.
* Create mode form: prefill dropdown selectors if options > 0.
* Edit mode form: do not reset the field values that depend on other values that are being edited (unlike Create mode).
* Fix broken link: cluster service.
* Fix error message style for non-existent cephfs path.
* nfs-service.ts: lsDir: thow error if volume is not provided.
* File renaming: nfsganesha.py => nfs.py; test_ganesha.py => test_nfs.py

Fixes: https://tracker.ceph.com/issues/53083
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

cephadm: Avoid "Could not locate podman: podman not found"

Right now, we're spamming the cephadm.log on docker deployments

Fixes: https://tracker.ceph.com/issues/52869
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43737 from AndrewSharapov/master

mgr/cephadm: Fixed spawning ip addresses list for public network interface

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43797 from rhcs-dashboard/fix-53144-master

mgr/dashboard: fix missing alert rule details

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>