Merge pull request #54569 from AliMasarweh/wip-alimasa-multi-pubsub-etag

RGW: pubsub publish commit with etag populated

revieweed-by: cbodley, yuvalif

Merge pull request #52354 from bluikko/patch-13

doc/radosgw/admin.rst: use underscores in config var names

Merge pull request #54888 from cbodley/wip-rgw-multisite-forward-empty-buffer

rgw/multisite: forwarded requests always pass a bufferlist

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #54739 from cbodley/wip-qa-rgw-upgrade-distros

qa/rgw/upgrade: stop testing upgrade from pacific

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #54841 from cbodley/wip-rgw-multisite-rados-crash

rgw/multisite: error repo coroutines store raw_obj by value

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #54920 from tchaikov/cls-2pc-reserv-dencode

cls/queue: always set member variables in ctor

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cls/queue: always set member variables in ctor

this should address the test failures like

```
/tmp/typ-WWFeFl6yK /tmp/typ-sMfwoaGMU differ: byte 24, line 1
**** cls_2pc_reservation test 2 binary reencode check failed ****
   ceph-dencoder type cls_2pc_reservation select_test 2 encode export /tmp/typ-WWFeFl6yK
   ceph-dencoder type cls_2pc_reservation select_test 2 encode decode encode export /tmp/typ-sMfwoaGMU
2c2
< 00000010  00 00 00 00 00 00 c0 c6  92 10                    |..........|
---
> 00000010  00 00 00 00 00 00 c0 e6  cd 53                    |.........S|
```

where we
1. encode the 2nd sample created by `generate_test_instances()`,
2. encode, decode, and encode again, the same sample

and compare the encoded blobs.

but if we fail to set any of the fields in `cls_2pc_reservation`,
we are at the mercy of the random bits on stack/heap.

in this change, all bits are initialized.

the flaky test was introduced by 1d7cabf3

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge PR #52196 into main

* refs/pull/52196/head:
qa: configure balancer for multi-mds workloads
qa: create qa subvolumes in named subvolumegroup
qa: do not rely on default max_mds value
qa: add automate_balance to dashboard qa schema
doc/cephfs: add docs for balance_automate
doc/cephfs: use bash prompt for shell code
mds: add balance_automate fs setting

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #54878 from yuvalif/wip-yuval-split-rgw-tools

rgw: split RGWDataAccess from rgw_tools.cc

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #54840 from cbodley/wip-qa-rgw-crypt-disable-kmip

qa/rgw/crypt: disable failing kmip testing

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #51732 from andreas-schwab/main

ceph.spec.in: enable build on riscv64 for openSUSE Factory

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #54893 from zdover23/wip-doc-2023-12-14-radosgw-role-formatting

doc/radosgw: format commands in role.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: format commands in role.rst

Format approximately one-hundred lines of doc/radosgw/role.rst to
include proper command prompts. I also made one small English usage
improvement.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54723 from phlogistonjohn/jjm-bib

script/build-integration-branch: minor improvements around github api access & tokens

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Laura Flores <lflores@ibm.com>

qa/rgw/upgrade: stop testing upgrade from pacific

squid won't support upgrades from pacific. stop all testing against
centos8 and ubuntu20

Signed-off-by: Casey Bodley <cbodley@redhat.com>

RGW:pubsub publish commit with etag populated

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

Merge PR #54733 into main

* refs/pull/54733/head:
client/fuse: handle case of renameat2 with non-zero flags

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53393 from rzarzynski/wip-ceph-run-op

ceph-run: fix string comparison around --no-restart

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

rgw/multisite: error repo coroutines initialize rados pointer

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/multisite: error repo coroutines store raw_obj by value

RGWErrorRepoWriteCR/RGWErrorRepoRemoveCR need to copy their rgw_raw_obj
arguments to extend the lifetimes until send_request() is called

potential regression from https://github.com/ceph/ceph/pull/50359

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54657 from a180285/patch-2

rbd: update import-diff command help message

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #54832 from hex2dec/fix-cephadm-shell-help

cephadm: Fix `cephadm shell` command help message

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54814 from rhcs-dashboard/update-grafana-version-cephadm

mgr/cephadm: bump grafana to 9.4.12

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54786 from guits/fix_63729

python-common: fix osdspec_affinity check

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54681 from adk3798/device-limit-testing

python-common/drive_selection: fix limit with existing devices

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #54536 from phlogistonjohn/jjm-cephadm-units-n-scripts

cephadm: add modules to abstract systemd units and run scripts

Reviewed-by: Adam King <adking@redhat.com>

Update rbd import-diff command help message

Signed-off-by: HuangWei <huangwei.a180285@gmail.com>

Merge pull request #54319 from ajarr/wip-63410

mgr/Mgr: remove shutdown() and handle_signal()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #54102 from ifed01/wip-ifed-better-vselector-calls

os/bluestore: rework vselector calls.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #52210 from ceph/wip-nitzan-ceph-dencoder-extend-common-types-available

ceph-dencoder: COMMON - Add missing types

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #48895 into main

* refs/pull/48895/head:
qa: test cases for checking the health status after scrub repair
mds: scrub repair does not clear earlier damage health status

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #54628 into main

* refs/pull/54628/head:
qa/fscrypt: enable debug log for bluestore

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #54669 into main

* refs/pull/54669/head:
mds: set the loner to true for LOCK_EXCL_XSYN

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #54704 into main

* refs/pull/54704/head:
qa: don't fail on gather_mount_info() failure

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge PR #54745 into main

* refs/pull/54745/head:
mds: encode `bal_rank_mask` using a higher (next) version

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw/multisite: forwarded requests always pass a bufferlist

d2dbe7550296da6db885b5344c71f77f9acbfd8f added a rgw_forward_request_to_master()
that took the input bufferlist by pointer instead of reference so it
could be optional; however, RGWRESTSimpleRequest::forward_request()
omits the Content-Length header when the data is nullptr. this was an
unintended change and broke the forwarding of some requests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54864 from cbodley/wip-mgr-rgw-realm-name

mgr/rgwam: period doesn't have a realm_name anymore

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #54615 from ronen-fr/wip-rf-schedRepair

osd/scrub: allow auto-repair on operator-initiated scrubs
Reviewed-by: Samuel Just <sjust@redhat.com>-

Merge pull request #54748 from cbodley/wip-63717

rgw/sysobj: pool_list_objects_init() initializes marker

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

qa: configure balancer for multi-mds workloads

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: create qa subvolumes in named subvolumegroup

So we can twiddle the subvolumegroup settings (like pins).

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: do not rely on default max_mds value

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add automate_balance to dashboard qa schema

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephfs: add docs for balance_automate

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephfs: use bash prompt for shell code

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #53841 from pilem/bug62320

src/ceph-volume/ceph_volume/devices/lvm/listing.py : lvm list filters with vg name

Merge pull request #54608 from dillona/main

ceph-volume: Use safe accessor to get TYPE info

Merge pull request #54453 from saschalucas/fix-blkid-missing-key-type

ceph-volume: fix blkid on inaccessible devices

Merge pull request #53804 from guits/fix_63086

ceph-volume: fix util.get_partitions

mds: add balance_automate fs setting

To turn off the automatic ("default") balancer in multiple MDS clusters. The
new default is "off" as the balancer  is a constant source of problems and
surprise for administrators trying multiple actives. Instead, it should be a
deliberate decision to turn it on and usually with customization like the
"bal_rank_mask" setting or pinning.

Fixes: https://tracker.ceph.com/issues/61378
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #54337 from batrick/i63433

pybind/mgr/devicehealth: replace SMART data if exists for same DATETIME

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

rgw: split RGWDataAccess from rgw_tools.cc

that class is not rados specific, while the rest of the code
in rgw_tools.cc is

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #54875 from zdover23/wip-doc-2023-12-12-radosgw-role-create-a-role

doc/radosgw: format "Create a Role"

doc/radosgw: format "Create a Role"

Properly format the section "Create a Role" in the file
doc/radosgw/role.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54750 from ivoalmeida/snapshot-schedule-list

mgr/dashboard: added snap-schedule api and ui list

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53483 from ifed01/wip-ifed-no-alloc-lba-align

os/bluestore: get rid off resulting lba alignment in allocators

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

mgr/dashboard: added snap-schedule api and ui list

Fixes: https://tracker.ceph.com/issues/63767
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge pull request #53830 from alimaredia/wip-test-librgw-file-sh-vstart

qa: enable test_librgw_file.sh to be run with vstart cluster

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/rgwam: period doesn't have a realm_name anymore

Fixes: https://tracker.ceph.com/issues/63783
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #54726 into main

* refs/pull/54726/head:
PendingReleaseNotes: announce cephfs-shell avail. on rhel9
qa: test fs:shell on all distros
qa: add cephfs-shell to installed rpm packages
ceph.spec.in: enable support for cephfs-shell by default via EPEL9

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #54779 from rishabh-d-dave/fs-release-notes

cephfs: add a couple of release notes

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #54848 from idryomov/wip-diff-iterate-test-fixup

test/librbd: close image in DiffIterateDeterministic

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #54847 from idryomov/wip-journaling-test-fixups

test/librbd: avoid config-related crashes in DiscardWithPruneWriteOverlap

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #53054 from xxhdx1985126/wip-crimson-pg-proj-last-update

crimson/osd/pg: set PG::projected_last_update when submitting transactions

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53078 from xxhdx1985126/wip-crimson-pg-recovery

crimson/osd/pg_recovery: avoiding duplicated object recovering

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #53127 from xxhdx1985126/wip-crimson-obc-iteration

crimson/osd/pg: discard watches' states after iterating all cached obcs

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #53409 from xxhdx1985126/wip-crimson-recover-head-first

crimson/osd/pg_recovery: recover head before recovering clones

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #53985 from xxhdx1985126/wip-63186

crimson/osd: store "final pool info" for pools that are deleted

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53306 from xxhdx1985126/wip-clone-wait-head

crimson/osd/osd_operations/client_request: recover the head and other necessary objects before proceeding

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53130 from cyx1231st/wip-crimson-drop-mempool

mempool: avoid true sharing for the counters, with crimson

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Laura Flores <lflores@ibm.com>

Merge pull request #54845 from zdover23/wip-doc-2023-12-09-radosgw-role-2

doc/radosgw: edit front matter - role.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit front matter - role.rst

Clarify the front matter in doc/radosgw/role.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54789 from yuvalif/wip-yuval-remove-lua-devel

build/rgw/lua: remove luarocks support from rhel8

reviewed-by: batrick, cbodley, mchangir

cephadm: black format runscripts.py

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move abstract script handling functions to runscripts.py

Add a new file runscripts.py for the lower-level management of scripts
and related files that are invoked by systemd units. This patch ended up
uglier than I desired because there was a bunch of daemon specific logic
that remains in cephadm.py and those functions all needed to be updated
to avoid calling functions that write to the scripts directly.
Now customizations are done by passing a list of commands: these
commands can be either a string that will be literally added to the
scipt, a list that will be quoted and then added to the script, or
a ContainerCommand which is basically a wrapper around the arguments to
_write_container_cmd_to bash.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add a higher-level function for managing systemd units

Add the function update_files to systemd_unit.py to encapsulate and
abstract the details regarding the generation of system unit files.
This will make it simpler in the future to add more advanced systemd
configurations include managing customized unit files and systemd
unit drop-in files.

Some additional work was needed to update the recently added
command_unit_install function. Because the new systemd_unit.update_files
function requires a full daemon identity. The command_unit_install
function now requires a daemon name. In addition, while testing this
change it was found that the function could not have worked as it was
because it required the fsid but neither used the infer_fsid decorator
nor provided a `--fsid` argument. Both were added.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: update unit file test imports

Update systemd unit file tests file to use the canonical module for
systemd units functions rather than importing them indirectly from
cephadm.py.
This future proofs the test in case the imports in cephadm.py
change.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: create systemd_unit.py for systemd unit related functions

Continue to modularize cephadm by moving the functions related to
generating systemd unit files to a new module.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: update container engine test imports

Update container engine tests file to use the canonical module for
container engines rather than importing them indirectly from cephadm.py.
This future proofs the test in case the imports in cephadm.py change.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add shellquote filter to template

This is to support generating command lines or shell scripts using
Jinja2 templates in the future.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

build/rgw/lua: remove lua-devel as a runtime dependency

this is due to the fact that lua-devel
cannot be installed in teuthology for rhel8

this would fail the build of some luarocks packages

Fixes: https://tracker.ceph.com/issues/63672
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

test/librbd: close image in DiffIterateDeterministic

... to avoid valgrind reporting a memory leak on ImageCtx.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: actually alternate overlaps in DiscardWithPruneWriteOverlap

Make sense of the inner loop in write_thread.  The crash on
"it != m_events.end()" assert reproduces even faster this way.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: avoid config-related crashes in DiscardWithPruneWriteOverlap

For reasons that I think no longer apply today, set_val() and
set_val_or_die() refuse to set "type: str" config options that aren't
marked as "can be changed at runtime" -- set_val() returns an error and
set_val_or_die() terminates the process.  What is and isn't marked as
"can be changed at runtime" seems to be pretty much random both within
and outside of RBD, so let's just refactor how config is set here.

While at it, I realized that reproducer config is underspecified:

- for rbd_cache_policy and rbd_cache_writethrough_until_flush settings
  to matter, rbd_cache must be set to true and rbd_cache_max_dirty must
  be set to a positive number

- order should be set explicitly, because rbd_default_order can be as
  low as 12 (for 4096-byte objects), interfering with the logic of the
  test

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #54378 from adk3798/cephadm-jinja2-logrotate

cephadm: move logrotate configs to jinja2 template

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #54846 from zdover23/wip-doc-2023-12-09-radosgw-role

doc/radosgw: format POST statements

doc/radosgw: format POST statements

Format the POST methods so that they appear in the rendered text as
examples of POST API calls and not as plain old unformatted text, which
is how they looked before this commit. The content of these API calls
remains to be tested and confirmed to work, but this is a first step.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54839 from cbodley/wip-librbd-strands

librbd: include asio strand headers in librbd.cc

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #54827 from cbodley/wip-vstart-dbstore

vstart: add --rgw_store option for rados|dbstore|posix

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #54546 from sonetto-lmz/post-etag

rgw: fix post obj does not return the response etag field

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

Merge pull request #54622 from BBoozmen/oozmen_lc_work_time

[rgw][lc][rgw_lifecycle_work_time] adjust timing if the configured end time is less than the start time

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #54668 from qiuxinyidian/rgw-frontend

add checking for rgw frontend init

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #54738 from cbodley/wip-63485

rgw/iam: admin/system users ignore iam policy parsing errors

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #54749 from cbodley/wip-rgw-zone-old-pools

rgw/zone: remove check for (very) old pools config

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

qa/rgw/crypt: disable failing kmip testing

kmip tests have been failing all year and are not being maintained:

https://tracker.ceph.com/issues/58751
https://tracker.ceph.com/issues/58523

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: include asio strand headers in librbd.cc

Fixes: https://tracker.ceph.com/issues/63682
Signed-off-by: Casey Bodley <cbodley@redhat.com>

vstart: add --rgw_store option for rados|dbstore|posix

enables dbstore for rgw_backend_store and rgw_config_store, allowing
vstart to run without any mons or osds. database files are put under
the dev subdirectory

when rgw_store=posix, the posix filter is added on top of dbstore

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cephfs: move release note for PR #41779 to right spot

Release note for PR #41779 applies releases after Reef, not  Reef
onwards. Move the release note accordingly.

Fixes: https://github.com/ceph/ceph/pull/41779
Fixes: https://tracker.ceph.com/issues/47264
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #54751 from zdover23/wip-doc-2023-12-02-radosgw-multisite-starting-the-gateway

doc/radosgw: add gateway starting command

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

cephadm: Fix `cephadm shell` command help message

The meaning of the `--volume` option is to bind mount a volume.

Signed-off-by: Zhiwei Huang <ai.william@outlook.com>