mgr/dashboard: fix e2e failure related to landing page

Fixes: https://tracker.ceph.com/issues/59142
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #54878 from yuvalif/wip-yuval-split-rgw-tools

rgw: split RGWDataAccess from rgw_tools.cc

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #54840 from cbodley/wip-qa-rgw-crypt-disable-kmip

qa/rgw/crypt: disable failing kmip testing

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #51732 from andreas-schwab/main

ceph.spec.in: enable build on riscv64 for openSUSE Factory

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #54893 from zdover23/wip-doc-2023-12-14-radosgw-role-formatting

doc/radosgw: format commands in role.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: format commands in role.rst

Format approximately one-hundred lines of doc/radosgw/role.rst to
include proper command prompts. I also made one small English usage
improvement.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54723 from phlogistonjohn/jjm-bib

script/build-integration-branch: minor improvements around github api access & tokens

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Laura Flores <lflores@ibm.com>

Merge PR #54733 into main

* refs/pull/54733/head:
client/fuse: handle case of renameat2 with non-zero flags

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53393 from rzarzynski/wip-ceph-run-op

ceph-run: fix string comparison around --no-restart

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #54657 from a180285/patch-2

rbd: update import-diff command help message

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #54832 from hex2dec/fix-cephadm-shell-help

cephadm: Fix `cephadm shell` command help message

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54814 from rhcs-dashboard/update-grafana-version-cephadm

mgr/cephadm: bump grafana to 9.4.12

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54786 from guits/fix_63729

python-common: fix osdspec_affinity check

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54681 from adk3798/device-limit-testing

python-common/drive_selection: fix limit with existing devices

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #54536 from phlogistonjohn/jjm-cephadm-units-n-scripts

cephadm: add modules to abstract systemd units and run scripts

Reviewed-by: Adam King <adking@redhat.com>

Update rbd import-diff command help message

Signed-off-by: HuangWei <huangwei.a180285@gmail.com>

Merge pull request #54319 from ajarr/wip-63410

mgr/Mgr: remove shutdown() and handle_signal()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #54102 from ifed01/wip-ifed-better-vselector-calls

os/bluestore: rework vselector calls.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #52210 from ceph/wip-nitzan-ceph-dencoder-extend-common-types-available

ceph-dencoder: COMMON - Add missing types

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #48895 into main

* refs/pull/48895/head:
qa: test cases for checking the health status after scrub repair
mds: scrub repair does not clear earlier damage health status

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #54628 into main

* refs/pull/54628/head:
qa/fscrypt: enable debug log for bluestore

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #54669 into main

* refs/pull/54669/head:
mds: set the loner to true for LOCK_EXCL_XSYN

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #54704 into main

* refs/pull/54704/head:
qa: don't fail on gather_mount_info() failure

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge PR #54745 into main

* refs/pull/54745/head:
mds: encode `bal_rank_mask` using a higher (next) version

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #54864 from cbodley/wip-mgr-rgw-realm-name

mgr/rgwam: period doesn't have a realm_name anymore

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #54615 from ronen-fr/wip-rf-schedRepair

osd/scrub: allow auto-repair on operator-initiated scrubs
Reviewed-by: Samuel Just <sjust@redhat.com>-

Merge pull request #54748 from cbodley/wip-63717

rgw/sysobj: pool_list_objects_init() initializes marker

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #53841 from pilem/bug62320

src/ceph-volume/ceph_volume/devices/lvm/listing.py : lvm list filters with vg name

Merge pull request #54608 from dillona/main

ceph-volume: Use safe accessor to get TYPE info

Merge pull request #54453 from saschalucas/fix-blkid-missing-key-type

ceph-volume: fix blkid on inaccessible devices

Merge pull request #53804 from guits/fix_63086

ceph-volume: fix util.get_partitions

Merge pull request #54337 from batrick/i63433

pybind/mgr/devicehealth: replace SMART data if exists for same DATETIME

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

rgw: split RGWDataAccess from rgw_tools.cc

that class is not rados specific, while the rest of the code
in rgw_tools.cc is

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #54875 from zdover23/wip-doc-2023-12-12-radosgw-role-create-a-role

doc/radosgw: format "Create a Role"

doc/radosgw: format "Create a Role"

Properly format the section "Create a Role" in the file
doc/radosgw/role.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54750 from ivoalmeida/snapshot-schedule-list

mgr/dashboard: added snap-schedule api and ui list

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #53483 from ifed01/wip-ifed-no-alloc-lba-align

os/bluestore: get rid off resulting lba alignment in allocators

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

mgr/dashboard: added snap-schedule api and ui list

Fixes: https://tracker.ceph.com/issues/63767
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge pull request #53830 from alimaredia/wip-test-librgw-file-sh-vstart

qa: enable test_librgw_file.sh to be run with vstart cluster

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/rgwam: period doesn't have a realm_name anymore

Fixes: https://tracker.ceph.com/issues/63783
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #54726 into main

* refs/pull/54726/head:
PendingReleaseNotes: announce cephfs-shell avail. on rhel9
qa: test fs:shell on all distros
qa: add cephfs-shell to installed rpm packages
ceph.spec.in: enable support for cephfs-shell by default via EPEL9

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #54779 from rishabh-d-dave/fs-release-notes

cephfs: add a couple of release notes

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #54848 from idryomov/wip-diff-iterate-test-fixup

test/librbd: close image in DiffIterateDeterministic

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #54847 from idryomov/wip-journaling-test-fixups

test/librbd: avoid config-related crashes in DiscardWithPruneWriteOverlap

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #53054 from xxhdx1985126/wip-crimson-pg-proj-last-update

crimson/osd/pg: set PG::projected_last_update when submitting transactions

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53078 from xxhdx1985126/wip-crimson-pg-recovery

crimson/osd/pg_recovery: avoiding duplicated object recovering

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #53127 from xxhdx1985126/wip-crimson-obc-iteration

crimson/osd/pg: discard watches' states after iterating all cached obcs

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #53409 from xxhdx1985126/wip-crimson-recover-head-first

crimson/osd/pg_recovery: recover head before recovering clones

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #53985 from xxhdx1985126/wip-63186

crimson/osd: store "final pool info" for pools that are deleted

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53306 from xxhdx1985126/wip-clone-wait-head

crimson/osd/osd_operations/client_request: recover the head and other necessary objects before proceeding

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53130 from cyx1231st/wip-crimson-drop-mempool

mempool: avoid true sharing for the counters, with crimson

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Laura Flores <lflores@ibm.com>

Merge pull request #54845 from zdover23/wip-doc-2023-12-09-radosgw-role-2

doc/radosgw: edit front matter - role.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit front matter - role.rst

Clarify the front matter in doc/radosgw/role.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54789 from yuvalif/wip-yuval-remove-lua-devel

build/rgw/lua: remove luarocks support from rhel8

reviewed-by: batrick, cbodley, mchangir

cephadm: black format runscripts.py

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move abstract script handling functions to runscripts.py

Add a new file runscripts.py for the lower-level management of scripts
and related files that are invoked by systemd units. This patch ended up
uglier than I desired because there was a bunch of daemon specific logic
that remains in cephadm.py and those functions all needed to be updated
to avoid calling functions that write to the scripts directly.
Now customizations are done by passing a list of commands: these
commands can be either a string that will be literally added to the
scipt, a list that will be quoted and then added to the script, or
a ContainerCommand which is basically a wrapper around the arguments to
_write_container_cmd_to bash.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add a higher-level function for managing systemd units

Add the function update_files to systemd_unit.py to encapsulate and
abstract the details regarding the generation of system unit files.
This will make it simpler in the future to add more advanced systemd
configurations include managing customized unit files and systemd
unit drop-in files.

Some additional work was needed to update the recently added
command_unit_install function. Because the new systemd_unit.update_files
function requires a full daemon identity. The command_unit_install
function now requires a daemon name. In addition, while testing this
change it was found that the function could not have worked as it was
because it required the fsid but neither used the infer_fsid decorator
nor provided a `--fsid` argument. Both were added.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: update unit file test imports

Update systemd unit file tests file to use the canonical module for
systemd units functions rather than importing them indirectly from
cephadm.py.
This future proofs the test in case the imports in cephadm.py
change.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: create systemd_unit.py for systemd unit related functions

Continue to modularize cephadm by moving the functions related to
generating systemd unit files to a new module.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: update container engine test imports

Update container engine tests file to use the canonical module for
container engines rather than importing them indirectly from cephadm.py.
This future proofs the test in case the imports in cephadm.py change.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add shellquote filter to template

This is to support generating command lines or shell scripts using
Jinja2 templates in the future.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

build/rgw/lua: remove lua-devel as a runtime dependency

this is due to the fact that lua-devel
cannot be installed in teuthology for rhel8

this would fail the build of some luarocks packages

Fixes: https://tracker.ceph.com/issues/63672
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

test/librbd: close image in DiffIterateDeterministic

... to avoid valgrind reporting a memory leak on ImageCtx.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: actually alternate overlaps in DiscardWithPruneWriteOverlap

Make sense of the inner loop in write_thread.  The crash on
"it != m_events.end()" assert reproduces even faster this way.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: avoid config-related crashes in DiscardWithPruneWriteOverlap

For reasons that I think no longer apply today, set_val() and
set_val_or_die() refuse to set "type: str" config options that aren't
marked as "can be changed at runtime" -- set_val() returns an error and
set_val_or_die() terminates the process.  What is and isn't marked as
"can be changed at runtime" seems to be pretty much random both within
and outside of RBD, so let's just refactor how config is set here.

While at it, I realized that reproducer config is underspecified:

- for rbd_cache_policy and rbd_cache_writethrough_until_flush settings
  to matter, rbd_cache must be set to true and rbd_cache_max_dirty must
  be set to a positive number

- order should be set explicitly, because rbd_default_order can be as
  low as 12 (for 4096-byte objects), interfering with the logic of the
  test

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #54378 from adk3798/cephadm-jinja2-logrotate

cephadm: move logrotate configs to jinja2 template

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #54846 from zdover23/wip-doc-2023-12-09-radosgw-role

doc/radosgw: format POST statements

doc/radosgw: format POST statements

Format the POST methods so that they appear in the rendered text as
examples of POST API calls and not as plain old unformatted text, which
is how they looked before this commit. The content of these API calls
remains to be tested and confirmed to work, but this is a first step.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54839 from cbodley/wip-librbd-strands

librbd: include asio strand headers in librbd.cc

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #54827 from cbodley/wip-vstart-dbstore

vstart: add --rgw_store option for rados|dbstore|posix

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #54546 from sonetto-lmz/post-etag

rgw: fix post obj does not return the response etag field

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

Merge pull request #54622 from BBoozmen/oozmen_lc_work_time

[rgw][lc][rgw_lifecycle_work_time] adjust timing if the configured end time is less than the start time

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #54668 from qiuxinyidian/rgw-frontend

add checking for rgw frontend init

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #54738 from cbodley/wip-63485

rgw/iam: admin/system users ignore iam policy parsing errors

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #54749 from cbodley/wip-rgw-zone-old-pools

rgw/zone: remove check for (very) old pools config

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

qa/rgw/crypt: disable failing kmip testing

kmip tests have been failing all year and are not being maintained:

https://tracker.ceph.com/issues/58751
https://tracker.ceph.com/issues/58523

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: include asio strand headers in librbd.cc

Fixes: https://tracker.ceph.com/issues/63682
Signed-off-by: Casey Bodley <cbodley@redhat.com>

vstart: add --rgw_store option for rados|dbstore|posix

enables dbstore for rgw_backend_store and rgw_config_store, allowing
vstart to run without any mons or osds. database files are put under
the dev subdirectory

when rgw_store=posix, the posix filter is added on top of dbstore

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cephfs: move release note for PR #41779 to right spot

Release note for PR #41779 applies releases after Reef, not  Reef
onwards. Move the release note accordingly.

Fixes: https://github.com/ceph/ceph/pull/41779
Fixes: https://tracker.ceph.com/issues/47264
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #54751 from zdover23/wip-doc-2023-12-02-radosgw-multisite-starting-the-gateway

doc/radosgw: add gateway starting command

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

cephadm: Fix `cephadm shell` command help message

The meaning of the `--volume` option is to bind mount a volume.

Signed-off-by: Zhiwei Huang <ai.william@outlook.com>

Merge pull request #54773 from phlogistonjohn/jjm-cephadm-zipapp-templating-again

cephadm: simplify loading jinja2 templates from zipapp

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54278 from smanjara/wip-fix-rgw-full-sync

rgw/multisite: fix marker_tracker flush() call in incremental sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54810 from mkogan1/rgw-wip-t63445-valg-supp-rb

rgw: d3n: fix valgrind reported leak related to libaio worker threads

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54809 from joscollin/wip-doc-mirror-add-supported-file-types

doc: add supported file types in cephfs-mirroring.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

client/fuse: handle case of renameat2 with non-zero flags

When user issues renameat(2) with non-zero flags (RENAME_EXCHANGE or
RENAME_NOREPALCE) the current code ignores those flags and treat the
call as ordinary rename. This, in turn, may yield successful rename with
wrong semantics then those expected by the caller.

Follow the same semantics as kernel's cephfs client: return -EINVAL when
having non-zero flags to renameat2 (see 'ceph_rename' at fs/ceph/dir.c).

Fixes: https://tracker.ceph.com/issues/63722
Signed-off-by: Shachar Sharon <ssharon@redhat.com>

Merge pull request #54377 from baergj/fix-journal-tid-association-try2-upstream

librbd: Append one journal event per image request

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mgr/cephadm: bump grafana to 9.4.12

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54799 from cbodley/wip-osd-scrub-signed-warning

osd/scrubber: fix signed comparison warning

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

doc: add supported file types in cephfs-mirroring.rst

Signed-off-by: Jos Collin <jcollin@redhat.com>

rgw: d3n: fix valgrind reported leak related to libaio worker threads

which sporadically reproduces on teuthology ubuntu instances

happens because a race between RGW shutdown occurring before
the libaio worker threads had terminated

to fix, reduced the libaio threads inactivity shutdown time

ref:
man aio_init
...
aio_idle_time
This field specifies the amount of time in seconds that a worker thread
should wait for further requests before terminating, after having
completed a previous request.   The
default value is 1.
...

Fixes: https://tracker.ceph.com/issues/63445
Signed-off-by: Mark Kogan <mkogan@ibm.com>

Merge pull request #54755 from idryomov/wip-63654

librbd: fix regressions in ObjectListSnapsRequest

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #54804 from rhcs-dashboard/upgrade-grafana-9.4.12

monitoring: upgrade grafana container to 9.4.12

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #54670 from trociny/wip-cython-3

pybind/rbd: fix compilation with cython3

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #54790 from zdover23/wip-doc-2023-12-05-radosgw-rgw-cache-nginx-link-update

doc/radosgw: update link in rgw-cache.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: add gateway starting command

Add a command that properly starts (or restarts) the RADOS gateway after
RGW settings have been changed. This commit has been added in response
to an issue reported anonymously on
https://pad.ceph.com/p/Report_Documentation_Bugs.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54215 from kchheda3/wip-fix-kafka-break

rgw/notification: Kafka persistent notifications not retried and removed even when the broker is down

reviewd-by: cbodley, yuvalif

monitoring: upgrade grafana container to 9.4.12

Fixes the CVEs mentioned here: https://grafana.com/blog/2023/06/06/grafana-security-release-new-grafana-versions-with-security-fixes-for-cve-2023-2183-and-cve-2023-2801/

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54787 from cbodley/wip-versioned-variant-inline

common: use inline for monostate dencoders

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

test/librbd: Add workaround for a journaling deadlock that this test exposes

Signed-off-by: Joshua Baergen <jbaergen@digitalocean.com>