Merge PR #56219 into main

* refs/pull/56219/head:
qa: change log-whitelist to log-ignorelist

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #56245 into main

* refs/pull/56245/head:
mds/quiesce-db: always clear the db if a membership is lost

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #56298 from batrick/i64986

qa/cephfs: ignorelist clog of MDS_UP_LESS_THAN_MAX

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56299 into main

* refs/pull/56299/head:
qa/suites/fs/nfs: use standard health ignorelist

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #56054 from rhcs-dashboard/open-swagger-new-tab

mgr/dashboard: open api swagger in new tab

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #56379 from Svelar/doc_oE

doc/install: fix typos in openEuler-installation doc

Merge pull request #56246 from xxhdx1985126/wip-64957

crimson/os/seastore/btree: check for reserved ptrs when determining children stability

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #56237 from rhcs-dashboard/store-token-in-cookie

mgr/dashboard: securely store remote cluster token

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: ivoalmeida <NOT@FOUND>

crimson/os/seastore/btree: always check the stability of extents within
the current transaction's view

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

doc/install: fix typos in openEuler-installation doc

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Merge pull request #55886 from tchaikov/wip-ceph.spec-gcc-13

ceph.spec.in: bump gcc-toolset to 13 and use it on rhel>=8

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #56369 from adk3798/nvmeof-new-fields

mgr/cephadm: add some NVMEof gateway fields to be configurable by cephadm

Reviewed-by: Gil Bregman <gbregman@il.ibm.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56283 from Svelar/remove_WITH_ZBD

run-make: remove WITH_ZBD

Merge pull request #55691 from ivoalmeida/snapshot-schedule-retention-fix

mgr/dashboard: fix retention add for subvolume

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/cephadm: add nvmeof configuration fields

Fixes: https://tracker.ceph.com/issues/64821
Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

python-common/ceph/deployment: add nvmeof configuration fields

Fixes: https://tracker.ceph.com/issues/64821
Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #56008 from kchheda3/wip-notification-subsys

rgw/notification: add rgw notification specific debug log subsystem

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #55348 from chenyuanrun/fix-io-ctx-impl-flags

librados: use CEPH_OSD_FLAG_FULL_FORCE for IoCtxImpl::remove

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #50622 from cofractal/close-range

common: use close_range on Linux

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

run-make: remove WITH_ZBD

This commit remove 'WITH_ZBD' in run-make.sh because of: https://github.com/ceph/ceph/pull/55054

Fixes: https://tracker.ceph.com/issues/64962
Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Merge pull request #56278 from sachinpunadikar/vstart_ganesha_fix

vstart: Ganesha should not be started in DEBUG mode

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #56284 from ivoalmeida/snapshot-schedule-monthly-retention-frequency

mgr/dashboard: fix M retention frequency display

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #56146 from ljflores/wip-tracker-64725

qa/suites/rados/singleton: add POOL_APP_NOT_ENABLED to ignorelist

Merge pull request #56183 from rhcs-dashboard/fix-tags-null

mgr/dashboard: fix empty tags

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52617 from TomNewChao/main

mgr/dashboard/frontend:Ceph dashboard supports multiple languages

Reviewed-by: Svelar <NOT@FOUND>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #55841 from Matan-B/wip-matanb-mon-rmsnap-fix

mon/OSDMonitor: fix rmsnap command

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #55409 from cybozu/add-ceph-exporter-to-deb-package

debian: add ceph-exporter package

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #53308 from NitzanMordhai/wip-nitzan-qa-tasks-with-crush-rules

suites: qa tasks with crush rules

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #56266 from cbodley/wip-qa-rgw-lua-pool-application

qa/rgw: ignore POOL_APP_NOT_ENABLED in rgw/lua subsuite

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #56220 from linuxbox2/wip-nfs-rename

rgw_file: fix mv/rename cases broken by zipper integration

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #56225 from linuxbox2/wip-cors-fix

rgw: Add missing empty checks to the split string in is_string_in_set().

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56044 from pr0ton11/rgw-disable-signature-url

RGW: allow user disabling presigned urls in rgw configuration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55795 from kchheda3/wip-rename-notif-events

rgw/notification: Make the LifeCycle & Replication events AWS Compatible and Support s3:Replication:Delete event

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #56333 from molpako/patch-3

doc/cephfs: fix architecture link to correct relative path

Reviewed-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: fix M retention frequency display

Fix snap schedule list monthly retention frequency display

Fixes: https://tracker.ceph.com/issues/64982
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

mgr/dashboard: open api swagger in new tab

Fixes: https://tracker.ceph.com/issues/62350
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

mgr/dashboard: fix empty tags

Fixes: https://tracker.ceph.com/issues/64911
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #55595 from VallariAg/wip-nvmeof-test-v3

qa/suite/rbd/nvmeof: Deploy multiple gateways and namespaces

Reviewed-by: Barak Davidov <barakda@il.ibm.com>
Reviewed-by: Aviv Caro <Aviv.Caro@ibm.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/cephfs: fix architecture link to correct relative path

Signed-off-by: molpako <35787116+molpako@users.noreply.github.com>

Merge pull request #55864 from rosinL/wip-speedup-check-generated

test/encoding/check-generated: parallel testing by type

Merge pull request #56203 from athanatos/sjust/wip-protoc-dependencies

crimson: add crimson-osd rpm and deb runtime dependencies for protobuf inherited from seastar

Merge pull request #56070 from phlogistonjohn/jjm-cephadm-sudo-cmd-auditing

cephadm: sudo cmd auditing

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #56178 from adk3798/cephadm-pub-net-config-check-out-of-date

mgr/cephadm: refresh public_network for config checks before checking

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56175 from asm0deuz/bootstrap_no_container

cephadm: bootstrap fails if no container engine installed

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56147 from adk3798/stopped-daemon-shell

cephadm: fix `cephadm shell --name <daemon-name>` for stopped/failed daemon

Reviewed-by: John Mulligan <jmulligan@redhat.com>

rgw/notification: add rgw notification specific debug log subsystem.

decorate the events with event details while logging.

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>

Merge pull request #56143 from insatomcat/timemaster

cephadm.py: add timemaster to timesync services list

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #56075 from phlogistonjohn/jjm-enable-black-cephadm

cephadm: enforce black formatting for cephadmlib

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #55883 from thegreenbear/ingress-haproxy-bind-ipv4v6

Ingress service: make HAProxy to listen on IPv4 and IPv6 …

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #55874 from adk3798/only-bind-to-network

mgr/cephadm: allow grafana and prometheus to only bind to specific network

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55645 from ronen-fr/wip-rf-nextobj

osd: improved get_objects_by_prefixes() ergonomics

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

qa/suite/rbd/nvmeof: Deploy multiple gateways and namespaces

1. Deploy 2 gateways on different nodes, then check for multi-path.
    To add another gateway, only "roles" need to be changed in job yaml.
2. Create "n" nvmeof namespaces, configured by 'namespaces_count'
3. Rename qa/suites/rbd/nvmeof/cluster/fixed-3.yaml to fixed-4.yaml
    which contains 2 gateways and 2 initiators.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

qa/suites/fs/nfs: use standard health ignorelist

Fixes: https://tracker.ceph.com/issues/64987
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/cephfs: ignorelist clog of MDS_UP_LESS_THAN_MAX

Fixes: https://tracker.ceph.com/issues/64986
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #56109 from anthonyeleven/improve-monitoring

doc/cephadm/services: Improve monitoring.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55877 from adk3798/cephadm-pull-usage

cephadm: improve cephadm pull usage message

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55615 from rhcs-dashboard/grafana-mount-dashboards

mgr/cephadm: mount grafana dashboards from the path

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #52760 from thotz/add-healthchecker-timeout-haproxy

cephadm: adjust the ingress ha proxy health check interval

Reviewed-by: Adam King <adking@redhat.com>

mgr/dashboard: fix retention add for subvolume

- Added parameters for subvolume and subvolume group when adding a new
snap schedule.
- Added call to remove retention policies when removing a snap schedule
  in case it is the last one with same path

Fixes: https://tracker.ceph.com/issues/64524
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

vstart: Ganesha should not be started in DEBUG mode

Currently vstart script, deploy NFS Ganesha in debug mode. Enabling
DEBUG mode for Ganesha leads to logging lot of debug messages, which may
not be required all the time. One can enable DEBUG mode on need basis.
Hence removing the default DEBUG mode.

Signed-off-by: Sachin Punadikar <sachin.punadikar@ibm.com>

Merge pull request #56189 from rhcs-dashboard/fix-64913-main

mgr/dashboard: Allow DELETE method api request, X-TOTAL-COUNT header in CORS config in dashboard

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #56250 from myoungwon/wip-laod-metadata-during-gc

crimson/os/seastore: cache metadata during trimming to prevent from disk read

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #56204 from zmc/do-cmake-args

do_cmake.sh: Use value of ARGS at runtime

osd/SnapMapper: add common logs

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit f302be018ebef365d8876d14185c4bdb246f22d8)

osd: improved get_objects_by_prefixes() ergonomics

Improved call signatures for get_next_objects_to_trim() &
get_objects_by_prefixes().

Also: as as get_next_objects_to_trim() has only a single
failure mode, we should not try to handle two distinct failures
in its callers' code.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55846 from athanatos/sjust/for-review/wip-crimson-coroutine-support

crimson: add coroutine support for errorated and interruptible futures

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #56034 from tchaikov/wip-msgr-dont-panic-at-hot-unplug

msg: do not abort if driver->del_event() returns -ENOENT

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55896 from ronen-fr/wip-rf-compactref

osd: avoid calling a virtual function in a ctor

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #55872 from aclamk/wip-aclamk-bs-fix-unused

os/bluestore: Mute warnings

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #54492 from Matan-B/wip-matanb-op-thread-timeout

qa/suites/rados/verify/validater/valgrind: increase op thread timeout

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #56240 from zdover23/wip-doc-2024-03-17-rados-config-common

doc/rados: update common.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

qa/rgw: ignore POOL_APP_NOT_ENABLED in rgw/lua subsuite

> cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/cephadm/services: Improve monitoring.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #56248 from nbalacha/wip-nbalacha-rbd-doc-man

doc/rbd: minor changes to the rbd man page

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/rbd: minor changes to the rbd man page

Fixes typos and grammar for some commands. Adds
additional details for some commandds.

Signed-off-by: N Balachandran <nibalach@redhat.com>

crimson/os/seastore: cache metadata during trimming to prevent from disk read

I encountered continous disk reads during trimming even though there are sufficient
cache available, in 4K random write test with RBM (RBD).
This is because metadata is note cached if its source is background transaction
within touch_extent(). So, seastore, including the trimming process,  needs to
constantly retrieve metadata (e.g., BACKREF_LEAF).

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/cached_extent: add comments to elaborate why
MUTATION_PENDING and under-io extents are "stable"

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/btree: check for reserved ptrs when determining
children stability

Fixes: https://tracker.ceph.com/issues/64957
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #56223 from guits/tracker_64951

node-proxy: support more Location value formats

Merge pull request #56247 from nbalacha/wip-nbalacha-rbd-doc-2

docs/rbd: fix typos in the rbd basic cmds documentation

Merge pull request #56198 from ivoalmeida/fix-nfs-attach-cmd

mgr/dashboard: fixed nfs attach command

Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: securely store remote cluster token

Instead of using the localStorage use cookies for storing the token more
securely

Fixes: https://tracker.ceph.com/issues/64958
Signed-off-by: Nizamudeen A <nia@redhat.com>

docs/rbd: typo in "retrieving image information"

Replace "for the image" with "of the image".

Signed-off-by: N Balachandran <nibalach@redhat.com>

docs/rbd: replace introspect with inspect

Replace "introspect" with "inspect" in the rbd basic commands
description.

Signed-off-by: N Balachandran <nibalach@redhat.com>

docs/rbd: fix typo in arg name

Replace "{image-}" with "{image-id}" in the "rbd trash rm"
command description.

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge pull request #56217 from xxhdx1985126/wip-seastore-remap-bug-fixes

test/crimson/sesastore/test_block: add constructor to construct

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

mds/quiesce-db: always clear the db if a membership is lost

Fixes: https://tracker.ceph.com/issues/64912
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

doc/cephdam: document limiting passwordless sudo commands

Based on the previous commits making the remote executables auditable
and explicit, document the admin's ability to restrict password-less
sudo access to only the set of commands cephadm actually uses.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: add a simple unit test for RemoteCommand class

Converting a remote command to something that other libs uses requires
converting the enum to a string. Python behavior in the area varies
across versions so add a unit test that verifies the conversion
behaves as intended.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: add test to ensure the list of remote commands is known

Add a test file to help ensure the audit of remote commands is kept up
to date.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: make remote command execution auditable

Update ssh.py and other code using it to only allow commands wrapped
in particular python types as executables on the remote hosts.
By using a specific type for remote executables we make the code more
auditable, avoiding the possibility of executing arbitrary strings
as commands with sudo. This is all enforced by mypy's type checking.

The result is a list of commands that the cephadm mgr module may
execute on a remote host using sudo:
```
$ git ls-files -z | xargs -0 grep 'RemoteExecutable(' -d skip -h | grep
-v '(str)' | sed -e 's/.*RemoteExecutable(//' -e 's/)//' -e 's/,$//'
'which'
'/usr/bin/cephadm'
python
'chmod'
'ls'
'sysctl'
'chown'
'mkdir'
'mv'
'touch'
'rm'
'true'
```
Note that *python* is special as it is based on the output of which and
may vary from OS to OS. The quoted items are used exactly as named.
Only the binary at `/usr/bin/cephadm` _or_ the dynamically discovered
python3 binary will be used. This depends on a configuration option for
the cephadm module.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

doc/rados: update common.rst

Clarify some overwritten paragraphs in
doc/rados/configuration/common.rst. Cf.
https://github.com/ceph/ceph/pull/50795/files#r1153897231.

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56182 from zdover23/wip-doc-2024-03-14-glossary-librados

doc/glossary: add "librados" entry

Merge PR #56218 into main

* refs/pull/56218/head:
qa/crontab: use historically normal priorities for nightlies
doc/dev: update teuthology priority guidance

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #56177 from lbausch/docs-typo-replace-osd

doc: cephadm/services/osd: fix typo

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>

mgr/cephadm: refresh public_network for config checks before checking

The place it was being run before meant it would only grab the
public_network setting once at startup of the module. This meant
if a user changed the setting, which they are likely to do if they
get the warning, cephadm would ignore the change and continue
reporting that the hosts don't match up with the old setting
for the public_network. This moves the call to refresh the
setting to right before we actually run the checks. It does
mean we'll do the `ceph config dump --format json` call
each serve loop iteration, but I've found that only tends
to take a few milliseconds, which is nothing compared to
the time to refresh other things we check during the serve
loop.

I additionally modified the use of this option to use
the attribute on the mgr, rather than calling
`get_module_option`. This was just to get it more in
line with how we tend to handle other config options

Fixes: https://tracker.ceph.com/issues/64902
Signed-off-by: Adam King <adking@redhat.com>

cephadm: fix `cephadm shell --name <daemon-name>` for stopped/failed daemon

This previously would always try to use 'podman
inspect' on the running container of the daemon,
but this doesn't work if the daemon is stopped
or failed. Doing this for stopped/failed daemons
is a valid use case as we recommend cephadm shell
with --name for running debugging tools (often
for OSDs)

Fixes: https://tracker.ceph.com/issues/64879
Signed-off-by: Adam King <adking@redhat.com>

cephadm: allow list_daemons for only a specific daemon

At the moment, my thoughts are to use this internally
in the binary for when we need infor from list_daemons
but only for a specific daemon. I could also see wanting
this just on the command line to get info on a certain
daemon, so I've added it as a flag for `cephadm ls` as well

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #55796 from kchheda3/wip-fix-filterrule-json

rgw/notification: Fix the notification FilterRule to emit unique key name in json & Log successful delivery of notification event.

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #56179 from cbodley/wip-doc-rgw-notification_v2

doc/rgw: document the notification_v2 zone feature

Reviewed-by: Krunal Chheda <kchheda3@bloomberg.net>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>