Merge pull request #58084 from asm0deuz/mgmt_gtw_e2e

mgmt-gateway: add e2e testing

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60328 from cbodley/wip-mstart-doc

rgw: document and label mstart.sh and related scripts

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #58486 from rishabh-d-dave/ignore-osd-down

qa/cephfs: ignore when specific OSD is reported down during upgrade

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59712 from rishabh-d-dave/mgr-vol-clone-stats-snap

qa/cephfs: test clone progress reporter after subvol is deleted but...

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #59848 from rishabh-d-dave/mgr-vol-wakeup-timeout

mgr/vol: use pre-defined timeout period instead of hardcoded value

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #60008 from rishabh-d-dave/mgr-util-improve

mgr/mgr_util: improvements for RTimer

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #60027 from rishabh-d-dave/mgr-vol-progbar-id

mgr/vol: don't define progress bar ID repeatedly

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59782 from aclamk/wip-aclamk-fix-67596-allocmap

os/bluestore: Fix ceph-bluestore-tool allocmap command

mgmt-gateway: add e2e testing

Add mgmt-gateway teuthology test scenarios

Signed-off-by: Teoman ONAY <tonay@ibm.com>

Merge pull request #60202 from athanatos/sjust/wip-crimson-io

crimson: replace do_osd_ops* with simpler, more general mechanism

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #60223 from guits/replace-osd-enhancement

orch: disk replacement enhancement

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60347 from avanthakkar/rm-smb-dump-everything

mgr/smb: rm all `smb dump` commands

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60167 from jmolmo/add_daemon_health_metric

exporter: New metric for report ceph daemons health

Merge pull request #59514 from kevinzs2048/add-uadk-doc

Compressor: Add UADK document

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60173 from rhcs-dashboard/fix-lc

mgr/dashboard: fix lifecycle issues

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60260 from ShwetaBhosale1/new_cephadm_command_to_list_default_images

cephadm: Added cephadm command to list all the default images

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55808 from neesingh-rh/wip-63700

cephfs-shell: fixing cephfs-shell test failures

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard: fix lifecycle issues

Fixes: https://tracker.ceph.com/issues/68434
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #58292 from neesingh-rh/wip-vol-method-name-change

mgr/vol : shortening the name of helper method

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

cephadm: Added new cephadm command to list all the default images

Fixes: https://tracker.ceph.com/issues/68438
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>

Merge pull request #60058 from mchangir/log-save-thread-name-in-log-entries

log: save/fetch thread name infra

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #60317 into main

* refs/pull/60317/head:
.github: detect GPL license in PRs

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #59996 into main

* refs/pull/59996/head:
mailmap: Add dashboard new joinees in maps

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #60053 from rkachach/fix_issue_68315

mgr/cephadm: move Grafana's subpath handling logic to grafan config

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/smb: rm all `smb dump` commands

Fixes: https://tracker.ceph.com/issues/68545
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #59680 from adk3798/bootstrap-apply-spec-fail-continue

cephadm: add ability to continue on failure when applying multiple specs

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60077 from adk3798/cephadm-ganesha-conf-allow-set-io-flusher-fail

mgr/cephadm: add "allow_set_io_flusher_fail = true;" to ganesha conf

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60300 from avanthakkar/add-smb-ok-to-stop

mgr/cephadm: add ok_to_stop func for smb service

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60112 from anoopcs9/cephadm-smb-add-ctdb-loglevel

cephadm/smb: Add a provision to specify ctdb log level

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60263 from afreen23/wip-api-update

mgr/dashboard: Adapt gateway group changes in nvmeof UI

Reviewed-by: Afreen Misbah <afreen@ibm.com>

exporter: New metric for report ceph daemons health

Ceph exporter provide metrics to report ceph daemons communication health using
 the admin socket

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2146728
       https://tracker.ceph.com/issues/68428

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #60306 from cbodley/wip-osdc-rm-completion

osdc: remove unused overloads for async::Completion

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

orch: disk replacement enhancement

This introduces a new `ceph orch device replace` command in order to
improve the user experience when it comes to replacing the underlying
device of an OSD.

Fixes: https://tracker.ceph.com/issues/68456
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: address mypy errors in disk.py

typical error:

```
ceph_volume/util/disk.py:1374: error: Incompatible types in assignment (expression has type "Optional[str]", variable has type "str")  [assignment]
```

This commits addresses it.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: address test_activate_dmcrypt_tpm

This mocks the call to `luks_close()`, otherwise this test
fails when run on a system where `cryptsetup` isn't available.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge PR #59503 into main

* refs/pull/59503/head:
client: Resolve symlink from dirfd for empty pathname
client: Fix symlink open with O_PATH and O_NOFOLLOW

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #60055 from guits/tracker_68312

ceph-volume: drop unnecessary call to `get_single_lv()`

Merge pull request #60291 from rhcs-dashboard/NFS-subvol-list

mgr/dashboard: The subvolumes are missing from the dropdown menu on the "Create NFS export" page

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #60256 from joscollin/wip-doc-update-key-idea

doc: update Key Idea in cephfs-mirroring.rst

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60222 from rhcs-dashboard/group-name-bug

mgr/dashboard: fix group name bugs in the nvmeof API

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60060 from guits/move-to-quay-io

cephadm: pull container images from quay.io

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #60241 from rhcs-dashboard/sync-policy-display-zonegroup

mgr/dashboard: sync policy's in Object >> Multi-site >> Sync-policy does not show the zonegroup to which policy belongs to

Reviewed-by: Afreen Misbah <afreen@ibm.com>

mailmap: Add dashboard new joinees in maps

- Afreen Misbah
- Dnyaneshwari Talwekar
- Naman Munet
- Prachi Goel
- Puja Shahu

Signed-off-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60125 from cbodley/wip-doc-rgw-index-filenames

doc/dev/radosgw: update paths that moved under src/rgw/driver/rados/

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #60128 from cbodley/wip-doc-rgw-multisite-secondary-commit

doc/radosgw/multisite: fix Configuring Secondary Zones -> Updating the Period

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

mgr/dashboard: Adapt gateway group changes in nvmeof UI

- Added gateway group param in namespace request - GET, POST, PATCH, DELETE
- Added gateway group param in Listeners request - GET
- Added gateway group param in Initiators - GET, POST, DELETE

Fixes https://tracker.ceph.com/issues/68510

Signed-off-by: Afreen Misbah <afreen@ibm.com>

rgw: add mstart-related scripts to CODEOWNERS and labeler

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: document mstart.sh and related scripts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc: update Key Idea in cephfs-mirroring.rst

Updates the snapdiff feature and it's url.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #59706 from Matan-B/wip-matanb-cleanup-snapmap

osd/PG: make use of SnapMapper::update_snap_map

Merge pull request #59844 from spuiuk/mgr_smb_add_public_addrs_cli

mgr/smb: accept public_addrs on cli when creating cluster

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60238 from anoopcs9/cephadm-add-net-raw-ctdbd

cephadm/smb: Add NET_RAW capability to deploy ctdbd

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60221 from rhcs-dashboard/fix-68461-main

mgr/cephadm: Update multi-site configs before deploying  daemons on rgw service create

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #60302 into main

* refs/pull/60302/head:
doc/governance: update my CSC email
mailmap: add my ibm email

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

.github: detect GPL license in PRs

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mgr/cephadm: disabling nginx buffering for grafana location

Disabling Nginx buffering for Grafana, as it may lead to errors or
delays while loading the main Grafana page, particularly when
receiving JavaScript files.

Fixes: https://tracker.ceph.com/issues/68315
Signed-off-by: Redouane Kachach <rkachach@ibm.com>

log: thread name save/fetch infra

* pthread name is saved in a thread_local storage
* the thread_local name is copied into Entry object's ctor
* Log::dump_recent() reads the thread name from the Entry
  object's data member when dumping logs

Fixes: https://tracker.ceph.com/issues/50743
Signed-off-by: Milind Changire <mchangir@redhat.com>

client: Resolve symlink from dirfd for empty pathname

man readlinkat(2)[1] points at a special case for readlinkat() syscall
as follows:

. . .
Since Linux 2.6.39, pathname can be an empty string, in which case the
call operates on the symbolic link referred to by dirfd (which should
have been obtained using open(2) with the O_PATH and O_NOFOLLOW flags).
. . .

man open(2)[2] further explains the need for such a special case when
a symlink is opened with O_PATH and O_NOFOLLOW:

. . .
If  pathname is a symbolic link and the O_NOFOLLOW flag is also
specified, then the call returns a file descriptor referring to the
symbolic link.  This file descriptor can be used as the dirfd argument
in calls to fchownat(2), fstatat(2), linkat(2), and readlinkat(2) with
an empty pathname to have the calls operate on the symbolic link.
. . .

Accordingly have a check to resolve symlinks out of dirfd when empty
pathnames are encountered within readlinkat(). In addition to that
match the standard file system behavior to return ENOENT instead of
EINVAL when the inode pointed to by dirfd is not a symbolic link with
empty pathnames.

Fixes: https://tracker.ceph.com/issues/67833
[1] https://www.man7.org/linux/man-pages/man2/readlinkat.2.html
[2] https://www.man7.org/linux/man-pages/man2/open.2.html

Signed-off-by: Anoop C S <anoopcs@cryptolab.net>

client: Fix symlink open with O_PATH and O_NOFOLLOW

man open(2)[1] says the following for O_PATH:

. . .
If  pathname is a symbolic link and the O_NOFOLLOW flag is also
specified, then the call returns a file descriptor referring to the
symbolic link.  This file descriptor can be used as the dirfd argument
in calls to fchownat(2), fstatat(2), linkat(2), and readlinkat(2) with
an empty pathname to have the calls operate on the symbolic link.
. . .

symlink check within may_open() failed to consider the O_PATH flag
resulting in a ELOOP error to the client. In order to return a valid
file descriptor we introduce a check for the presence of O_PATH in
the client provided flags.

Fixes: https://tracker.ceph.com/issues/67833
[1] https://www.man7.org/linux/man-pages/man2/open.2.html

Signed-off-by: Anoop C S <anoopcs@cryptolab.net>

Merge pull request #60212 from VallariAg/nvmeof-labeler

labeler: add nvmeof labelers

mgr/dashboard: The subvolumes are missing from the dropdown menu on the "Create NFS export" page

Fixes: https://tracker.ceph.com/issues/68519
Signed-off-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>

crimson: remove unused OpsExecuter::rollback_obc_if_modified overload

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: OpsExecuter no longer needs to be a lw shared ptr

ClientRequest and InternalClientRequest can declare them
as auto variables.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: remove now unused PG::do_osd_ops* and log_reply

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: clarify ops_executer.h comment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: convert InternalClientRequest::do_request to use *_executer rather than do_osd_ops*

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: factor out InternalClientRequest::do_process

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: switch ClientRequest::do_request to use *_executer rather than do_osd_ops

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: introduce PG::run_executer,submit_executer

These are intended to replace do_osd_ops*.  The implementation
is simpler and does not involve passing success and failure
callbacks.  It also moves responsibility for dealing with
the MOSDOpReply and client related error handling over to
ClientRequest.

do_osd_op* will be removed once users are switched over.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: PG::submit_error_log returns eversion_t rather than optional

It seems like the motivation here was to allow do_osd_ops_execute to
communicate that it didn't submit an error log by making
maybe_submit_error_log a std::optional<eversion_t>.  However,
submit_error_log itself always returns a version.  Fix submit_error_log
and compensate in do_osd_ops_execute.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: introduce rollback_obc_if_modified without an error argument

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: futures from flush_changes_n_do_ops_effects must not fail

The return signature previously suggested that the second future
returned could be an error.  This seemed necessary due to how
effects are handled:

template <typename MutFunc>
OpsExecuter::rep_op_fut_t
OpsExecuter::flush_changes_n_do_ops_effects(
  const std::vector<OSDOp>& ops,
  SnapMapper& snap_mapper,
  OSDriver& osdriver,
  MutFunc mut_func) &&
{
...
    all_completed =
      std::move(all_completed).then_interruptible([this, pg=this->pg] {
      // let's do the cleaning of `op_effects` in destructor
      return interruptor::do_for_each(op_effects,
        [pg=std::move(pg)](auto& op_effect) {
        return op_effect->execute(pg);
      });

However, all of the actual execute implementations (created via
OpsExecuter::with_effect_on_obc) return a bare seastar::future and
cannot fail.

In a larger sense, it's actually critical that neither future returned
from flush_changes_n_do_ops_effects may fail -- they represent applying
the transaction locally and remotely.  If either portion fails, there
would need to be an interval change to recover.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: OpsExecutor::flush_clone_metadata no longer needs to return a future

Snapmapper updates happen during log commit now.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: remove the eagain error from PG::do_osd_ops

The idea here is that PG::do_osd_ops propogates an eagain after starting
a repair upon encountering an eio to indicate that the op should restart
from the top of ClientRequest::process_op.

However, InternalClientRequest's handler for this error simply ignores
it.  ClientRequest's handling, while superficially reasonable, doesn't
actually work.  Re-calling process_op would mean reentering previous
stages.  This is problematic for at least a few reasons:
1. Reentering a prior stage with the same handler doesn't actually work
   since the corresponding event entries will already be populated.
2. There might be other ops on the same object waiting on the process
   stage.  They'd need to be sent back as well in order to preserve
   ordering.

Because this mechanism doesn't really seem to be fully baked, let's
remove it for now and try to reintroduce it later after
do_osd_ops[_execute] are a bit simpler.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: move pipelines to osd_operation.h

Each of the two existing pipelines are shared across multiple
ops.  Rather than defining them in a specific op or in
osd_operations/common/pg_pipeline.h, just declare them in
osd_operation.h.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: eliminate get_obc stage

f90af12d introduced check_already_complete_get_obc to replace get_obc,
but left get_obc and didn't update the other users.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../internal_client_request: convert with_interruption to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../internal_client_request: factor out with_interruption

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../internal_client_request: remove unnecessary system_shutdown guard

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: fix typo OpsExecutor->OpsExecuter

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: add missing field to SUBLOGDPPI and LOGDPPI

SUBLOGDPPI and LOGDPPI need an extra {} for the interrupt_cond.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: remove watchers upon object deletion

Fixes: https://tracker.ceph.com/issues/68538
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #60301 into main

* refs/pull/60301/head:
doc/governance: add new CSC members

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

osdc: remove unused overloads for async::Completion

ea67f3dee2a3f8fcdcbb0bc0e80e38ec70378f05 switched to
asio::any_completion_handler<> for completions, but left some converting
overloads behind for compatibility. none of those overloads appear to be
used, so remove them

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/governance: update my CSC email

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

mailmap: add my ibm email

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

doc/governance: add new CSC members

Congratulations!

Election: https://vote.heliosvoting.org/helios/elections/f276a15a-84c5-11ef-a0e4-b69e035002b0/view
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/cephadm: add ok_to_stop func for smb service

Fixes: https://tracker.ceph.com/issues/68527
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #60158 from aclamk/wip-aclamk-bluefs-truncate-allocations

os/bluestore: Make truncate() drop unused allocations

Merge pull request #59914 from xxhdx1985126/wip-68174

crimson/osd/pg: remove snapmapper objects when eventually removing collections at the last moment of pg deleting, just as pg meta objects

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #60169 from lee-j-sanders/wip-ljs-rmcosbench

qa/suites/tasks/cbt.py: Deprecating cosbench from Teuthology in preparation for deletion of cosbench from CBT

Merge pull request #59888 from phlogistonjohn/jjm-mypy-more

prepare mypy checking for newer python (3.12)

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #59942 from ronen-fr/wip-rf-store2-steps

osd/scrub: separate shallow vs deep errors storage

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #60242 from zdover23/wip-doc-2024-10-10-SubmittingPatches-backports

doc: SubmittingPatches-backports - remove backports team

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge PR #60219 into main

* refs/pull/60219/head:
qa/cephfs: update earmark values to valid ones in test_volumes.py

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard: sync policy's in Object >> Multi-site >> Sync-policy, does not show the zonegroup to which policy belongs to

Fixes: https://tracker.ceph.com/issues/68355
Fixes Includes: Added default zonegroup name with the sync policy details

Signed-off-by: Naman Munet <namanmunet@li-ff83bccc-26af-11b2-a85c-a4b04bfb1003.ibm.com>

Merge pull request #60247 from leonidc/101024-fix-no-listeners

mon/nvmeofgw*: fix HA usecase when gateway has no listeners: behaves …

Merge pull request #59905 from rhcs-dashboard/osd-perf-impr

mgr/dashboard: introduce server side pagination for osds

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #60259 from afreen23/wip-listener-del

mgr/dashboard: Fix listener deletion

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59572 from xxhdx1985126/wip-67874

crimson/osd/backfill_state: add the object to be pushed in the peer missing set of PeeringState

Reviewed-by: Samuel Just <sjust@redhat.com>