crimson/os/seastore/async_cleaner: add comments to clarify what kind of
extents would be considered dead by the cleaner

This could prevent future developers from rendering the cleaner abnormal
unexpectedly

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #55878 from athanatos/sjust/wip-seastar-module

crimson: update seastar submodule to fix prometheus build error

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/.../interruptible_future: remove SEASTAR_CONCEPT guard

Seastar commit 8dc3398a removed this macro, no longer necessary.

Signed-off-by: Samuel Just <sjust@redhat.com>

src/seastar: update seastar submodule to fix prometheus build failure

Fixes: https://tracker.ceph.com/issues/64589
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #55856 from dmick/wip-workflow-update

.github/workflows/create-backport-trackers.yml: update actions

Merge pull request #55869 from zdover23/wip-doc-2024-03-01-install-manual-radosgw

doc/install: add manual RADOSGW install procedure

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

.github/workflows/create-backport-trackers.yml: update versions of actions

Getting warning about node16 being deprecated.  The workflow doesn't use node
directly, but through the external actions.  Moving to node20 requires
changing setup-python version; Bhacaz/checkout-files is deprecated and
recommends actions/checkout.

Signed-off-by: Dan Mick <dmick@redhat.com>

doc/install: add manual RADOSGW install procedure

Add a manual RADOSGW installation procedure to
doc/install/manual-deployment.rst. This procedure was developed by Janne
Johansson and reported to the ceph-users mailing list on 29 Jan 2024
here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/LB3YRIKAPOHXYCW7MKLVUJPYWYRQVARU/

Co-authored-by: Janne Johansson <icepic.dz@gmail.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55797 from ajarr/wip-64574

qa: add diff-continuous and compare-mirror-image tests to rbd and krbd suites respectively

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #54767 from climb-mountain123/worm_multipart

src/rgw: fix for the multipart interface in the WORM function

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55719 from phlogistonjohn/jjm-teuth-tasks-cephadm-jt

qa/tasks/cephadm: add generic templating where subst_vip was used

Reviewed-by: Adam King <adking@redhat.com>

rgw: remove unused object lock stuff from CompleteMultipart

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55817 from rkhudov/src-test-common-test_hobject-remove-constexpr

src/test/common/test_hobject: remove constexpr

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55727 from cbodley/wip-64549

rgw/auth: do_aws4_auth_completion() catches exceptions

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #55855 from xxhdx1985126/wip-seastore-interface

crimson/os/seastore: adjust SeaStore::_omap_set_kvs() params

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: adjust SeaStore::_omap_set_kvs() params

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #55834 from zdover23/wip-doc-2024-02-29-dev-internals

doc/dev: edit internals.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55835 from zdover23/wip-doc-2024-02-29-glossary-mds

doc/glossary: improve "MDS" entry

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: improve "MDS" entry

Improve the entry for "MDS" in doc/glossary.rst by linking to the
"ceph-mds" man page and mentioning the relationship between clients and
MDS (or MDSes).

Signed-off-by: Zac Dover <zac.dover@proton.me>

qa/suites: add diff-continuous and compare-mirror-image tests

... to rbd and krbd suites respectively.

This allows the compare-mirror-image tests introduced in ea3a567
to be run against various kernel branches, e.g., testing branch.
And allows diff_continuous test in rbd_suite to run against distro
kernel.

Fixes: https://tracker.ceph.com/issues/64574
Signed-off-by: Ramana Raja <rraja@redhat.com>

qa/suites/rbd: rename nbd folder to device folder

Signed-off-by: Ramana Raja <rraja@redhat.com>

rgw: don't overwrite target attrs checking mpu info

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

src/rgw: fix for the multipart interface in the WORM function

1. Save the WORM configuration information in the initialization chunk information for use when merging chunks.
2. Support x-amz-bypass-governance-retention when merging chunks.

Fixes: https://tracker.ceph.com/issues/63724
Signed-off-by: daijufang <daijufang_yewu@cmss.chinamobile.com>

Merge pull request #55594 from ifed01/wip-ifed-fix-64443

test/store_test: fix DeferredWrite test when prefer_deferred_size=0

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

qa/tasks: replace uses of subst_vip with new templating function

Signed-off-by: John Mulligan <jmulligan@redhat.com>

qa/tasks: add templating functions to cephadm module

Add functions to cephadm.py that will be later used to template
strings within the yaml files in the cephadm suites. This will be used
to replace the specific subst_vip call with generic calls that let
tests access "any" variables stored on the test ctx.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

qa/tasks: fix VIPs log line

While testing my previous patches were correct I noticed that the string
here was logged exactly as written, and was thus pretty useless.  This
was probably meant to be an f-string. So make it one. Also get rid of
the unnecessary map call, the list and IP address type can repr
themselves just fine IMO.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

qa/tasks: make _map_vips private

Nothing outside of vip.py called map_vips, so let us make sure this
is considered a private function and prefix it with the underscore.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

qa/tasks: change map_vips to raise exceptions instead of returning None

None of the callers of map_vips ever checks for a None return. So
instead of handling any error conditions it would always just blow
up with a semi-obscure TypeError. Convert the function to always
raise an exception (one that tries to breifly explain the condition)
when something goes wrong. I also take the opportunity to make
more clearer logging and reduce an indentation level.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55534 from clwluvw/cephadm-device-class

cephadm: remove restriction for crush device classes

doc/dev: edit internals.rst

Edit doc/dev/internals.rst. Make a few sentences pithier.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55806 from xxhdx1985126/wip-seastore-clone-attrs-omaps

crimson/os/seastore: copy attrs and omaps when cloning objects

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #55771 from rhcs-dashboard/e2e-fix

mgr/dashboard: fix roles page for roles without policies

Reviewed-by: afreen23 <NOT@FOUND>

Merge pull request #55510 from pcuzner/add-nvmeof-alerts

ceph-mixin: Update mixin to include alerts for the nvmeof gateway(s)

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #55822 from dmick/wip-release-build-prerelease-update

doc/dev/release-process.rst: modify for prerelease mechanism

doc/dev/release-process.rst: modify for prerelease mechanism

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #55710 from rhcs-dashboard/nvmeof-prometheus-endpoint

cephadm/nvmeof: scrape nvmeof prometheus endpoint

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Paul Cuzner <pcuzner@ibm.com>

Merge pull request #55600 from adk3798/no-agent-workunits

qa/cephadm: don't test certain workunits with agent

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55588 from adk3798/rm-cluster-podman-auth

cephadm: rm podman-auth.json if removing last cluster

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55788 from rkachach/fix_issue_64596

doc: adding documentation for secure monitoring stack configuration

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #55620 from adk3798/cephadm-timeout-cancelled-error

mgr/cephadm: catch CancelledError in asyncio timeout handler

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55650 from adk3798/cephadm-ceph-exporter-sock-dir

cephadm: create ceph-exporter sock dir if it's not present

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55531 from roysahar/roys/wip-nvmeof-new-mount-for-logs

nvmeof: Add mount for log location

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Aviv Caro <Aviv.Caro@ibm.com>

src/test/common/test_hobject: remove constexpr

Signed-off-by: rkhudov <rhudov@gmail.com>

Merge pull request #55273 from pcuzner/add-nvmeof-to-autotune

cephadm: Add nvmeof to autotuner calculation

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #53840 from rhcs-dashboard/add-health-check-haproxy

mgr/cephadm is not defining haproxy tcp healthchecks for Ganesha

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #55812 from ivoalmeida/add-snap-schedule-repeat-frequencies

mgr/dashboard: add snap schedule M, Y frequencies

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #55207 into main

* refs/pull/55207/head:
mds: check file layout in mknod

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: adding documentation for secure monitoring stack configuration
Fixes: https://tracker.ceph.com/issues/64596
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #55794 from kamoltat/wip-ksirivad-ceph-backport-script

src/scripts/ceph-backport.sh: Add PR checklist
Reviewed by: Laura Flores <lfores@ibm.com>

Merge pull request #55809 from ivoalmeida/fix-snap-schedule-month

mgr/dashboard: fix snap schedule date format

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: add snap schedule M, Y frequencies

Added M and Y (month and year) repeat frequencies to snap schedule
create form.

Fixes: https://tracker.ceph.com/issues/64614
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

mgr/dashboard: fix snap schedule date format

Fixes: https://tracker.ceph.com/issues/64613
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

crimson/os/seastore: copy attrs and omaps when cloning objects

At present, we just copy attrs and omaps one by one, which is not
efficient but very important in terms of functionality especially for
the teuthology tests

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

mgr/dashboard: fix roles page for roles without policies

If there is no permission policies attached to a role, it fails the
UI with an exception.

Fixes: https://tracker.ceph.com/issues/64588
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55757 from rhcs-dashboard/replace-piechart-panel-plugin-grafana

mgr/dashboard: replace piechart plugin charts with native pie chart panel

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #52258 into main

* refs/pull/52258/head:
client: check mds down status bofore getting mds_gid_t from mdsmap
mgr/dashboard: allow sending back error status code fetching clients fails

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #55772 from xxhdx1985126/wip-seastore-attr-related-bug-fixes

crimson/os/seastore: two attrs related bug fixes

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #55782 from zdover23/wip-doc-2024-02-27-rados-operations-add-or-rm-mons

doc/rados: add "change public network" procedure

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #53222 from myoungwon/wip-62167

osd: check if adjacent clones are unreadable when rollback is called.
Reviewed-by: athanatos

Merge pull request #55147 from Matan-B/wip-matanb-skip-past-interval-cheeck

osd/PeeringState: introduce osd_skip_check_past_interval_bounds
Reviewed-by athanatos

Merge pull request #55227 from Svelar/wip-ci-clang

ARM CI: clang should >= 12 for supporting C++20
Reviewed by: Laura Flores <lajefl@gmail.com>

src/scripts/ceph-backport.sh: Add PR checklist

Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #54872 from amathuria/wip-amat-mclock-adjust-deletion-cost

osd: Change PG Deletion cost for mClock

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #52891 from kamoltat/wip-ksirivad-fix-59172

qa/tasks/ceph_manager.py: Rewrite test_pool_min_size

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55760 from cbodley/wip-64568

test/rgw: increase timeouts in unittest_rgw_dmclock_scheduler

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

mds: check file layout in mknod
Fixes: https://tracker.ceph.com/issues/64061
Signed-off-by: Xue Yantao <xueyantao2114@163.com>

Merge pull request #55752 from rhcs-dashboard/fix-vol-create

mgr/dashboard: fix volume creation with multiple hosts

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/rados: add "change public network" procedure

Add a procedure to /doc/rados/operations/add-or-rm-mons.rst that
explains how to change the public_network in a Ceph cluster deployed
with cephadm. This procedure was developed by Eugen Block, and can be
seen in its original form here:
https://heiterbiswolkig.blogs.nde.ag/2024/02/22/cephadm-change-public-network/

Co-authored-by: Eugen Block <eblock@nde.ag>
Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore: also remove xattrs when removing onodes

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

mgr/dashboard: replace piechart plugin charts with native pie chart
panel

Fixes: https://tracker.ceph.com/issues/64579
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

crimson/os/seastore: get_attrs should return all attrs

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: attr related debug outputs

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge PR #52859 into main

* refs/pull/52859/head:
qa: test cases to make sure invalid paths don't get updated
mgr/nfs: use helper to validate cephfs path
mgr/nfs: validate path before updating a cephfs export
mgr/nfs: add a helper to validate cephfs path

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #54687 into main

* refs/pull/54687/head:
mds: allow lock state to be LOCK_MIX_SYNC in replica for filelock

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #55435 into main

* refs/pull/55435/head:
qa: Fix fs/full suite

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55751 from cyx1231st/wip-fix-test-socket

test/crimson/test_socket: complete write-side errors in test_unexpected_down()

Reviewed-by: Samuel Just <sjust@redhat.com>

ceph-mixins: Update MIB to include nvmeof notification

Signed-off-by: Paul Cuzner <pcuzner@ibm.com>

ceph-mixins: Add test cases for nvmeof alerts

Signed-off-by: Paul Cuzner <pcuzner@ibm.com>

ceph-mixins: nvmeof alerts added

Signed-off-by: Paul Cuzner <pcuzner@ibm.com>

ceph-mixins: Add nvmeof alerts

Signed-off-by: Paul Cuzner <pcuzner@ibm.com>

mgr/cephadm: catch CancelledError in asyncio timeout handler

Specifically, concurrent.futures.CancelledError. At least on
python 3.9, this error can be raised when certain commands
being run asynchronously fail. Not catching this results in
the whole cephadm module crashing with something like

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/utils.py", line 94, in do_work
    return f(*arg)
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 267, in refresh
    r = self._refresh_facts(host)
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 370, in _refresh_facts
    val = self.mgr.wait_async(self._run_cephadm_json(
  File "/usr/share/ceph/mgr/cephadm/module.py", line 671, in wait_async
    return self.event_loop.get_result(coro, timeout)
  File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result
    return future.result(timeout)
  File "/lib64/python3.9/concurrent/futures/_base.py", line 444, in result
    raise CancelledError()
concurrent.futures._base.CancelledError

Fixes: https://tracker.ceph.com/issues/64473
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #54999 from Matan-B/wip-matanb-mon-osd-epochs

mon/OSDMonitor: fix get_min_last_epoch_clean()

Reviewed-by: Samuel Just <sjust@redhat.com>

test/rgw: increase timeouts in unittest_rgw_dmclock_scheduler

1ms sleeps are generally below the timer's resolution. increase run_for()
durations to 50ms to make the tests far less sensitive to timing. in
practice, none of the sleeps actually wait the full 50ms

Fixes: https://tracker.ceph.com/issues/64568
Signed-off-by: Casey Bodley <cbodley@redhat.com>

client: check mds down status bofore getting mds_gid_t from mdsmap

Get mds_gid_t from up of MDSMap, will cause to core dump if target
mds_rank_t does not exist up map:

from: src/mds/MDSMap.h
  const auto& get_info(mds_rank_t m) const {
    return mds_info.at(up.at(m));
  }

reproduct:
1. ceph fs fail <fs_name>
2. curl -X GET "example.com:8080/api/cephfs/1/clients" ...
up.at(m) will cause to core dump.

Fixes: https://tracker.ceph.com/issues/61844
Signed-off-by: Yite Gu <yitegu0@gmail.com>

Merge pull request #55683 from lxbsz/wip-62837-new

doc/man/8/mount.ceph.rst: add more mount options

Reviewed-by: Zac Dover <zac.dover@proton.me>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55705 from xxhdx1985126/wip-64535

crimson/os/seastore: fix wrong-positioned asserts

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/dashboard: fix volume creation with multiple hosts

Fixes: https://tracker.ceph.com/issues/64559
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

test/crimson/test_socket: complete write-side errors in test_unexpected_down()

Related: https://tracker.ceph.com/issues/64457
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #55739 from zdover23/wip-doc-2024-02-26-glossary-omap

doc/glossary: Add "OMAP" to glossary

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/man/8/mount.ceph.rst: add more mount options

Fixes: https://tracker.ceph.com/issues/62837
Signed-off-by: Xiubo Li <xiubli@redhat.com>

ceph-mixins: Add vars to support nvmeof alerts

Signed-off-by: Paul Cuzner <pcuzner@ibm.com>

doc/glossary: Add "OMAP" to glossary

Add an entry for "OMAP" to the glossary. This commit represents only the
first improvement to the documentation of OMAP in the Ceph docs suite.

Signed-off-by: Zac Dover <zac.dover@proton.me>

cephadm/nvmeof: scrape nvmeof prometheus endpoint

Fixes: https://tracker.ceph.com/issues/64536
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #54802 from ajarr/wip-61617

qa: Add tests to validate synced images on rbd-mirror

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55722 from cbodley/wip-64543

rgw/aio: avoid infinite recursion in aio_abstract()

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #55729 from zdover23/wip-doc-2024-02-24-rbd-nvmof-target-configure-list-repair

doc/rbd: repair ordered list

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55731 from zdover23/wip-doc-2024-02-24-dev-crimson

doc/dev: fix spelling in crimson.rst

doc/dev: fix spelling in crimson.rst

Repair a confusing sentence in doc/dev/crimson.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55699 from Svelar/seastore_doc

doc/crimson: cleanup duplicate seastore description

Reviewed-by: Zac Dover <zac.dover@proton.me>