Merge pull request #55663 from yuvalif/wip-yuval-63909

test/rgw/notifications: do not delay http responses to notifications

Reviewed-by: Casey Bodley <cbodley@redhat.com>, Ali Marsawa <ali.masarwa@ibm.com>, Krunal Chheda <kchheda3@bloomberg.net>

Merge pull request #56774 from idryomov/wip-doc-releases-reef-latest

doc: reorder "releases" entries for reef to fix the diagram

Merge pull request #56796 from zdover23/wip-doc-2024-04-10-mgr-zabbix

doc/mgr: remove Zabbix 1 information

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/mgr: remove Zabbix 1 information

Remove information about the installation of the Zabbix module and link
to a discussion of the reasoning behind Ceph's refusal to support
Zabbix.

John Jasen developed a procedure explaining how to install "Zabbix 2".
This commit removes outdated procedures and explains why those
procedures were removed. Immediately following this explanation, the
text includes an explanation of how to install "Zabbix 2".

Signed-off-by: Zac Dover <zac.dover@proton.me>

test/rgw/notification: fix threading issue in test

test_ps_s3_persistent_topic_stats may delete boto client threads
before they are done.
also, no need to run the http server twice

Fixes: https://tracker.ceph.com/issues/63909
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #56577 from nbalacha/wip-nbalacha-code-cleanup

rbd-mirror: rename on_stop_journal_replay() and some operator<< arguments

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #56754 from zdover23/wip-doc-2024-04-08-cephfs-client-auth-1-of-3

doc/cephfs: refine client-auth (1 of 3)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #56459 from ronen-fr/wip-rf-nohist3

osd/scrub: implement reservation state as a ReplicaActive flag

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #56641 from rhcs-dashboard/wip-65268-main

mgr/dashboard: update NVMeoF gRPC: "listener add" changes

Reviewed-by: gbregman <NOT@FOUND>

test/rgw/notification: use polling instead of sleep

when checking persistent stats queue length

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: cleanup leftover topics

this may happen when a test fails, and does not cleanup topics
it created. other tests that verify the number of topics may fail
because of that.
all tests that verify number of topics, should delete all topics at the
start of the test.

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: use simpler multithreaded http server

Fixes: https://tracker.ceph.com/issues/63909
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #56383 from xxhdx1985126/wip-seastore-btree-cleanup

crimson/os/seastore/btree: clean up `FixedKVLeafNode::get_logical_child()`

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

doc: reorder "releases" entries for reef to fix the diagram

The Gantt diagram currently shows "reef (latest 18.2.0)" instead of
18.2.2.  This is because ReleasesGantt expects releases array to be
sorted in reverse order:

    for code_name, info in releases.items():
        last_release = info['releases'][0]
        first_release = info['releases'][-1]

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd_mirror: rename method

Rename on_stop_journal_replay to on_stop_replay as the
function doesn't do anything journal specific.

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge pull request #56764 from ceph/wip-yuriw-archive-pacific-main

doc: moved pacific release to Archived Releases

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Josh Durgin <jdurgin@ibm.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56469 from asm0deuz/bz2270211_ganesha_nlm

mgr/cephadm: Allows enabling NFS Ganesha NLM

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #56637 from adk3798/node-proxy-service-expected-daemon-count

mgr/cephadm: fix node-proxy service size

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #56674 from adk3798/wrong-grafana-uid

mgr/cephadm: pass daemon's current image when reconfiguring

Reviewed-by: Laura Flores <lflores@ibm.com>

mgr/dashboard: update NVMeoF gRPC

To include latest changes to the listener add call.

Fixes: https://tracker.ceph.com/issues/65268
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #56705 from adk3798/no-osd-require-osd-release

mgr/cephadm: handle setting required osd release with no OSDs during upgrade

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #56713 from adk3798/client-keyring-ceph-conf

mgr/cephadm: make client-keyring deploying ceph.conf optional

Reviewed-by: John Mulligan <jmulligan@redhat.com>

doc: moved pacific release to Archived Releases

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

test/rgw/notifications: do not delay http responses to notifications

we also increase the wating time after rerunning the http server
to allow fo the persistent notifications retries to finish

Fixes: https://tracker.ceph.com/issues/63909
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #56760 from adk3798/cephadm-pin-pyfakefs-version

cephadm: pin pyfakefs version for tox tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56712 from kchheda3/wip-notif-reload-bkt

rgw/notify: fix crashes in lc due to reload of bucket.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56619 from ljflores/wip-tracker-64864

qa/suites: add "mon down" log variations to ignorelist

cephadm: pin pyfakefs version for tox tests

5.4.0 seems to have caused a number of permission issues
in the test. In order to unblock make check, let's pin it
to 5.3.5 for now.

Signed-off-by: Adam King <adking@redhat.com>

doc/cephfs: refine client-auth (1 of 3)

Refine the first third of doc/cephfs/client-auth.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56530 from rhcs-dashboard/enable-grpc-copr

install-deps: enable copr ceph/grpc

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #54955 from leonid-s-usov/always_on_config

MgrMonitor: consider the build config when defining the always-on modules

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #54335 from batrick/i63432

qa: test test_kill_mdstable for all mount types

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/btree: clean up `FixedKVLeafNode::get_logical_child()`

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #56627 from zhscn/wip-rewrite-snap-refcount

crimson/os/seastore: alloc mapping with refcount when rewriting logical extents

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #56738 from zdover23/wip-doc-2024-04-05-dev-peering-concepts-4-of-3

doc/dev: refine "Concepts" 4 of 3

Merge pull request #56703 from tchaikov/rgw-test_posix_bucket_cache-fixture

test/rgw/test_posix_bucket_cache: structure tests with fixture

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

doc/dev: refine "Concepts" 4 of 3

s/PG's PG logs/PG's logs/

re https://github.com/ceph/ceph/pull/56727#discussion_r1553541922

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55711 from rishabh-d-dave/fsshell-deprecation

cephfs-shell: prints warning, hangs and aborts when launched

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #56535 from linuxbox2/wip-lc-65188

rgwlc: check for no-bucket at bucket_lc_process() preamble

Reviewed-by: Lei Zhang <1091517373@qq.com>
Reviewed-by: Jiffin Tony Thottan <jthottan@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56657 from soumyakoduri/wip-skoduri-cloud-trans

rgw/cloud: Handle RGWRESTStreamS3PutObj initialization failures

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56727 from zdover23/wip-doc-2024-04-05-dev-peering-concepts-3-of-3

doc/dev: refine "Concepts" 3 of 3

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

test/rgw/test_posix_bucket_cache: structure tests with fixture

before this change, we use individual tests to setup/teardown
the fixtures. this renders it difficult to run a certain test.

in this change, we structure the tests with fixtures, so that
we can run any test without worrying the dependencies between
them.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

install-deps: enable copr ceph/grpc

In dashboard, to generate nvmeof apis in el8 this is needed so that it
can download the python3-grpcio packages.
https://copr.fedorainfracloud.org/coprs/ceph/grpc/

Fixes: https://tracker.ceph.com/issues/65184
Signed-off-by: Nizamudeen A <nia@redhat.com>

doc/dev: refine "Concepts" 3 of 3

Refine the third third of the entries in the section "Concepts" in
doc/dev/peering.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55725 from neesingh-rh/wip-63699

qa: fixing tests in test_cephfs_shell.TestShellOpts

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #56646 from zdover23/wip-doc-2024-04-02-dev-peering-concepts-2-of-3

doc/dev: refine "Concepts" 2 of 3

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #56645 from ivancich/wip-remove-example-ldap-domain

rgw: udpate options yaml file so LDAP uri isn't an invalid example

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #55203 from jzhu116-bloomberg/wip-63995

rgw/lc: delete expired delete-marker when processing Expiration lc action with Days specified

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mgr/cephadm: pass daemon's current image when reconfiguring

Important to note here is that a reconfig will rewrite
the config files of the daemon and restart it, but not
rewrite the unit files. This lead to a bug where the
grafana image we used between the quincy and squid release
used different UIDs internally, which caused us to rewrite
the config files as owned by a UID that worked with the
new image but did not work with the image still specified
in the unit.run file. This meant the grafana daemon was down
from a bit after the mgr upgrade until the end
of the upgrade when we redeploy all the monitoring images.

Fixes: https://tracker.ceph.com/issues/65234
Signed-off-by: Adam King <adking@redhat.com>

doc/cephadm: document new --no-ceph-conf flag for `orch client-keyring set`

Signed-off-by: Adam King <adking@redhat.com>

rgw/notification: Load bucket attrs before calling publish_reserve.

As part of PR# 55657, publish_reserve would reload bucket to ensure bucket_attrs are loaded. However for lc events, where the bucket attrs were already loaded, the reloading was causing crash but there was no obvious root cause, so to avoid the crashes, remove reloading of bucket in publish_reserve and put the onus on callers to load the bucket before calling publish_reserve.

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>

mgr/cephadm: make client-keyring deploying ceph.conf optional

There are cases where users would like to manage their own
ceph.conf but still have cephadm deploy the client keyrings,
so this is being added to facilitate that.

Fixes: https://tracker.ceph.com/issues/65335
Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: rename ceph_admin_key -> client_key in _calc_client_files

I think this variable name is misleading, as we use this same
logic for all client-keyrings that can be setup, not just
the client.admin keyring

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #52444 from pritha-srivastava/wip-rgw-d4n

d4n topic branch

Reviewed-by: Ali Maredia <amaredia@redhat.com>

mgr/cephadm: handle setting required osd release with no OSDs during upgrade

A change to the `ceph osd require-osd-release` command made it so
it fails if no OSDs are up unless --yes-i-really-mean-it is passed.
For real clusters this is likely not an issue, but it can be an
annoyance for trying upgrades on test clusters that may not have
OSDs deployed. This patch is to try and just pass the flag in cases
where we have no OSDs rather than failing the upgrade

Signed-off-by: Adam King <adking@redhat.com>

doc/dev: refine "Concepts" 2 of 3

Refine the second third of the entries in the section "Concepts" in
doc/dev/peering.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54778 from lxbsz/wip-63141

qa/cephfs: fix root_squash check failure bug

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: don't use pkg_resources since its deprecated

Currently, cephfs-shell prints warnings, hangs and aborts when launched.
This occurs because Python module "pkg_resources" has been deprecated.
We use that module only checking version of other Python modules used in
cephfs-shell. Use "Version" from "packaging.version" instead.

Fixes: https://tracker.ceph.com/issues/64538
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #55701 from joscollin/wip-B62265-use-monotonic-clocks

cephfs-mirror: use monotonic clocks

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52648 from lxbsz/wip-62036

mds: defer trim() until after the last cache_rejoin ack being received

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #51993 from dparmar18/wip-58072

mon/MDSMonitor: enable 'ceph fs new' use 'ceph fs set' options

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #55914 into main

* refs/pull/55914/head:
mds: do not use item item_caps in two different lists
mds: add new item item_to_flush

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #56153 from kotreshhr/qa-data-scan

qa: Test lost+found object removal without unmounting client

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #56514 from afreen23/wip-advanced-pools

mgr/dashboard:Use advanced fieldset for rbd image

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #56662 from zdover23/wip-doc-2024-04-03-zabbix-john-jasen-credit

doc/mgr: credit John Jasen for Zabbix 2

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55847 from athanatos/sjust/for-review/wip-crimson-clientio-coroutine

crimson: convert some of client_request to use coroutines

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54735 from cbodley/wip-corpus-18.2.0

dencoder: update ceph-object-corpus submodule for 18.2.0 coverage

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #56628 from gbregman/main

mgr/cephadm: add some NVMEof gateway fields to be configurable by cephadm

Reviewed-by: Adam King <adking@redhat.com>

rgw: udpate options yaml file so LDAP uri isn't an invalid example

LDAP tries to bind the URI configuration option when RGW starts. The
default value is an example used to show the form of the URI and is
not itself valid. The default value is used, unless overrideen, and
can cause delays in start-up in some situations. The example is now
provided in the description and the default is the empty string.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #55768 from batrick/i53192

mds: skip sr moves when target is an unlinked dir

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55214 from AliMasarweh/wip-alimasa-notif-data-path-v2

RGW: Migrate topics to data path v2

reviewed-by: kchheda3, cbodley, yuvalif

Merge pull request #56643 from ceph/wip-lusov-with-bluestore

test/objectstore: honor WITH_BLUESTORE config for dependent tests

rbd-mirror: code cleanup

Renamed ostream argument in overloaded operator implementations.

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge pull request #56562 from cbodley/wip-65212

rgw/pubsub: CreateTopic validates topic name

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge PR #56542 into main

* refs/pull/56542/head:
mds/quiesce: prevent an overflow of the wait duration

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/mgr: credit John Jasen for Zabbix 2

Credit John Jasen for the development of the procedures describing the
installation of Zabbix 2.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56635 from zdover23/wip-doc-2024-04-02-dev-peering-concepts-1-of-3

doc/dev: refine "Concepts"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw/cloud: Handle RGWRESTStreamS3PutObj initialization failures

With the recent code added to handle connection errors
(commit#e200499bb3c5703862b92a4d7fb534d98601f1bf), RGWRESTStreamS3PutObj
initialization could fail at times if there were any failed requests to the
cloud endpoint within CONN_STATUS_EXPIRE_SECS period.

This fix is to handle such errors and abort the transition/sync
requests which can be retried later by LC/Sync worker threads.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

crimson/os/seastore: alloc mapping with refcount when rewriting logical extents

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

crimson/os/seastore/btree_lba_manager: update_refcount returns the refcount of intermediate mapping

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

crimson/os/seastore/btree_lba_manager: cleanup methods that return std::pair

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

crimson/os/seastore: introduce extent_ref_count_t

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

crimson/os/seastore: remove unused return value of RecordScanner::scan_valid_records

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

osd/scrub: uniform handling of reservation requests

we now allow - on the replica side - reservation requests
regardless of the ReplicaActive sub-state. I.e. - we will
honor such requests even when handling a chunk request
(in ReplicaActiveOp).

Note that the current Primary code would never send such
a request. But a future primary code might.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson/common/interruptible_future: add discard_result

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Merge pull request #56509 from rhcs-dashboard/replace-table-panel-grafana

mgr/dashboard: replace deprecated table panel in grafana with a newer table panel

Reviewed-by: Nizamudeen A <nia@redhat.com>

python-common/ceph/deployment: add nvmeof configuration fields
Fixes https://tracker.ceph.com/issues/65259

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge PR #56228 into main

* refs/pull/56228/head:
doc: fixing doc/cephfs/fs-volumes

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

doc/dev: refine "Concepts"

Refine the text in the first third of the entries in the section
"Concepts" in doc/dev/peering.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56625 from tchaikov/wip-osd-hitset-free

test/osd/hitset: free allocated HitSet

Reviewed-by: Samuel Just <sjust@redhat.com>

test/objectstore: honor WITH_BLUESTORE config for dependent tests

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #56565 into main

* refs/pull/56565/head:
msg: trivial destructor should be default

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

mds/quiesce: prevent an overflow of the wait duration

QuiesceTimeInterval::max() may overflow inside of a call to
std::condition_variable::wait_for and result in a busy-loop,
making the call to timeout immediately

The solution is to cap the wait duration to a value which can
certainly fit in whichever clock std library is using internally.

Fixes: https://tracker.ceph.com/issues/65276
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #56566 into main

* refs/pull/56566/head:
qa: add missing pg_health fragment links in fs suite
qa: ignore PG health warnings in CephFS QA

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #56529 into main

* refs/pull/56529/head:
mds: remove unnecssary quiesce finisher variable
mds: attach quiesce_path mdr to finisher at creation not dispatch

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #55647 into main

* refs/pull/55647/head:
mds: use cephfs errno for fragment size check

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #56617 from adk3798/wip-revert-55346

Revert "mgr: use un-deprecated APIs to initialize Python interpretor"

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

test/rgw/noitifications: run basic/http/migration tests with 2 RGWs

this is useful to:
* cover the cls_lock code with persistent notifications
* cover v1 to v2 migration collisions between RGWs

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/notifications: run v2 migration tests in teuthology

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

RGW/Rados: Migrate topics to data path v2

also add migration tests

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>