cephadm: pin pyfakefs version for tox tests

5.4.0 seems to have caused a number of permission issues
in the test. In order to unblock make check, let's pin it
to 5.3.5 for now.

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #56530 from rhcs-dashboard/enable-grpc-copr

install-deps: enable copr ceph/grpc

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #54955 from leonid-s-usov/always_on_config

MgrMonitor: consider the build config when defining the always-on modules

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #54335 from batrick/i63432

qa: test test_kill_mdstable for all mount types

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #56627 from zhscn/wip-rewrite-snap-refcount

crimson/os/seastore: alloc mapping with refcount when rewriting logical extents

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #56738 from zdover23/wip-doc-2024-04-05-dev-peering-concepts-4-of-3

doc/dev: refine "Concepts" 4 of 3

Merge pull request #56703 from tchaikov/rgw-test_posix_bucket_cache-fixture

test/rgw/test_posix_bucket_cache: structure tests with fixture

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

doc/dev: refine "Concepts" 4 of 3

s/PG's PG logs/PG's logs/

re https://github.com/ceph/ceph/pull/56727#discussion_r1553541922

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55711 from rishabh-d-dave/fsshell-deprecation

cephfs-shell: prints warning, hangs and aborts when launched

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #56535 from linuxbox2/wip-lc-65188

rgwlc: check for no-bucket at bucket_lc_process() preamble

Reviewed-by: Lei Zhang <1091517373@qq.com>
Reviewed-by: Jiffin Tony Thottan <jthottan@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56657 from soumyakoduri/wip-skoduri-cloud-trans

rgw/cloud: Handle RGWRESTStreamS3PutObj initialization failures

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56727 from zdover23/wip-doc-2024-04-05-dev-peering-concepts-3-of-3

doc/dev: refine "Concepts" 3 of 3

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

test/rgw/test_posix_bucket_cache: structure tests with fixture

before this change, we use individual tests to setup/teardown
the fixtures. this renders it difficult to run a certain test.

in this change, we structure the tests with fixtures, so that
we can run any test without worrying the dependencies between
them.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

install-deps: enable copr ceph/grpc

In dashboard, to generate nvmeof apis in el8 this is needed so that it
can download the python3-grpcio packages.
https://copr.fedorainfracloud.org/coprs/ceph/grpc/

Fixes: https://tracker.ceph.com/issues/65184
Signed-off-by: Nizamudeen A <nia@redhat.com>

doc/dev: refine "Concepts" 3 of 3

Refine the third third of the entries in the section "Concepts" in
doc/dev/peering.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55725 from neesingh-rh/wip-63699

qa: fixing tests in test_cephfs_shell.TestShellOpts

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #56646 from zdover23/wip-doc-2024-04-02-dev-peering-concepts-2-of-3

doc/dev: refine "Concepts" 2 of 3

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #56645 from ivancich/wip-remove-example-ldap-domain

rgw: udpate options yaml file so LDAP uri isn't an invalid example

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #55203 from jzhu116-bloomberg/wip-63995

rgw/lc: delete expired delete-marker when processing Expiration lc action with Days specified

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #52444 from pritha-srivastava/wip-rgw-d4n

d4n topic branch

Reviewed-by: Ali Maredia <amaredia@redhat.com>

doc/dev: refine "Concepts" 2 of 3

Refine the second third of the entries in the section "Concepts" in
doc/dev/peering.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54778 from lxbsz/wip-63141

qa/cephfs: fix root_squash check failure bug

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: don't use pkg_resources since its deprecated

Currently, cephfs-shell prints warnings, hangs and aborts when launched.
This occurs because Python module "pkg_resources" has been deprecated.
We use that module only checking version of other Python modules used in
cephfs-shell. Use "Version" from "packaging.version" instead.

Fixes: https://tracker.ceph.com/issues/64538
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #55701 from joscollin/wip-B62265-use-monotonic-clocks

cephfs-mirror: use monotonic clocks

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52648 from lxbsz/wip-62036

mds: defer trim() until after the last cache_rejoin ack being received

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #51993 from dparmar18/wip-58072

mon/MDSMonitor: enable 'ceph fs new' use 'ceph fs set' options

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #55914 into main

* refs/pull/55914/head:
mds: do not use item item_caps in two different lists
mds: add new item item_to_flush

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #56153 from kotreshhr/qa-data-scan

qa: Test lost+found object removal without unmounting client

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #56514 from afreen23/wip-advanced-pools

mgr/dashboard:Use advanced fieldset for rbd image

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #56662 from zdover23/wip-doc-2024-04-03-zabbix-john-jasen-credit

doc/mgr: credit John Jasen for Zabbix 2

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55847 from athanatos/sjust/for-review/wip-crimson-clientio-coroutine

crimson: convert some of client_request to use coroutines

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54735 from cbodley/wip-corpus-18.2.0

dencoder: update ceph-object-corpus submodule for 18.2.0 coverage

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #56628 from gbregman/main

mgr/cephadm: add some NVMEof gateway fields to be configurable by cephadm

Reviewed-by: Adam King <adking@redhat.com>

rgw: udpate options yaml file so LDAP uri isn't an invalid example

LDAP tries to bind the URI configuration option when RGW starts. The
default value is an example used to show the form of the URI and is
not itself valid. The default value is used, unless overrideen, and
can cause delays in start-up in some situations. The example is now
provided in the description and the default is the empty string.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #55768 from batrick/i53192

mds: skip sr moves when target is an unlinked dir

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55214 from AliMasarweh/wip-alimasa-notif-data-path-v2

RGW: Migrate topics to data path v2

reviewed-by: kchheda3, cbodley, yuvalif

Merge pull request #56643 from ceph/wip-lusov-with-bluestore

test/objectstore: honor WITH_BLUESTORE config for dependent tests

Merge pull request #56562 from cbodley/wip-65212

rgw/pubsub: CreateTopic validates topic name

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge PR #56542 into main

* refs/pull/56542/head:
mds/quiesce: prevent an overflow of the wait duration

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/mgr: credit John Jasen for Zabbix 2

Credit John Jasen for the development of the procedures describing the
installation of Zabbix 2.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56635 from zdover23/wip-doc-2024-04-02-dev-peering-concepts-1-of-3

doc/dev: refine "Concepts"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw/cloud: Handle RGWRESTStreamS3PutObj initialization failures

With the recent code added to handle connection errors
(commit#e200499bb3c5703862b92a4d7fb534d98601f1bf), RGWRESTStreamS3PutObj
initialization could fail at times if there were any failed requests to the
cloud endpoint within CONN_STATUS_EXPIRE_SECS period.

This fix is to handle such errors and abort the transition/sync
requests which can be retried later by LC/Sync worker threads.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

crimson/os/seastore: alloc mapping with refcount when rewriting logical extents

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

crimson/os/seastore/btree_lba_manager: update_refcount returns the refcount of intermediate mapping

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

crimson/os/seastore/btree_lba_manager: cleanup methods that return std::pair

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

crimson/os/seastore: introduce extent_ref_count_t

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

crimson/os/seastore: remove unused return value of RecordScanner::scan_valid_records

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

crimson/common/interruptible_future: add discard_result

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Merge pull request #56509 from rhcs-dashboard/replace-table-panel-grafana

mgr/dashboard: replace deprecated table panel in grafana with a newer table panel

Reviewed-by: Nizamudeen A <nia@redhat.com>

python-common/ceph/deployment: add nvmeof configuration fields
Fixes https://tracker.ceph.com/issues/65259

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge PR #56228 into main

* refs/pull/56228/head:
doc: fixing doc/cephfs/fs-volumes

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

doc/dev: refine "Concepts"

Refine the text in the first third of the entries in the section
"Concepts" in doc/dev/peering.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56625 from tchaikov/wip-osd-hitset-free

test/osd/hitset: free allocated HitSet

Reviewed-by: Samuel Just <sjust@redhat.com>

test/objectstore: honor WITH_BLUESTORE config for dependent tests

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #56565 into main

* refs/pull/56565/head:
msg: trivial destructor should be default

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

mds/quiesce: prevent an overflow of the wait duration

QuiesceTimeInterval::max() may overflow inside of a call to
std::condition_variable::wait_for and result in a busy-loop,
making the call to timeout immediately

The solution is to cap the wait duration to a value which can
certainly fit in whichever clock std library is using internally.

Fixes: https://tracker.ceph.com/issues/65276
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #56566 into main

* refs/pull/56566/head:
qa: add missing pg_health fragment links in fs suite
qa: ignore PG health warnings in CephFS QA

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #56529 into main

* refs/pull/56529/head:
mds: remove unnecssary quiesce finisher variable
mds: attach quiesce_path mdr to finisher at creation not dispatch

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #55647 into main

* refs/pull/55647/head:
mds: use cephfs errno for fragment size check

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #56617 from adk3798/wip-revert-55346

Revert "mgr: use un-deprecated APIs to initialize Python interpretor"

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

test/rgw/noitifications: run basic/http/migration tests with 2 RGWs

this is useful to:
* cover the cls_lock code with persistent notifications
* cover v1 to v2 migration collisions between RGWs

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/notifications: run v2 migration tests in teuthology

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

RGW/Rados: Migrate topics to data path v2

also add migration tests

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>

vstart: Add `flushall` call if redis server found

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
Signed-off-by: Samarah <samarah.uriarte@ibm.com>

qa/d4n: Remove unused import

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/redis: Remove unused lambda variable

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

qa/d4n: Start redis up before rgw

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/redis: Implement RedisDriver::get_async and RedisDriver::put_async

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

qa/d4n: Add test for large object

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/policy: Add test for redis_sync method

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

qa/d4n: Update D4N workunit to test small object read workflow

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

d4n/directory: Fix oid for directory blocks

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/policy: Update local weight sum + age handling and d4n policy
test

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/d4n: removing lock from code path that writes to the cache
backend, and ensuring that write to the cache is atomic.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: implementation of `put` using yield context
as completion token and adding throttling to `put_async`
in the cache driver api. Also added a test case to the
ssd driver unit test for `put_async`.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: updating the boost_redis submodule version to point to
`develop` branch.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/d4n: fixing the 'get' call hanging issues and
Operation Canceled errors, by wrapping async_exec
call in boost::asio::dispatch() call.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/d4n: modifications to the filter driver:
1. replaced put_async() with put() in handle_data()
2. moved calls to update() from iterate() to flush()

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/d4n: Perform minor cleanup; add error checking for `localWeights` and fix order of entry deletion in policy

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n: Add a central Redis connection to the filter to be shared by the rest of the D4N classes and update D4N unit tests appropriately

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/policy: Change `size_t` argument type and add error checking to
`set_local_weight_sum` method

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

test/redis: Remove temporary boost.redis test program

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw: this commit squashes commits performing cleanup in various files
like d4n filter, policy, cache backend, aio and process_env.

d4n/filter: Remove unnecessary erasure of attrs in read op's `prepare`
method
d4n: Remove `dout` definitions from header files
rgw: Remove unnecessary includes in `rgw_process_env.h`
rgw: Remove `cache_read_op` from `rgw_aio.h`
rgw: Return POSIX error codes and perform minor cleanup for D4N and
Redis files

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

build: Remove build/boost_redis

Signed-off-by: Adam Emerson <aemerson@redhat.com>

d4n/filter: Correct `CacheObj` `hostsList` value so it's empty since the
object isn't being locally cached

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/filter: Update `creationTime` calculation

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/policy: Add additional lock guards for other `entries_heap` accesses

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/policy: Fix `find_victim` logic so victim is found in `eviction`
method

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/filter: Fix cache `exist_key` check to ensure the return value is positive

Signed-off-by: samarah <suriarte@redhat.com>

rgw/cache: Make `bl` and `attrs` params `const` for cache write ops

Signed-off-by: samarah <suriarte@redhat.com>

d4n/policy: Make `shutdown` method specific to LFUDA

Signed-off-by: samarah <suriarte@redhat.com>

rgw: Take in `io_context` by reference

Signed-off-by: samarah <suriarte@redhat.com>

rgw/cache: Make protected members private

Signed-off-by: samarah <suriarte@redhat.com>

d4n/policy: Initialize `age` to 1

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/policy: Fix minor logical errors

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/cache: this commit squashes commits for removing CephContext* from
ssd, redis driver and related test files.

rgw/cache: removing CephContext* from ssd driver and redis driver.
It can be accessed in every api using dpp->get_cct(), where dpp
is of type DoutPrefixProvider*.
rgw/test: Remove `cct` parameter from cache driver initialization call
redis driver test file.
rgw/qa: removing cct parameter from ssd driver test file.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/cache: changing `get_attr` api such that it returns
int signifying success or error, and `attr_val` is added
as an out param.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: removing `experimental` fallback from ssd backed cache driver.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: removing interface related to Partition.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/d4n: using 'id_tag' as 'version' of an object for non-versioned
objects. Using 'instance' of 'oid' as 'version' for
versioned objects.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>