qa/cephadm: update images for test_cephadm workunit

This patch doesn't actually fix any tests, but looking
at https://tracker.ceph.com/issues/64208 has made me
realize we haven't been updating the images on here
at all. This patch just removes the octopus and pacific
image variables and puts in ones for reef and squid. I
think this is a reasonable way to try and maintain these,
as we're just keeping around the image for the last two
releases, which matches with what versions could one day
upgrade to what main will become (the "T" release as
of writing this)

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #52444 from pritha-srivastava/wip-rgw-d4n

d4n topic branch

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #54778 from lxbsz/wip-63141

qa/cephfs: fix root_squash check failure bug

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #55701 from joscollin/wip-B62265-use-monotonic-clocks

cephfs-mirror: use monotonic clocks

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52648 from lxbsz/wip-62036

mds: defer trim() until after the last cache_rejoin ack being received

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #51993 from dparmar18/wip-58072

mon/MDSMonitor: enable 'ceph fs new' use 'ceph fs set' options

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #55914 into main

* refs/pull/55914/head:
mds: do not use item item_caps in two different lists
mds: add new item item_to_flush

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #56153 from kotreshhr/qa-data-scan

qa: Test lost+found object removal without unmounting client

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #56514 from afreen23/wip-advanced-pools

mgr/dashboard:Use advanced fieldset for rbd image

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #56662 from zdover23/wip-doc-2024-04-03-zabbix-john-jasen-credit

doc/mgr: credit John Jasen for Zabbix 2

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55847 from athanatos/sjust/for-review/wip-crimson-clientio-coroutine

crimson: convert some of client_request to use coroutines

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54735 from cbodley/wip-corpus-18.2.0

dencoder: update ceph-object-corpus submodule for 18.2.0 coverage

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #56628 from gbregman/main

mgr/cephadm: add some NVMEof gateway fields to be configurable by cephadm

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #55768 from batrick/i53192

mds: skip sr moves when target is an unlinked dir

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55214 from AliMasarweh/wip-alimasa-notif-data-path-v2

RGW: Migrate topics to data path v2

reviewed-by: kchheda3, cbodley, yuvalif

Merge pull request #56643 from ceph/wip-lusov-with-bluestore

test/objectstore: honor WITH_BLUESTORE config for dependent tests

Merge pull request #56562 from cbodley/wip-65212

rgw/pubsub: CreateTopic validates topic name

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge PR #56542 into main

* refs/pull/56542/head:
mds/quiesce: prevent an overflow of the wait duration

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/mgr: credit John Jasen for Zabbix 2

Credit John Jasen for the development of the procedures describing the
installation of Zabbix 2.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56635 from zdover23/wip-doc-2024-04-02-dev-peering-concepts-1-of-3

doc/dev: refine "Concepts"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #56509 from rhcs-dashboard/replace-table-panel-grafana

mgr/dashboard: replace deprecated table panel in grafana with a newer table panel

Reviewed-by: Nizamudeen A <nia@redhat.com>

python-common/ceph/deployment: add nvmeof configuration fields
Fixes https://tracker.ceph.com/issues/65259

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge PR #56228 into main

* refs/pull/56228/head:
doc: fixing doc/cephfs/fs-volumes

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

doc/dev: refine "Concepts"

Refine the text in the first third of the entries in the section
"Concepts" in doc/dev/peering.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56625 from tchaikov/wip-osd-hitset-free

test/osd/hitset: free allocated HitSet

Reviewed-by: Samuel Just <sjust@redhat.com>

test/objectstore: honor WITH_BLUESTORE config for dependent tests

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #56565 into main

* refs/pull/56565/head:
msg: trivial destructor should be default

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

mds/quiesce: prevent an overflow of the wait duration

QuiesceTimeInterval::max() may overflow inside of a call to
std::condition_variable::wait_for and result in a busy-loop,
making the call to timeout immediately

The solution is to cap the wait duration to a value which can
certainly fit in whichever clock std library is using internally.

Fixes: https://tracker.ceph.com/issues/65276
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #56566 into main

* refs/pull/56566/head:
qa: add missing pg_health fragment links in fs suite
qa: ignore PG health warnings in CephFS QA

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #56529 into main

* refs/pull/56529/head:
mds: remove unnecssary quiesce finisher variable
mds: attach quiesce_path mdr to finisher at creation not dispatch

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #55647 into main

* refs/pull/55647/head:
mds: use cephfs errno for fragment size check

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #56617 from adk3798/wip-revert-55346

Revert "mgr: use un-deprecated APIs to initialize Python interpretor"

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

test/rgw/noitifications: run basic/http/migration tests with 2 RGWs

this is useful to:
* cover the cls_lock code with persistent notifications
* cover v1 to v2 migration collisions between RGWs

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/notifications: run v2 migration tests in teuthology

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

RGW/Rados: Migrate topics to data path v2

also add migration tests

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>

vstart: Add `flushall` call if redis server found

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
Signed-off-by: Samarah <samarah.uriarte@ibm.com>

qa/d4n: Remove unused import

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/redis: Remove unused lambda variable

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

qa/d4n: Start redis up before rgw

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/redis: Implement RedisDriver::get_async and RedisDriver::put_async

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

qa/d4n: Add test for large object

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/policy: Add test for redis_sync method

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

qa/d4n: Update D4N workunit to test small object read workflow

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

d4n/directory: Fix oid for directory blocks

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/policy: Update local weight sum + age handling and d4n policy
test

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/d4n: removing lock from code path that writes to the cache
backend, and ensuring that write to the cache is atomic.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: implementation of `put` using yield context
as completion token and adding throttling to `put_async`
in the cache driver api. Also added a test case to the
ssd driver unit test for `put_async`.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: updating the boost_redis submodule version to point to
`develop` branch.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/d4n: fixing the 'get' call hanging issues and
Operation Canceled errors, by wrapping async_exec
call in boost::asio::dispatch() call.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/d4n: modifications to the filter driver:
1. replaced put_async() with put() in handle_data()
2. moved calls to update() from iterate() to flush()

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/d4n: Perform minor cleanup; add error checking for `localWeights` and fix order of entry deletion in policy

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n: Add a central Redis connection to the filter to be shared by the rest of the D4N classes and update D4N unit tests appropriately

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/policy: Change `size_t` argument type and add error checking to
`set_local_weight_sum` method

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

test/redis: Remove temporary boost.redis test program

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw: this commit squashes commits performing cleanup in various files
like d4n filter, policy, cache backend, aio and process_env.

d4n/filter: Remove unnecessary erasure of attrs in read op's `prepare`
method
d4n: Remove `dout` definitions from header files
rgw: Remove unnecessary includes in `rgw_process_env.h`
rgw: Remove `cache_read_op` from `rgw_aio.h`
rgw: Return POSIX error codes and perform minor cleanup for D4N and
Redis files

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

build: Remove build/boost_redis

Signed-off-by: Adam Emerson <aemerson@redhat.com>

d4n/filter: Correct `CacheObj` `hostsList` value so it's empty since the
object isn't being locally cached

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/filter: Update `creationTime` calculation

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/policy: Add additional lock guards for other `entries_heap` accesses

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/policy: Fix `find_victim` logic so victim is found in `eviction`
method

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/filter: Fix cache `exist_key` check to ensure the return value is positive

Signed-off-by: samarah <suriarte@redhat.com>

rgw/cache: Make `bl` and `attrs` params `const` for cache write ops

Signed-off-by: samarah <suriarte@redhat.com>

d4n/policy: Make `shutdown` method specific to LFUDA

Signed-off-by: samarah <suriarte@redhat.com>

rgw: Take in `io_context` by reference

Signed-off-by: samarah <suriarte@redhat.com>

rgw/cache: Make protected members private

Signed-off-by: samarah <suriarte@redhat.com>

d4n/policy: Initialize `age` to 1

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/policy: Fix minor logical errors

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/cache: this commit squashes commits for removing CephContext* from
ssd, redis driver and related test files.

rgw/cache: removing CephContext* from ssd driver and redis driver.
It can be accessed in every api using dpp->get_cct(), where dpp
is of type DoutPrefixProvider*.
rgw/test: Remove `cct` parameter from cache driver initialization call
redis driver test file.
rgw/qa: removing cct parameter from ssd driver test file.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/cache: changing `get_attr` api such that it returns
int signifying success or error, and `attr_val` is added
as an out param.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: removing `experimental` fallback from ssd backed cache driver.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: removing interface related to Partition.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/d4n: using 'id_tag' as 'version' of an object for non-versioned
objects. Using 'instance' of 'oid' as 'version' for
versioned objects.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/d4n: restructuring LRUPolicy::erase(), update() and eviction() method.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: adding unittests for ssd backed cache backend, and
fixing bugs that the tests revealed.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: fixing the free space available in ssd backed cache
backend by taking into account the size of xattrs in addition to
the file size.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

d4n: Update block `mtime` in filter and change `creationTime` to string
in directory due to seg faults

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/directory: Add `ObjectDirectory::update_field` method

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/d4n: this commit squashes the following commits related to LFUDA Policy.

rgw/d4n: Make minor updates, improve `LFUDAPolicy::erase` method, and
fix `localWeight` policy sets
rgw/d4n: Update `LFUDA::find_victim` method, add locks in policy code,
and remove setting of attrs in filter's `put_async` calls
d4n/policy: Update logic for minimum average weight calculation
d4n/policy: Remove iteration in `eviction`

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/filter: Change from copying block to object in `copy_obj`

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/d4n: this commit squashes the following two commits.

rgw/d4n: Remove `get_block` from policy and add logic to D4N filter
d4n/policy: Create a min heap of the local weights

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n: Store `optional_yield` in client cb and update index used in
D4N/cache classes

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/policy: Update policy to use block name

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n: Add connection client names for policy and directory

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw: Add `rgw_local_cache_address` yaml config

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/d4n: Remove `cacheBlock` and commented out read workflow from D4N filter

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/directory: Update directory `exist_key` method to take in
block/object; fix endpoint generation; add `remove_host` method

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

d4n/policy: this commit squashes the following changes to policy and filter driver files.

d4n/policy: Update policy with correct directory usage; create LFUDAEntry struct + entries map; add `optional_yield` to `insert` method
d4n/policy: Add `version` to Entry struct/policy methods and calls in
the D4N filter; add error checking and fix logic in LFUDA code

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/cache: this commit squashes following commits related to changes in
redis driver and ssd driver.

rgw/redis: Add `del`, `put_async`, and `calculate_free_space` methods to RedisDriver
rgw/redis: Work on RedisDriver `put_async` and `get_async` methods
rgw/cache: Add error-checking to `SSDDriver::set_attr`
rgw/redis: Remove `calculate_free_space` method from RedisDriver, fix
`free_space` calculations, clean up async code

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

test/d4n: this commit squashes the following commits related to tests for d4n.

test/redis: Add temporary `boost::redis` unit test program for experimentation
test/rgw: Add/update unit test programs for D4N classes + RedisDriver; switch to `boost::redis` usage
test/d4n: Begin `cpp_redis` removal from D4N filter test program; needs
to be fixed
test/d4n: Temporarily remove D4N filter test program until D4N workflows
are fully implemented

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/d4n: this commit squashes the following commits related to
clean up and removal of cpp_redis.

d4n/filter: Add `optional_yield` to CacheDriver calls in D4N Filter and
make minor updates to several filter methods
rgw/d4n: fix compilation issue.
rgw/cache: Add `del` method to CacheDriver and SSDDriver
cmake/d4n: Remove unnecessary D4N lines
rgw: Add `io_context` to D4N Filter and RedisDriver, remove `cpp_redis`
library from RedisDriver, and perform minor cleanup
d4n: Remove `cpp_redis` library from D4N directory and policy; update calls in filter; move Entry struct to base CachePolicy class
build/cpp_redis: Remove `cpp_redis` library
rgw/d4n: including <boost/asio/detached.hpp> wheerever needed.
rgw/d4n : fixes to d4n filter, policy, directory and redis driver files
for compilation errors.

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Co-authored-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: Change D4N and RedisDriver `ifndef` directives to `#pragma once`; make integer errors negative; fix filter includes

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw/cache: this commit adds entries to redis driver and squashes the
following two commits.

rgw: Update D4N policy and RedisDriver with entries
rgw/d4n: fixing compilation error in d4n policy code.

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
Co-authored-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: fixing update_attrs and get_attrs in ssd backed cache backend.

update_attrs did not check whether an xattr already existed before
modifying it - added check for that. corrected converting bufferlist
to string.

get_attrs did not check whether attribute size is zero or negative -
added check for that.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/d4n: copying over read op params of the filter driver to
the read op params of 'next' in prepare() method, to be used
while storing objects to the backend store.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: adding d4n config options for ssd backed
cache driver.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: saving dpp in set_client_cb() instead
of flush_write_part() so that the correct value
is available everywhere in handle_data().

Saving dpp value in flush_last_part() caused a crash in
rgw in handle_data().

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: This commit fixes two things in ssd backed cache backend:
1. fixing free space calcuation when data is deleted.

2. implementation of append_data().

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: moving LRU related data structures and interfaces to the Policy Manager.
Replacing the related calls in d4n filter driver with that of the cachepolicy in the
Policy Manager and adding cache block to the block directory
once the block is written to the cache backend and the
policy manager data structure.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: this commit adds `optional_yield` to ssd and redis driver
and squashes the following commits.

RGW: Add `optional_yield` parameter to Cache Driver methods
RGW: Add `optional_yield` parameter to SSD Driver methods
rgw/cache: fix compilation errors related to optional_yield.

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
Co-authored-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/cache: this commit removes indexing from ssd and redis
driver and squashes the following commits.

rgw/cache: modifications to the ssd cache driver after removing
indexing from the cache backend.
rgw/cache: modifications to redis cache driver, after
removing indexing from cache backend.
rgw/cache: modifications to policy driver code, commenting out
some pieces which refer to cache apis. Those apis have been
removed as part of modification to remove indexing from the
cache backend. These need to appropriately re-instated, once
we finalize which module will contain the data structures
related to cache policy.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>