qa/workunits/rbd: use jammy version of qemu-iotests for centos 9

It's the one we are using for all recent distros.

While at it, get rid of custom bin directory -- it appears that both
v2.3.0 and v2.11.0 tests are happy with just symlinks in the current
directory.

Fixes: https://tracker.ceph.com/issues/61565
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #52523 into main

* refs/pull/52523/head:
mds: remove dead method

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52429 from guits/fix-raw-list

ceph-volume: fix raw list for lvm devices

Merge PR #52597 into main

* refs/pull/52597/head:
qa: use centos 8.stream for fs:upgrade tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52599 from Matan-B/wip-matanb-crimson-cbt-fix

qa: Fix cbt usage with Crimson

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52598 from Matan-B/wip-matanb-crimson-centos-8

qa/suites/crimson-rados: Use centos8 for testing

Reviewed-by: Samuel Just <sjust@redhat.com>

qa: use centos 8.stream for fs:upgrade tests

fs:upgrade tests use n/o/p release packages which aren't
available for centos 9.stream - switch using 8.stream instead.

Fixes: http://tracker.ceph.com/issues/62146
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge PR #48038 into main

* refs/pull/48038/head:
client test: Add fsync to ll_preadv_pwritev test
libcephfs: Option to write + fsync via ceph_ll_nonblocking_readv_writev
Client: Hook nonblocking fsync into the write path of ll_preadv_pwritev
Client: Add non-blocking fsync
Client/Inode: wait_for_caps fixups
Client: change several waitfor_* to use Context list
test: Add nonblocking I/O client test
libcephfs: Add nonblocking readv/writev I/O interface
Client: Add ll_preadv_pwritev to expose non-blocking I/O to libcephfs
Client: Add non-blocking helper classes
Client: Break some code into new methods in prep for non-blocking I/O
Buffers: Add function to buffer.h to copy bufferlist to an iovec
ObjectCacher: Prepare file_write path for non-blocking I/O

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #52607 from zap51/patch-1

doc/rbd/rbd-cloudstack.rst: include support for multiple monitors

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49889 from bluikko/bluikko-doc-srv-lookup-link

doc/rados/configuration: add links to MON DNS

doc/rbd/rbd-cloudstack.rst: include support for multiple monitors

Signed-off-by: Jayanth Reddy <45934027+zap51@users.noreply.github.com>

client test: Add fsync to ll_preadv_pwritev test

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

libcephfs: Option to write + fsync via ceph_ll_nonblocking_readv_writev

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Hook nonblocking fsync into the write path of ll_preadv_pwritev

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Add non-blocking fsync

We will need the ability to do an non-blocking write that finishes with
fsync so we need non-blocking fsync.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client/Inode: wait_for_caps fixups

The non-blocking flush requires us to be able to re-add to
wait_for_caps but if we simply add to the list, we get stuck in an
infinite loop. Add a wait_for_caps_pending list to add to, and then
when done signalling, we move the wait_for_caps_pending items onto the
wait_for_caps list.

Also in handle_cap_flush_ack(), we need to complete the caps flushing
before signalling since with non-blocking flush, we will be actually
examining the caps from the completion rather than signalling a
condition variable in the completion.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: change several waitfor_* to use Context list

Change waitfor_caps, waitfor_safe and waitfor_commit to Context list.

To make a non-blocking version of fsync (to be used for non-blocking write
and commit), we need to be able to signal an arbitrary Context on completion
of either of these lists.

add_nonblocking_onfinish_to_context_list Adds such a Context to the list.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

test: Add nonblocking I/O client test

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

libcephfs: Add nonblocking readv/writev I/O interface

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Add ll_preadv_pwritev to expose non-blocking I/O to libcephfs

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Add non-blocking helper classes

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Break some code into new methods in prep for non-blocking I/O

These bits of code need to be invoked from a separate spot when we
introduce non-blocking I/O, so break them out now.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Buffers: Add function to buffer.h to copy bufferlist to an iovec

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

ObjectCacher: Prepare file_write path for non-blocking I/O

For non-blocking I/O, we will want to be able to override
block_writes_upfront so rename the member cfg_block_writes_upfront and add
an option to pass block_writes_upfront as a parameter along with a member
access method so caller can pass cfg_block_writes_upfront.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Merge pull request #52521 from cbodley/wip-62059

valgrind: update suppression for SyscallParam under call_init

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #52601 from idryomov/wip-doc-redundant-tiering-note

doc/rados: remove redundant cache tiering deprecation note

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #51934 from kotreshhr/fix-assert-rename

mds: Fix the linkmerge assert check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52473 from vedanshbhartia/null_deref

rgw: add coverity annotations and asserts for null dereferences

Merge pull request #52437 from vedanshbhartia/coverity_missing_break

rgw: Add coverity annotation for missing break statement in switch

Merge pull request #52436 from vedanshbhartia/coverity_logical_dead

rgw: Remove logically dead code

Merge pull request #52434 from vedanshbhartia/coverity_1511179

rgw: Fix potential null pointer dereference in rgw_user.cc

Merge pull request #52433 from vedanshbhartia/coverity_1510266

rgw: When finding bucket owner info, handle account name being empty

Merge pull request #52432 from vedanshbhartia/coverity_1511177

rgw: Fix potential null dereference error in bucket name parsing

Merge pull request #52253 from vedanshbhartia/coverity_1515338

rgw: Init res id with cls_2pc_reservation::NO_ID

Merge pull request #51648 from vedanshbhartia/coverity_1510684

rgw: refactor D3nDataCache::d3n_io_write to avoid resource leaks

Merge pull request #51772 from caisan/multisite-hang-period-update

rgw:multisite hang when update and commit period

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #52056 from cbodley/wip-56993

rgw: s3 object lock avoids overflow in retention date

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #52036 from adamemerson/wip-rgw-mdlog-yield

rgw: Thread optional_yield through RGWMetadataLog operations

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

valgrind: update suppression for SyscallParam under call_init

centos9 has a slightly different call stack here

Fixes: https://tracker.ceph.com/issues/62059
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52529 from xxhdx1985126/wip-61943

crimson/osd/osd_operations: add pipeline to LogMissingRequest to sync it

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

doc/rados: remove redundant cache tiering deprecation note

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/crimson-rados/perf: add monitoring_profiles for cbt

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/tasks/cbt.py: Fix perf package naming

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados: Use centos8 for testing

centos_latest was moved to centos9, we only schedule centos8
Crimson builds.
Use centos8 until Crimson will support 9.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #52439 from AliMasarweh/wip-alimasa-persistant-q-observability

RGW: fix issue in observability over the persistent topics queue

crimson/osd/osd_operations: add pipeline to LogMissingRequest to sync it
with ReplicateRequest

This commit is a follow-up for e14142894922b403344d7f08315b67a291b9a27a
and f4d336a2dbf864fe798ca7f500d44576d3f626d5

Fixes: https://tracker.ceph.com/issues/61943
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

doc/rados/configuration: add links to MON DNS

Add two links to MON DNS lookup and an anchor in the destination document.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #52221 from xxhdx1985126/wip-collection-bug-fix

crimson/os/seastore/collection_manager: fill CollectionNode::decoded on clean reads

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #52446 from xxhdx1985126/wip-seastore-lazy-read-patch

crimson/os/seastore/transaction_manager: data loss issues

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xinyu Huang <xinyu.huang@intel.com>

Merge pull request #52445 from soumyakoduri/wip-skoduri-syncpolicy-fixes

rgw/sync-policy: Correct "sync status" & "sync group" commands

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: Read `sync status` from only the zones allowed to sync from

Using multisite sync policy, zones within a zonegroup can be configured
not to sync from certain zones. Filter such zones while reading sync
status.

Fixes: https://tracker.ceph.com/issues/62014
Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/sync-policy: Do not allow empty group names

Fixes: https://tracker.ceph.com/issues/62015
Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #52587 from zdover23/wip-doc-2023-07-22-rados-configuration-ceph-conf-4-of-x

doc/rados: edit ceph-conf.rst (4 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit ceph-conf.rst (4 of x)

Edit doc/rados/configuration/ceph-conf.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51820 from ifed01/wip-ifed-fragmentation-info

os/bluestore: introduce allocator state histogram

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #52542 from zdover23/wip-doc-2023-07-19-rados-configuration-ceph-conf-3-of-x

doc/rados: edit ceph-conf.rst (3 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit ceph-conf.rst (3 of x)

Edit doc/rados/configuration/ceph-conf.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52475 from yuvalif/wip-yuval-multisite-vstart-params

rgw/test: allow passing vstart parameters to multisite tests script

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52057 from nbalacha/tracker-61672

rbd-mirror: fix race preventing local image deletion

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #52025 from rzarzynski/wip-ec-drop-unused-requires_overwrite

osd: drop the unused ECTransaction::requires_overwrite()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #52491 from cbodley/wip-61566

valgrind: replace suppression for EVP_DecryptFinal_ex

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52427 from petrutlucian94/fix_win32_boost

win32_deps_build: skip patching removed boost files

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #52143 from cbodley/wip-61567

test/pybind: replace nose with pytest

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52563 from zdover23/wip-doc-2023-07-21-governance-orman-email-update

doc: governance.rst - update D Orman

doc: governance.rst - update D Orman

Update David Orman's email address in doc/governance.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52544 from rhcs-dashboard/debug-make-check

install-deps: remove the legacy resolver flags

valgrind: remove stale openssl suppressions

years ago in commit fe97a000990632dbe9734e0d3a96df627f104f7a, the
signature of authenticated_decrypt_update_final() changed and this
suppression no longer matches its mangled form:

-  fun:_ZN4ceph6crypto6onwire25AES128GCM_OnWireRxHandler34authenticated_decrypt_update_finalEONS_6buffer7v*4listEj
+  fun:_ZN4ceph6crypto6onwire25AES128GCM_OnWireRxHandler34authenticated_decrypt_update_finalERNS_6buffer7v15_2_04listE

the fact that we didn't notice must mean that we don't need the
suppressions anymore

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52374 from rosinL/wip-cleanup

crush/CrushLocation: remove useless flag

Reviewed-by: Samuel Just <sjust@redhat.com>

valgrind: suppress uninitialized value in OpenSSL

Fixes: https://tracker.ceph.com/issues/61566
Signed-off-by: Casey Bodley <cbodley@redhat.com>

install-deps: remove the legacy resolver flags

This was a workaround that was introduced long time ago. This will be
something that could be deprectaed at some point [1]. And its preventing some of the dependencies to be
downloaded or stored into the wheelhouse. Deps like jsonschema, parse,
mypy, cryptography etc.

[1] https://pip.pypa.io/en/latest/user_guide/#deprecation-timeline

Fixes: https://tracker.ceph.com/issues/62082
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #49332 from shreyanshjain7174/stuck_peering

mon: stuck peering since warning is misleading

Reviewed-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

Merge pull request #51438 from NitzanMordhai/wip-nitzan-cbt-perf-ci

cbt perf ci

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50301 from liuqinfei/support_compile_on_openeuler_main

ceph.spec.in: add support for openEuler OS

Reviewed-by: Rongqi Sun <sunrongqi@huawei.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #49480 from flyingcircusio/fix-mon-delete-iterator

 mon: fix iterator mishandling in PGMap::apply_incremental

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52021 from xxhdx1985126/wip-61650

osd/PeeringState: recheck pg readability when renewing leases for no-replica pools

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50441 from cbodley/wip-qa-distros-centos9

qa/distros: add centos stream 9 as supported distro

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #52124 from idryomov/wip-61733

mon/MonClient: resurrect original client_mount_timeout handling

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-volume: fix raw list for lvm devices

raw list shouldn't list lvm OSDs.
This confuses the generic activate process given that it relies
on the raw list to detect which kind of OSD is being activated.

Fixes: https://tracker.ceph.com/issues/62002
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: fix broken workaround for atari partitions

broken by bea9f4b643ce32268ad79c0fc257b25ff2f8333c
This commits fixes that regression.

Fixes: https://tracker.ceph.com/issues/62001
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #52266 from amathuria/wip-crimson-amat-fix-config-show

crimson/mgr: Fix config show command

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

mds: remove dead method

Fixes: cf5fdcd6e2d22f35b670d3c13ae8b7c9b9d2616d
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #51539 into main

* refs/pull/51539/head:
doc: users now need to provide scrub_mdsdir and recursive flags
qa: add recursive flag to test_flag_scrub_mdsdir
mds: remove code to bypass dumping empty header scrub info
mds: dump_values no more needed
mds: enqueue ~mdsdir at the time of enqueing root

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51700 from cfsnyder/wip-cfsnyder-put-404

rgw: fix consistency bug with OLH objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/common: skip unittest_back_trace with mingw-gcc

We're getting an ICE when trying to compile this test using
mingw-gcc and recent Boost versions. Note that mingw-llvm works fine.

    during IPA pass: inline
    /mnt/data/workspace/ceph.pr/src/test/common/test_back_trace.cc:44:1:
    internal compiler error: Segmentation fault
       44 | }
          | ^
    0x7f9c4a86c51f ???
            ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
    0x7f9c4a853d8f __libc_start_call_main
            ../sysdeps/x86/libc-start.c:58
    0x7f9c4a853e3f __libc_start_main_impl
            ../sysdeps/nptl/libc_start_call_main.h:392

For now, we'll just skip the test when using mingw-gcc. Note
that we're planing to switch to mingw-llvm anyway:
https://github.com/ceph/ceph/pull/51197

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

win32_deps_build: skip patching removed boost files

We're attempting to patch some Python files that have been removed
from recent Boost versions.

Now that the Boost version has been bumped, we'll need to address
this in order to unblock the Windows build.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #52193 from aclamk/wip-aclamk-fix-55260

os/bluestore: Fixes ceph-bluestore-tool bluefs-bdev-new-db

Merge PR #52173 into main

* refs/pull/52173/head:
mds: MDLog::_recovery_thread: handle the errors gracefully

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/collection_manager: fill CollectionNode::decoded on
clean reads

Otherwise, deltas of collection nodes can't be replayed correctly,
because the data of nodes is not loaded to CollectionNode::decoded

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

qa/suites: install pytest for pybind tasks

Signed-off-by: Casey Bodley <cbodley@redhat.com>

RGW: fix issue in observability over the persistent topics queue
when releasing entires, we don't decrease them in the urgent data

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

os/bluestore: do not include single AU allocations to allocation
fragmentation stats.

The rationale is that such allocations always get single framgent.

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

test/allocator_replay: introduce allocator fragmentation histogram.

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

os/bluestore: implement allocator fragmentation histogram

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #52483 from bluikko/patch-15

doc/mgr/rgw.rst: add missing "ceph" command in cli specification

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/mgr/rgw.rst: add missing "ceph" command in cli specification

The line begins with a prompt so I would
expect it to be a "complete" command,
including the `ceph` binary to run with all
the various listed arguments.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge PR #51383 into main

* refs/pull/51383/head:
client: clean up the queue_cap_snap() code to remove unnecessary check
client: always add one new capsnap if Fb is used and Fw is not used

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #51850 into main

* refs/pull/51850/head:
client: do not send metrics until the MDS rank is ready

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #52422 from guits/cv-add-raw-osd-id-param

ceph-volume: add --osd-id option to raw prepare

Merge pull request #52476 from bluikko/patch-14

doc/mgr/rgw.rst: fix multisite typed wrong

Reviewed-by: Rongqi Sun (Svelar) <sunrongqi@huawei.co>

doc/mgr/rgw.rst: multisite typed wrong

Multisite is typed wrong as mutlisite so fix the spelling.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>