Merge PR #51536 into main

* refs/pull/51536/head:
mds: do not send split_realms for CEPH_SNAP_OP_UPDATE msg

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #52275 into main

* refs/pull/52275/head:
mds: update mdlog perf counters during replay

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52624 from idryomov/wip-61565

qa/workunits/rbd: use jammy version of qemu-iotests for centos 9

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #52546 from rhcs-dashboard/node-16-main

deps: increase the node version to 16

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #49810 from baergj/fix-create_time_event-header

async: Fix units in the create_time_event() declaration.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52644 from rhcs-dashboard/fix-62124-main

mgr/dashboard: add validations to zone access/secret key in zone creation/edit form

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #52570 from rhcs-dashboard/fix-62110-main

mgr/dashboard: fix removal of non-master zone from the zone group via dashboard

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #52600 into main

* refs/pull/52600/head:
qa: fix cephfs-mirror unwinding and 'fs volume create/rm' order

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard: add validations to zone access/secret key in zone creation/edit form

Fixes: https://tracker.ceph.com/issues/62124
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

qa: fix cephfs-mirror unwinding and 'fs volume create/rm' order

* Fixes the 'fs volume create' happens before the cephfs-mirror daemon start.
* Fixes the 'fs volume rm' happen only after the cephfs-mirror daemon unwinding.
  - This prevents the issue of mirror-daemon not returning from a libcephfs call, as
    the volumes were deleted during the cephfs_mirror_thrash ing.

Fixes: https://tracker.ceph.com/issues/61182
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #52630 from zdover23/wip-doc-2023-07-25-readmemd-1-of-x

doc: update README.md

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52596 from yuvalif/wip-yuval-fix-61868-new

rgw/lua: allow passing tenant without uid to manage lua scripts

async: Fix units in the create_time_event() declaration.

Commit 19a9d9531fab4627cafbb2915566c512c41077a8 changed the definition
to microseconds but neglected to update the header.

Signed-off-by: Joshua Baergen <jbaergen@digitalocean.com>

Merge pull request #52592 from xxhdx1985126/wip-62098

crimson/net: set TCP_NODELAY according to ms_tcp_nodelay

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kefu.chai@scylladb.com>

doc: update README.md

Correct and improve ceph/README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50423 from rkachach/fix_issue_nmveof

mgr/cephadm: adding support for nvmeof

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51933 from aclamk/wip-aclamk-bs-onode-doc

os/bluestore: Some in-code documentation for Onode / blob

Merge pull request #52218 from YiteGu/cephfs-get-statfs

mgr/dashboard: CephFS statfs REST API

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

qa/workunits/rbd: use jammy version of qemu-iotests for centos 9

It's the one we are using for all recent distros.

While at it, get rid of custom bin directory -- it appears that both
v2.3.0 and v2.11.0 tests are happy with just symlinks in the current
directory.

Fixes: https://tracker.ceph.com/issues/61565
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #52523 into main

* refs/pull/52523/head:
mds: remove dead method

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52429 from guits/fix-raw-list

ceph-volume: fix raw list for lvm devices

Merge PR #52597 into main

* refs/pull/52597/head:
qa: use centos 8.stream for fs:upgrade tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52599 from Matan-B/wip-matanb-crimson-cbt-fix

qa: Fix cbt usage with Crimson

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52598 from Matan-B/wip-matanb-crimson-centos-8

qa/suites/crimson-rados: Use centos8 for testing

Reviewed-by: Samuel Just <sjust@redhat.com>

qa: use centos 8.stream for fs:upgrade tests

fs:upgrade tests use n/o/p release packages which aren't
available for centos 9.stream - switch using 8.stream instead.

Fixes: http://tracker.ceph.com/issues/62146
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge PR #48038 into main

* refs/pull/48038/head:
client test: Add fsync to ll_preadv_pwritev test
libcephfs: Option to write + fsync via ceph_ll_nonblocking_readv_writev
Client: Hook nonblocking fsync into the write path of ll_preadv_pwritev
Client: Add non-blocking fsync
Client/Inode: wait_for_caps fixups
Client: change several waitfor_* to use Context list
test: Add nonblocking I/O client test
libcephfs: Add nonblocking readv/writev I/O interface
Client: Add ll_preadv_pwritev to expose non-blocking I/O to libcephfs
Client: Add non-blocking helper classes
Client: Break some code into new methods in prep for non-blocking I/O
Buffers: Add function to buffer.h to copy bufferlist to an iovec
ObjectCacher: Prepare file_write path for non-blocking I/O

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #52607 from zap51/patch-1

doc/rbd/rbd-cloudstack.rst: include support for multiple monitors

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49889 from bluikko/bluikko-doc-srv-lookup-link

doc/rados/configuration: add links to MON DNS

doc/rbd/rbd-cloudstack.rst: include support for multiple monitors

Signed-off-by: Jayanth Reddy <45934027+zap51@users.noreply.github.com>

client test: Add fsync to ll_preadv_pwritev test

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

libcephfs: Option to write + fsync via ceph_ll_nonblocking_readv_writev

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Hook nonblocking fsync into the write path of ll_preadv_pwritev

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Add non-blocking fsync

We will need the ability to do an non-blocking write that finishes with
fsync so we need non-blocking fsync.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client/Inode: wait_for_caps fixups

The non-blocking flush requires us to be able to re-add to
wait_for_caps but if we simply add to the list, we get stuck in an
infinite loop. Add a wait_for_caps_pending list to add to, and then
when done signalling, we move the wait_for_caps_pending items onto the
wait_for_caps list.

Also in handle_cap_flush_ack(), we need to complete the caps flushing
before signalling since with non-blocking flush, we will be actually
examining the caps from the completion rather than signalling a
condition variable in the completion.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: change several waitfor_* to use Context list

Change waitfor_caps, waitfor_safe and waitfor_commit to Context list.

To make a non-blocking version of fsync (to be used for non-blocking write
and commit), we need to be able to signal an arbitrary Context on completion
of either of these lists.

add_nonblocking_onfinish_to_context_list Adds such a Context to the list.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

test: Add nonblocking I/O client test

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

libcephfs: Add nonblocking readv/writev I/O interface

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Add ll_preadv_pwritev to expose non-blocking I/O to libcephfs

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Add non-blocking helper classes

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Break some code into new methods in prep for non-blocking I/O

These bits of code need to be invoked from a separate spot when we
introduce non-blocking I/O, so break them out now.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Buffers: Add function to buffer.h to copy bufferlist to an iovec

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

ObjectCacher: Prepare file_write path for non-blocking I/O

For non-blocking I/O, we will want to be able to override
block_writes_upfront so rename the member cfg_block_writes_upfront and add
an option to pass block_writes_upfront as a parameter along with a member
access method so caller can pass cfg_block_writes_upfront.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Merge pull request #52521 from cbodley/wip-62059

valgrind: update suppression for SyscallParam under call_init

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #52601 from idryomov/wip-doc-redundant-tiering-note

doc/rados: remove redundant cache tiering deprecation note

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #51934 from kotreshhr/fix-assert-rename

mds: Fix the linkmerge assert check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52473 from vedanshbhartia/null_deref

rgw: add coverity annotations and asserts for null dereferences

Merge pull request #52437 from vedanshbhartia/coverity_missing_break

rgw: Add coverity annotation for missing break statement in switch

Merge pull request #52436 from vedanshbhartia/coverity_logical_dead

rgw: Remove logically dead code

Merge pull request #52434 from vedanshbhartia/coverity_1511179

rgw: Fix potential null pointer dereference in rgw_user.cc

Merge pull request #52433 from vedanshbhartia/coverity_1510266

rgw: When finding bucket owner info, handle account name being empty

Merge pull request #52432 from vedanshbhartia/coverity_1511177

rgw: Fix potential null dereference error in bucket name parsing

Merge pull request #52253 from vedanshbhartia/coverity_1515338

rgw: Init res id with cls_2pc_reservation::NO_ID

Merge pull request #51648 from vedanshbhartia/coverity_1510684

rgw: refactor D3nDataCache::d3n_io_write to avoid resource leaks

Merge pull request #51772 from caisan/multisite-hang-period-update

rgw:multisite hang when update and commit period

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #52056 from cbodley/wip-56993

rgw: s3 object lock avoids overflow in retention date

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #52036 from adamemerson/wip-rgw-mdlog-yield

rgw: Thread optional_yield through RGWMetadataLog operations

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

valgrind: update suppression for SyscallParam under call_init

centos9 has a slightly different call stack here

Fixes: https://tracker.ceph.com/issues/62059
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52529 from xxhdx1985126/wip-61943

crimson/osd/osd_operations: add pipeline to LogMissingRequest to sync it

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

doc/rados: remove redundant cache tiering deprecation note

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/crimson-rados/perf: add monitoring_profiles for cbt

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/tasks/cbt.py: Fix perf package naming

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados: Use centos8 for testing

centos_latest was moved to centos9, we only schedule centos8
Crimson builds.
Use centos8 until Crimson will support 9.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #52439 from AliMasarweh/wip-alimasa-persistant-q-observability

RGW: fix issue in observability over the persistent topics queue

crimson/osd/osd_operations: add pipeline to LogMissingRequest to sync it
with ReplicateRequest

This commit is a follow-up for e14142894922b403344d7f08315b67a291b9a27a
and f4d336a2dbf864fe798ca7f500d44576d3f626d5

Fixes: https://tracker.ceph.com/issues/61943
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

doc/rados/configuration: add links to MON DNS

Add two links to MON DNS lookup and an anchor in the destination document.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #52221 from xxhdx1985126/wip-collection-bug-fix

crimson/os/seastore/collection_manager: fill CollectionNode::decoded on clean reads

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #52446 from xxhdx1985126/wip-seastore-lazy-read-patch

crimson/os/seastore/transaction_manager: data loss issues

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/net: set TCP_NODELAY according to ms_tcp_nodelay

There are cases in which we need low latency of an individual op, like
updating RBD image's object map which would block other ops for a
object not created yet. We need to make sure the rbd-object-map-update
like ops have as low latency as possible

Fixes: https://tracker.ceph.com/issues/62098
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #52445 from soumyakoduri/wip-skoduri-syncpolicy-fixes

rgw/sync-policy: Correct "sync status" & "sync group" commands

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw/lua: allow passing tenant without uid to manage lua scripts

Fixes: https://tracker.ceph.com/issues/61868
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw: Read `sync status` from only the zones allowed to sync from

Using multisite sync policy, zones within a zonegroup can be configured
not to sync from certain zones. Filter such zones while reading sync
status.

Fixes: https://tracker.ceph.com/issues/62014
Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/sync-policy: Do not allow empty group names

Fixes: https://tracker.ceph.com/issues/62015
Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #52587 from zdover23/wip-doc-2023-07-22-rados-configuration-ceph-conf-4-of-x

doc/rados: edit ceph-conf.rst (4 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit ceph-conf.rst (4 of x)

Edit doc/rados/configuration/ceph-conf.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51820 from ifed01/wip-ifed-fragmentation-info

os/bluestore: introduce allocator state histogram

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #52542 from zdover23/wip-doc-2023-07-19-rados-configuration-ceph-conf-3-of-x

doc/rados: edit ceph-conf.rst (3 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit ceph-conf.rst (3 of x)

Edit doc/rados/configuration/ceph-conf.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52475 from yuvalif/wip-yuval-multisite-vstart-params

rgw/test: allow passing vstart parameters to multisite tests script

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: fix removal of non-master zone from the zone group via dashboard

Fixes: https://tracker.ceph.com/issues/62110
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #52057 from nbalacha/tracker-61672

rbd-mirror: fix race preventing local image deletion

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

deps: increase the node version to 16

With node 14, the angular 13 is going into some issues with node-gyp.
```
prebuild-install || node-gyp rebuild

prebuild-install WARN install No prebuilt binaries found (target=14.20.0 runtime=node arch=x64 libc= platform=linux)
make: Entering directory '/home/jenkins-build/build/workspace/ceph-dashboard-cephadm-e2e/src/pybind/mgr/dashboard/frontend/node_modules/tree-sitter/build'
  CC(target) Release/obj.target/tree_sitter/vendor/tree-sitter/lib/src/lib.o
  AR(target) Release/obj.target/tree_sitter.a
  COPY Release/tree_sitter.a
  CXX(target) Release/obj.target/tree_sitter_runtime_binding/src/binding.o
```
and in some other environments its like `cb() never called` and stops
the whole `npm ci` process.

After some research, looks like updating node is the way to stop this.

Also, ` npm 7 will block installations if an upstream dependency
conflict is present that cannot be automatically resolved.` https://github.blog/2021-02-02-npm-7-is-now-generally-available/#peer-dependencies

Fixes: https://tracker.ceph.com/issues/62107
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52025 from rzarzynski/wip-ec-drop-unused-requires_overwrite

osd: drop the unused ECTransaction::requires_overwrite()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #52491 from cbodley/wip-61566

valgrind: replace suppression for EVP_DecryptFinal_ex

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52427 from petrutlucian94/fix_win32_boost

win32_deps_build: skip patching removed boost files

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #52143 from cbodley/wip-61567

test/pybind: replace nose with pytest

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52563 from zdover23/wip-doc-2023-07-21-governance-orman-email-update

doc: governance.rst - update D Orman

doc: governance.rst - update D Orman

Update David Orman's email address in doc/governance.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52544 from rhcs-dashboard/debug-make-check

install-deps: remove the legacy resolver flags

valgrind: remove stale openssl suppressions

years ago in commit fe97a000990632dbe9734e0d3a96df627f104f7a, the
signature of authenticated_decrypt_update_final() changed and this
suppression no longer matches its mangled form:

-  fun:_ZN4ceph6crypto6onwire25AES128GCM_OnWireRxHandler34authenticated_decrypt_update_finalEONS_6buffer7v*4listEj
+  fun:_ZN4ceph6crypto6onwire25AES128GCM_OnWireRxHandler34authenticated_decrypt_update_finalERNS_6buffer7v15_2_04listE

the fact that we didn't notice must mean that we don't need the
suppressions anymore

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52374 from rosinL/wip-cleanup

crush/CrushLocation: remove useless flag

Reviewed-by: Samuel Just <sjust@redhat.com>

valgrind: suppress uninitialized value in OpenSSL

Fixes: https://tracker.ceph.com/issues/61566
Signed-off-by: Casey Bodley <cbodley@redhat.com>

install-deps: remove the legacy resolver flags

This was a workaround that was introduced long time ago. This will be
something that could be deprectaed at some point [1]. And its preventing some of the dependencies to be
downloaded or stored into the wheelhouse. Deps like jsonschema, parse,
mypy, cryptography etc.

[1] https://pip.pypa.io/en/latest/user_guide/#deprecation-timeline

Fixes: https://tracker.ceph.com/issues/62082
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #49332 from shreyanshjain7174/stuck_peering

mon: stuck peering since warning is misleading

Reviewed-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

Merge pull request #51438 from NitzanMordhai/wip-nitzan-cbt-perf-ci

cbt perf ci

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50301 from liuqinfei/support_compile_on_openeuler_main

ceph.spec.in: add support for openEuler OS

Reviewed-by: Rongqi Sun <sunrongqi@huawei.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #49480 from flyingcircusio/fix-mon-delete-iterator

 mon: fix iterator mishandling in PGMap::apply_incremental

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52021 from xxhdx1985126/wip-61650

osd/PeeringState: recheck pg readability when renewing leases for no-replica pools

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50441 from cbodley/wip-qa-distros-centos9

qa/distros: add centos stream 9 as supported distro

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #52124 from idryomov/wip-61733

mon/MonClient: resurrect original client_mount_timeout handling

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-volume: fix raw list for lvm devices

raw list shouldn't list lvm OSDs.
This confuses the generic activate process given that it relies
on the raw list to detect which kind of OSD is being activated.

Fixes: https://tracker.ceph.com/issues/62002
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>