rgw: robust_notify() logs the watchers that timed out

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: reenable cache notify config variables

these config options are meant to control this cache notification
behavior, but were never moved from RGWRados into RGWSI_Notify

raise the default rgw_max_notify_retries to 10 to match the adjusted
value from https://github.com/ceph/ceph/pull/42251

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: retry metadata cache notifications with INVALIDATE_OBJ

we encode the `INVALIDATE_OBJ` response in `retrybl`, but were
accidentally resending the notification with `bl` instead

Fixes: https://tracker.ceph.com/issues/62250
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #51987 from rkachach/fix_issue_61628

mgr/cephadm: storing prometheus/alertmanager credentials in monstore

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #52622 from rhcs-dashboard/allow-put-cors

mgr/dashboard: allow PUT in CORS

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #52317 from rhcs-dashboard/rgw-overview-dashboard

mgr/dashboard: add inventory card  and single stat cards to rgw overview dashboard

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52435 from vedanshbhartia/coverity_invalid_typename

rgw: typecast long long to int when passing length to format string

Merge pull request #52688 from petrutlucian94/avoid_tzset

build: globally set FMT_USE_TZSET=0 for Windows

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52668 from bluikko/patch-16

doc/radosgw: Add missing space to date option spec in admin.rst

Merge pull request #52685 from bluikko/patch-17

doc/mgr/ceph_api: Promptify example commands in index.rst

build: globally set FMT_USE_TZSET=0 for Windows

We're currently setting FMT_USE_TZSET=0 when building libfmt
in order to avoid the _tzset function, which is unavailable
under Mingw:
https://github.com/ceph/ceph/commit/aa5769ecf1d80fc9824280d2e90fd4c61a0e7769

The issue is that it still gets used by fmt/chrono.h, which is
why we'll move this definition to the top level cmake file.

Note that the Windows build is currently failing as a result of
a recent change: https://github.com/ceph/ceph/pull/52590/files

  In file included from ceph/src/common/ceph_time.h:22,
                   from ceph/src/include/encoding.h:31,
                   from ceph/src/include/uuid.h:9,
                   from ceph/src/include/types.h:21,
                   from ceph/src/crush/CrushWrapper.h:14,
                   from ceph/src/crush/CrushCompiler.h:7,
                   from ceph/src/crush/CrushCompiler.cc:4:
  ceph/src/fmt/include/fmt/chrono.h: In lambda function:
  ceph/src/fmt/include/fmt/chrono.h:953:5: error: ‘_tzset’ was
  not declared in this scope; did you mean ‘tzset’?
    953 |     _tzset();
        |     ^~~~~~
        |     tzset

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #52191 from rhcs-dashboard/fix-daemon-labels-exporter

mgr/dashboard: empty grafana panels for performance of daemons

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: allow PUT in CORS

Fixes: https://tracker.ceph.com/issues/62222
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: empty grafana panels for performance of daemons

Fixes: https://tracker.ceph.com/issues/61792
Signed-off-by: avanthakkar <avanjohn@gmail.com>
Removing the `ceph-` prefix from ceph_daemon label to adopt it with the label
format used by queries in grafana dashboards. Also changing the
`instance_id` label for rgw to match the values coming from
exporter and prometheus module

doc/mgr/ceph_api: Promptify example commands in index.rst

Use the more modern prompt block instead of
using code blocks for example commands.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc/radosgw: Add missing space to date option spec in admin.rst

The start time and end time CLI option specification is missing a space between the date and the optional time value. Also expand the text to talk about "optional time" after the date.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #52549 from cbodley/wip-qa-distros-centos9-no-copr

qa/distros: disable ceph/el9 copr workaround for missing python deps

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52671 from zdover23/wip-doc-2023-07-27-readmemd-2-of-x

doc: update README.md install procedure

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52591 from ronen-fr/wip-rf-cephx-init

cephx: initializing two member variables in the ctors

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #52590 from ronen-fr/wip-rf-res-type2

osd/scrub: fixing & improving ReservationTimeout handler messages

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52084 from rhcs-dashboard/fix-exporter-addrs

exporter: ceph-exporter scrapes failing on multi-homed server

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@ibm.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

doc: update README.md install procedure

Add instructions directing the reader to install the "python3-routes"
package. This package is required in order to launch the dashboard after
the installation procedure has completed, but is not yet included in the
install-deps.sh script.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #52376 into main

* refs/pull/52376/head:
mds: skip forwarding request if the session were removed

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #52176 into main

* refs/pull/52176/head:
mds: remove calculating caps after adding revokes back

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #51536 into main

* refs/pull/51536/head:
mds: do not send split_realms for CEPH_SNAP_OP_UPDATE msg

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #52275 into main

* refs/pull/52275/head:
mds: update mdlog perf counters during replay

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard: Add inventory card and two single stat panels to rgw
overview dashboard

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #52624 from idryomov/wip-61565

qa/workunits/rbd: use jammy version of qemu-iotests for centos 9

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #52546 from rhcs-dashboard/node-16-main

deps: increase the node version to 16

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #49810 from baergj/fix-create_time_event-header

async: Fix units in the create_time_event() declaration.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52644 from rhcs-dashboard/fix-62124-main

mgr/dashboard: add validations to zone access/secret key in zone creation/edit form

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #52570 from rhcs-dashboard/fix-62110-main

mgr/dashboard: fix removal of non-master zone from the zone group via dashboard

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #52600 into main

* refs/pull/52600/head:
qa: fix cephfs-mirror unwinding and 'fs volume create/rm' order

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard: add validations to zone access/secret key in zone creation/edit form

Fixes: https://tracker.ceph.com/issues/62124
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

qa: fix cephfs-mirror unwinding and 'fs volume create/rm' order

* Fixes the 'fs volume create' happens before the cephfs-mirror daemon start.
* Fixes the 'fs volume rm' happen only after the cephfs-mirror daemon unwinding.
  - This prevents the issue of mirror-daemon not returning from a libcephfs call, as
    the volumes were deleted during the cephfs_mirror_thrash ing.

Fixes: https://tracker.ceph.com/issues/61182
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #52630 from zdover23/wip-doc-2023-07-25-readmemd-1-of-x

doc: update README.md

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52596 from yuvalif/wip-yuval-fix-61868-new

rgw/lua: allow passing tenant without uid to manage lua scripts

async: Fix units in the create_time_event() declaration.

Commit 19a9d9531fab4627cafbb2915566c512c41077a8 changed the definition
to microseconds but neglected to update the header.

Signed-off-by: Joshua Baergen <jbaergen@digitalocean.com>

Merge pull request #52592 from xxhdx1985126/wip-62098

crimson/net: set TCP_NODELAY according to ms_tcp_nodelay

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kefu.chai@scylladb.com>

doc: update README.md

Correct and improve ceph/README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>

using json file to get alertmanager/prometheus credentials

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

mgr/cephadm: storing prometheus/alertmanager credentials in monstore
Fixes: https://tracker.ceph.com/issues/61628
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #50423 from rkachach/fix_issue_nmveof

mgr/cephadm: adding support for nvmeof

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51933 from aclamk/wip-aclamk-bs-onode-doc

os/bluestore: Some in-code documentation for Onode / blob

Merge pull request #52218 from YiteGu/cephfs-get-statfs

mgr/dashboard: CephFS statfs REST API

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

qa/workunits/rbd: use jammy version of qemu-iotests for centos 9

It's the one we are using for all recent distros.

While at it, get rid of custom bin directory -- it appears that both
v2.3.0 and v2.11.0 tests are happy with just symlinks in the current
directory.

Fixes: https://tracker.ceph.com/issues/61565
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #52523 into main

* refs/pull/52523/head:
mds: remove dead method

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52429 from guits/fix-raw-list

ceph-volume: fix raw list for lvm devices

Merge PR #52597 into main

* refs/pull/52597/head:
qa: use centos 8.stream for fs:upgrade tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52599 from Matan-B/wip-matanb-crimson-cbt-fix

qa: Fix cbt usage with Crimson

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52598 from Matan-B/wip-matanb-crimson-centos-8

qa/suites/crimson-rados: Use centos8 for testing

Reviewed-by: Samuel Just <sjust@redhat.com>

qa: use centos 8.stream for fs:upgrade tests

fs:upgrade tests use n/o/p release packages which aren't
available for centos 9.stream - switch using 8.stream instead.

Fixes: http://tracker.ceph.com/issues/62146
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge PR #48038 into main

* refs/pull/48038/head:
client test: Add fsync to ll_preadv_pwritev test
libcephfs: Option to write + fsync via ceph_ll_nonblocking_readv_writev
Client: Hook nonblocking fsync into the write path of ll_preadv_pwritev
Client: Add non-blocking fsync
Client/Inode: wait_for_caps fixups
Client: change several waitfor_* to use Context list
test: Add nonblocking I/O client test
libcephfs: Add nonblocking readv/writev I/O interface
Client: Add ll_preadv_pwritev to expose non-blocking I/O to libcephfs
Client: Add non-blocking helper classes
Client: Break some code into new methods in prep for non-blocking I/O
Buffers: Add function to buffer.h to copy bufferlist to an iovec
ObjectCacher: Prepare file_write path for non-blocking I/O

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #52607 from zap51/patch-1

doc/rbd/rbd-cloudstack.rst: include support for multiple monitors

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49889 from bluikko/bluikko-doc-srv-lookup-link

doc/rados/configuration: add links to MON DNS

doc/rbd/rbd-cloudstack.rst: include support for multiple monitors

Signed-off-by: Jayanth Reddy <45934027+zap51@users.noreply.github.com>

client test: Add fsync to ll_preadv_pwritev test

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

libcephfs: Option to write + fsync via ceph_ll_nonblocking_readv_writev

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Hook nonblocking fsync into the write path of ll_preadv_pwritev

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Add non-blocking fsync

We will need the ability to do an non-blocking write that finishes with
fsync so we need non-blocking fsync.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client/Inode: wait_for_caps fixups

The non-blocking flush requires us to be able to re-add to
wait_for_caps but if we simply add to the list, we get stuck in an
infinite loop. Add a wait_for_caps_pending list to add to, and then
when done signalling, we move the wait_for_caps_pending items onto the
wait_for_caps list.

Also in handle_cap_flush_ack(), we need to complete the caps flushing
before signalling since with non-blocking flush, we will be actually
examining the caps from the completion rather than signalling a
condition variable in the completion.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: change several waitfor_* to use Context list

Change waitfor_caps, waitfor_safe and waitfor_commit to Context list.

To make a non-blocking version of fsync (to be used for non-blocking write
and commit), we need to be able to signal an arbitrary Context on completion
of either of these lists.

add_nonblocking_onfinish_to_context_list Adds such a Context to the list.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

test: Add nonblocking I/O client test

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

libcephfs: Add nonblocking readv/writev I/O interface

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Add ll_preadv_pwritev to expose non-blocking I/O to libcephfs

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Add non-blocking helper classes

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Break some code into new methods in prep for non-blocking I/O

These bits of code need to be invoked from a separate spot when we
introduce non-blocking I/O, so break them out now.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Buffers: Add function to buffer.h to copy bufferlist to an iovec

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

ObjectCacher: Prepare file_write path for non-blocking I/O

For non-blocking I/O, we will want to be able to override
block_writes_upfront so rename the member cfg_block_writes_upfront and add
an option to pass block_writes_upfront as a parameter along with a member
access method so caller can pass cfg_block_writes_upfront.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Merge pull request #52521 from cbodley/wip-62059

valgrind: update suppression for SyscallParam under call_init

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #52601 from idryomov/wip-doc-redundant-tiering-note

doc/rados: remove redundant cache tiering deprecation note

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: typecast long long to int when passing length to format string

Signed-off-by: Vedansh Bhartia <vedanshbhartia@gmail.com>

Merge pull request #51934 from kotreshhr/fix-assert-rename

mds: Fix the linkmerge assert check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52473 from vedanshbhartia/null_deref

rgw: add coverity annotations and asserts for null dereferences

Merge pull request #52437 from vedanshbhartia/coverity_missing_break

rgw: Add coverity annotation for missing break statement in switch

Merge pull request #52436 from vedanshbhartia/coverity_logical_dead

rgw: Remove logically dead code

Merge pull request #52434 from vedanshbhartia/coverity_1511179

rgw: Fix potential null pointer dereference in rgw_user.cc

Merge pull request #52433 from vedanshbhartia/coverity_1510266

rgw: When finding bucket owner info, handle account name being empty

Merge pull request #52432 from vedanshbhartia/coverity_1511177

rgw: Fix potential null dereference error in bucket name parsing

Merge pull request #52253 from vedanshbhartia/coverity_1515338

rgw: Init res id with cls_2pc_reservation::NO_ID

Merge pull request #51648 from vedanshbhartia/coverity_1510684

rgw: refactor D3nDataCache::d3n_io_write to avoid resource leaks

Merge pull request #51772 from caisan/multisite-hang-period-update

rgw:multisite hang when update and commit period

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #52056 from cbodley/wip-56993

rgw: s3 object lock avoids overflow in retention date

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #52036 from adamemerson/wip-rgw-mdlog-yield

rgw: Thread optional_yield through RGWMetadataLog operations

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

valgrind: update suppression for SyscallParam under call_init

centos9 has a slightly different call stack here

Fixes: https://tracker.ceph.com/issues/62059
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52529 from xxhdx1985126/wip-61943

crimson/osd/osd_operations: add pipeline to LogMissingRequest to sync it

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

doc/rados: remove redundant cache tiering deprecation note

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/crimson-rados/perf: add monitoring_profiles for cbt

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/tasks/cbt.py: Fix perf package naming

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados: Use centos8 for testing

centos_latest was moved to centos9, we only schedule centos8
Crimson builds.
Use centos8 until Crimson will support 9.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #52439 from AliMasarweh/wip-alimasa-persistant-q-observability

RGW: fix issue in observability over the persistent topics queue

crimson/osd/osd_operations: add pipeline to LogMissingRequest to sync it
with ReplicateRequest

This commit is a follow-up for e14142894922b403344d7f08315b67a291b9a27a
and f4d336a2dbf864fe798ca7f500d44576d3f626d5

Fixes: https://tracker.ceph.com/issues/61943
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

doc/rados/configuration: add links to MON DNS

Add two links to MON DNS lookup and an anchor in the destination document.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #52221 from xxhdx1985126/wip-collection-bug-fix

crimson/os/seastore/collection_manager: fill CollectionNode::decoded on clean reads

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #52446 from xxhdx1985126/wip-seastore-lazy-read-patch

crimson/os/seastore/transaction_manager: data loss issues

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xinyu Huang <xinyu.huang@intel.com>

crimson/net: set TCP_NODELAY according to ms_tcp_nodelay

There are cases in which we need low latency of an individual op, like
updating RBD image's object map which would block other ops for a
object not created yet. We need to make sure the rbd-object-map-update
like ops have as low latency as possible

Fixes: https://tracker.ceph.com/issues/62098
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #52445 from soumyakoduri/wip-skoduri-syncpolicy-fixes

rgw/sync-policy: Correct "sync status" & "sync group" commands

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw/lua: allow passing tenant without uid to manage lua scripts

Fixes: https://tracker.ceph.com/issues/61868
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw: Read `sync status` from only the zones allowed to sync from

Using multisite sync policy, zones within a zonegroup can be configured
not to sync from certain zones. Filter such zones while reading sync
status.

Fixes: https://tracker.ceph.com/issues/62014
Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/sync-policy: Do not allow empty group names

Fixes: https://tracker.ceph.com/issues/62015
Signed-off-by: Soumya Koduri <skoduri@redhat.com>