test/TestOSDMap: don't use the deprecated std::random_shuffle method

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Fixes: https://tracker.ceph.com/issues/62203

Merge pull request #52701 from rhcs-dashboard/flake8-fix

mgr: fix flake8 compliants

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: phlogistonjohn <NOT@FOUND>

Merge pull request #52341 from mkogan1/wip-T52363-xrmeng8756

rgw: fix the Content-Length in response header of static website

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52131 from cityofships/docs_radosgw

doc/radosgw: correct emphasis in rate limit section

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/radosgw: correct emphasis in rate limit section

Signed-off-by: Piotr Parczewski <piotr@stackhpc.com>

Merge pull request #52603 from cbodley/wip-62135

qa/rgw: pacific upgrade disables centos9

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #51987 from rkachach/fix_issue_61628

mgr/cephadm: storing prometheus/alertmanager credentials in monstore

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #52622 from rhcs-dashboard/allow-put-cors

mgr/dashboard: allow PUT in CORS

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr: fix some flake8 complaints

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52317 from rhcs-dashboard/rgw-overview-dashboard

mgr/dashboard: add inventory card  and single stat cards to rgw overview dashboard

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52435 from vedanshbhartia/coverity_invalid_typename

rgw: typecast long long to int when passing length to format string

Merge pull request #52688 from petrutlucian94/avoid_tzset

build: globally set FMT_USE_TZSET=0 for Windows

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52668 from bluikko/patch-16

doc/radosgw: Add missing space to date option spec in admin.rst

Merge pull request #52685 from bluikko/patch-17

doc/mgr/ceph_api: Promptify example commands in index.rst

build: globally set FMT_USE_TZSET=0 for Windows

We're currently setting FMT_USE_TZSET=0 when building libfmt
in order to avoid the _tzset function, which is unavailable
under Mingw:
https://github.com/ceph/ceph/commit/aa5769ecf1d80fc9824280d2e90fd4c61a0e7769

The issue is that it still gets used by fmt/chrono.h, which is
why we'll move this definition to the top level cmake file.

Note that the Windows build is currently failing as a result of
a recent change: https://github.com/ceph/ceph/pull/52590/files

  In file included from ceph/src/common/ceph_time.h:22,
                   from ceph/src/include/encoding.h:31,
                   from ceph/src/include/uuid.h:9,
                   from ceph/src/include/types.h:21,
                   from ceph/src/crush/CrushWrapper.h:14,
                   from ceph/src/crush/CrushCompiler.h:7,
                   from ceph/src/crush/CrushCompiler.cc:4:
  ceph/src/fmt/include/fmt/chrono.h: In lambda function:
  ceph/src/fmt/include/fmt/chrono.h:953:5: error: ‘_tzset’ was
  not declared in this scope; did you mean ‘tzset’?
    953 |     _tzset();
        |     ^~~~~~
        |     tzset

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #52191 from rhcs-dashboard/fix-daemon-labels-exporter

mgr/dashboard: empty grafana panels for performance of daemons

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: allow PUT in CORS

Fixes: https://tracker.ceph.com/issues/62222
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: empty grafana panels for performance of daemons

Fixes: https://tracker.ceph.com/issues/61792
Signed-off-by: avanthakkar <avanjohn@gmail.com>
Removing the `ceph-` prefix from ceph_daemon label to adopt it with the label
format used by queries in grafana dashboards. Also changing the
`instance_id` label for rgw to match the values coming from
exporter and prometheus module

doc/mgr/ceph_api: Promptify example commands in index.rst

Use the more modern prompt block instead of
using code blocks for example commands.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc/radosgw: Add missing space to date option spec in admin.rst

The start time and end time CLI option specification is missing a space between the date and the optional time value. Also expand the text to talk about "optional time" after the date.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #52549 from cbodley/wip-qa-distros-centos9-no-copr

qa/distros: disable ceph/el9 copr workaround for missing python deps

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52671 from zdover23/wip-doc-2023-07-27-readmemd-2-of-x

doc: update README.md install procedure

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52591 from ronen-fr/wip-rf-cephx-init

cephx: initializing two member variables in the ctors

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #52590 from ronen-fr/wip-rf-res-type2

osd/scrub: fixing & improving ReservationTimeout handler messages

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52084 from rhcs-dashboard/fix-exporter-addrs

exporter: ceph-exporter scrapes failing on multi-homed server

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@ibm.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

doc: update README.md install procedure

Add instructions directing the reader to install the "python3-routes"
package. This package is required in order to launch the dashboard after
the installation procedure has completed, but is not yet included in the
install-deps.sh script.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #52376 into main

* refs/pull/52376/head:
mds: skip forwarding request if the session were removed

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #52176 into main

* refs/pull/52176/head:
mds: remove calculating caps after adding revokes back

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #51536 into main

* refs/pull/51536/head:
mds: do not send split_realms for CEPH_SNAP_OP_UPDATE msg

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #52275 into main

* refs/pull/52275/head:
mds: update mdlog perf counters during replay

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard: Add inventory card and two single stat panels to rgw
overview dashboard

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #52624 from idryomov/wip-61565

qa/workunits/rbd: use jammy version of qemu-iotests for centos 9

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #52546 from rhcs-dashboard/node-16-main

deps: increase the node version to 16

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #49810 from baergj/fix-create_time_event-header

async: Fix units in the create_time_event() declaration.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52644 from rhcs-dashboard/fix-62124-main

mgr/dashboard: add validations to zone access/secret key in zone creation/edit form

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

rgw: fix the Content-Length in response header is inconsistent with response body size when rgw returns default html error page in static website

The default html error page as response body should be built completely include three ending html symbols(/ul, /body and /html) before rgw computes Content-Length in response header. The Content-Length in response header will be consistent with response body size. Client can get complete page.

Fixes: https://tracker.ceph.com/issues/52363
(cherry picked from commit cbeaef9fbe96c930fc10b793773e7ecd137fde81)

Co-authored-by: aicun hu <huaicun@chinatelecom.cn>
Co-authored-by: yupeng chen <chenyupeng@chinatelecom.cn>
Signed-off-by: xiangrui meng <mengxr@chinatelecom.cn>

Merge pull request #52570 from rhcs-dashboard/fix-62110-main

mgr/dashboard: fix removal of non-master zone from the zone group via dashboard

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #52600 into main

* refs/pull/52600/head:
qa: fix cephfs-mirror unwinding and 'fs volume create/rm' order

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard: add validations to zone access/secret key in zone creation/edit form

Fixes: https://tracker.ceph.com/issues/62124
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

qa: fix cephfs-mirror unwinding and 'fs volume create/rm' order

* Fixes the 'fs volume create' happens before the cephfs-mirror daemon start.
* Fixes the 'fs volume rm' happen only after the cephfs-mirror daemon unwinding.
  - This prevents the issue of mirror-daemon not returning from a libcephfs call, as
    the volumes were deleted during the cephfs_mirror_thrash ing.

Fixes: https://tracker.ceph.com/issues/61182
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #52630 from zdover23/wip-doc-2023-07-25-readmemd-1-of-x

doc: update README.md

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52596 from yuvalif/wip-yuval-fix-61868-new

rgw/lua: allow passing tenant without uid to manage lua scripts

async: Fix units in the create_time_event() declaration.

Commit 19a9d9531fab4627cafbb2915566c512c41077a8 changed the definition
to microseconds but neglected to update the header.

Signed-off-by: Joshua Baergen <jbaergen@digitalocean.com>

Merge pull request #52592 from xxhdx1985126/wip-62098

crimson/net: set TCP_NODELAY according to ms_tcp_nodelay

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kefu.chai@scylladb.com>

doc: update README.md

Correct and improve ceph/README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>

using json file to get alertmanager/prometheus credentials

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

mgr/cephadm: storing prometheus/alertmanager credentials in monstore
Fixes: https://tracker.ceph.com/issues/61628
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #50423 from rkachach/fix_issue_nmveof

mgr/cephadm: adding support for nvmeof

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51933 from aclamk/wip-aclamk-bs-onode-doc

os/bluestore: Some in-code documentation for Onode / blob

Merge pull request #52218 from YiteGu/cephfs-get-statfs

mgr/dashboard: CephFS statfs REST API

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

qa/workunits/rbd: use jammy version of qemu-iotests for centos 9

It's the one we are using for all recent distros.

While at it, get rid of custom bin directory -- it appears that both
v2.3.0 and v2.11.0 tests are happy with just symlinks in the current
directory.

Fixes: https://tracker.ceph.com/issues/61565
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #52523 into main

* refs/pull/52523/head:
mds: remove dead method

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52429 from guits/fix-raw-list

ceph-volume: fix raw list for lvm devices

Merge PR #52597 into main

* refs/pull/52597/head:
qa: use centos 8.stream for fs:upgrade tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52599 from Matan-B/wip-matanb-crimson-cbt-fix

qa: Fix cbt usage with Crimson

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52598 from Matan-B/wip-matanb-crimson-centos-8

qa/suites/crimson-rados: Use centos8 for testing

Reviewed-by: Samuel Just <sjust@redhat.com>

qa: use centos 8.stream for fs:upgrade tests

fs:upgrade tests use n/o/p release packages which aren't
available for centos 9.stream - switch using 8.stream instead.

Fixes: http://tracker.ceph.com/issues/62146
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge PR #48038 into main

* refs/pull/48038/head:
client test: Add fsync to ll_preadv_pwritev test
libcephfs: Option to write + fsync via ceph_ll_nonblocking_readv_writev
Client: Hook nonblocking fsync into the write path of ll_preadv_pwritev
Client: Add non-blocking fsync
Client/Inode: wait_for_caps fixups
Client: change several waitfor_* to use Context list
test: Add nonblocking I/O client test
libcephfs: Add nonblocking readv/writev I/O interface
Client: Add ll_preadv_pwritev to expose non-blocking I/O to libcephfs
Client: Add non-blocking helper classes
Client: Break some code into new methods in prep for non-blocking I/O
Buffers: Add function to buffer.h to copy bufferlist to an iovec
ObjectCacher: Prepare file_write path for non-blocking I/O

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #52607 from zap51/patch-1

doc/rbd/rbd-cloudstack.rst: include support for multiple monitors

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49889 from bluikko/bluikko-doc-srv-lookup-link

doc/rados/configuration: add links to MON DNS

doc/rbd/rbd-cloudstack.rst: include support for multiple monitors

Signed-off-by: Jayanth Reddy <45934027+zap51@users.noreply.github.com>

client test: Add fsync to ll_preadv_pwritev test

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

libcephfs: Option to write + fsync via ceph_ll_nonblocking_readv_writev

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Hook nonblocking fsync into the write path of ll_preadv_pwritev

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Add non-blocking fsync

We will need the ability to do an non-blocking write that finishes with
fsync so we need non-blocking fsync.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client/Inode: wait_for_caps fixups

The non-blocking flush requires us to be able to re-add to
wait_for_caps but if we simply add to the list, we get stuck in an
infinite loop. Add a wait_for_caps_pending list to add to, and then
when done signalling, we move the wait_for_caps_pending items onto the
wait_for_caps list.

Also in handle_cap_flush_ack(), we need to complete the caps flushing
before signalling since with non-blocking flush, we will be actually
examining the caps from the completion rather than signalling a
condition variable in the completion.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: change several waitfor_* to use Context list

Change waitfor_caps, waitfor_safe and waitfor_commit to Context list.

To make a non-blocking version of fsync (to be used for non-blocking write
and commit), we need to be able to signal an arbitrary Context on completion
of either of these lists.

add_nonblocking_onfinish_to_context_list Adds such a Context to the list.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

test: Add nonblocking I/O client test

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

libcephfs: Add nonblocking readv/writev I/O interface

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Add ll_preadv_pwritev to expose non-blocking I/O to libcephfs

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Add non-blocking helper classes

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Client: Break some code into new methods in prep for non-blocking I/O

These bits of code need to be invoked from a separate spot when we
introduce non-blocking I/O, so break them out now.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Buffers: Add function to buffer.h to copy bufferlist to an iovec

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

ObjectCacher: Prepare file_write path for non-blocking I/O

For non-blocking I/O, we will want to be able to override
block_writes_upfront so rename the member cfg_block_writes_upfront and add
an option to pass block_writes_upfront as a parameter along with a member
access method so caller can pass cfg_block_writes_upfront.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>

Merge pull request #52521 from cbodley/wip-62059

valgrind: update suppression for SyscallParam under call_init

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #52601 from idryomov/wip-doc-redundant-tiering-note

doc/rados: remove redundant cache tiering deprecation note

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: typecast long long to int when passing length to format string

Signed-off-by: Vedansh Bhartia <vedanshbhartia@gmail.com>

Merge pull request #51934 from kotreshhr/fix-assert-rename

mds: Fix the linkmerge assert check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52473 from vedanshbhartia/null_deref

rgw: add coverity annotations and asserts for null dereferences

Merge pull request #52437 from vedanshbhartia/coverity_missing_break

rgw: Add coverity annotation for missing break statement in switch

Merge pull request #52436 from vedanshbhartia/coverity_logical_dead

rgw: Remove logically dead code

Merge pull request #52434 from vedanshbhartia/coverity_1511179

rgw: Fix potential null pointer dereference in rgw_user.cc

Merge pull request #52433 from vedanshbhartia/coverity_1510266

rgw: When finding bucket owner info, handle account name being empty

Merge pull request #52432 from vedanshbhartia/coverity_1511177

rgw: Fix potential null dereference error in bucket name parsing

Merge pull request #52253 from vedanshbhartia/coverity_1515338

rgw: Init res id with cls_2pc_reservation::NO_ID

Merge pull request #51648 from vedanshbhartia/coverity_1510684

rgw: refactor D3nDataCache::d3n_io_write to avoid resource leaks

Merge pull request #51772 from caisan/multisite-hang-period-update

rgw:multisite hang when update and commit period

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #52056 from cbodley/wip-56993

rgw: s3 object lock avoids overflow in retention date

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #52036 from adamemerson/wip-rgw-mdlog-yield

rgw: Thread optional_yield through RGWMetadataLog operations

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

valgrind: update suppression for SyscallParam under call_init

centos9 has a slightly different call stack here

Fixes: https://tracker.ceph.com/issues/62059
Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: pacific upgrade disables centos9

we don't build centos 9 packages for pacific, so the pacific upgrade
suite was failing with errors like:

Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=pacific

Fixes: https://tracker.ceph.com/issues/62135
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52529 from xxhdx1985126/wip-61943

crimson/osd/osd_operations: add pipeline to LogMissingRequest to sync it

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

doc/rados: remove redundant cache tiering deprecation note

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/crimson-rados/perf: add monitoring_profiles for cbt

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/tasks/cbt.py: Fix perf package naming

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados: Use centos8 for testing

centos_latest was moved to centos9, we only schedule centos8
Crimson builds.
Use centos8 until Crimson will support 9.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #52439 from AliMasarweh/wip-alimasa-persistant-q-observability

RGW: fix issue in observability over the persistent topics queue

crimson/osd/osd_operations: add pipeline to LogMissingRequest to sync it
with ReplicateRequest

This commit is a follow-up for e14142894922b403344d7f08315b67a291b9a27a
and f4d336a2dbf864fe798ca7f500d44576d3f626d5

Fixes: https://tracker.ceph.com/issues/61943
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

doc/rados/configuration: add links to MON DNS

Add two links to MON DNS lookup and an anchor in the destination document.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #52221 from xxhdx1985126/wip-collection-bug-fix

crimson/os/seastore/collection_manager: fill CollectionNode::decoded on clean reads

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>