mgr/dashboard: Add more decimals in latency graph
Fixes: https://tracker.ceph.com/issues/61930
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #52293 from ronen-fr/wip-rf-consttime

common: make some common ctors and accessors constexpr

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge PR #52290 into main

* refs/pull/52290/head:
pybind/rbd: drop GIL when calling into librbd
pybind/cephfs: drop gil during cephfs callouts

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #50558 from rhcs-dashboard/fix-dataUnits-graph-plotting

mgr/dashboard: fix a bug where data would plot wrongly

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

common: make some common ctors and accessors constexpr

Add the constexpr qualifier to:
- some utime_t ctors and accessors;
- a shard_id_t ctor;
- ceph_le operator

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #51852 from Matan-B/wip-matanb-crimson-watch-notify-logs

crimson/osd/watch: Add logs around Watch/Notify

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52086 from pkalever/fix-mirror-status-desc

rbd-mirror: fix image replayer shut down description on force promote

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52207 from zdover23/wip-doc-2023-06-27-rados-operations-crush-map-edits-2-of-x-2nd-attempt

doc/rados: edit crush-map-edits (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit crush-map-edits (2 of x)

Edit doc/rados/operations/crush-map-edits.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52308 from zdover23/wip-doc-2023-07-05-seqdiag-test

doc/dev: remove seqdiag assets

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev: remove seqdiag assets

Remove seqdiag assets to determine whether the docs can be built if they
are absent. (Currently they cannot be built when they are present.) If
this works, then these diagrams will be replaced, probably with .png
files.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51422 from yuvalif/wip-yuval-cephadm-lua-doc

doc/rgw/lua: add info uploading a script in cephadm deployment

Reviewed-by: Zac Dover <zac.dover@proton.me>

pybind/rbd: drop GIL when calling into librbd

This was missing for rbd_mirror_peer_site_add() and
rbd_get_data_pool_id().

While at it, add a test for data_pool_id().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52247 from cbodley/wip-57905

rgw: rgwx-skip-decrypt also skips decompression of encrypted objects

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #52249 from mkogan1/wip-t61661-d3n-valg

rgw: d3n: fix valgrind reported leak related to libaio ops

Merge pull request #52160 from soumyakoduri/wip-skoduri-lc-acl

rgw/set_attrs: Do not modify mtime

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #51983 from clwluvw/aws4-date-header

rgw: pick http_date in case of http_x_amz_date absence

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

pybind/cephfs: drop gil during cephfs callouts

This has disastorous consequences including the possibility of deadlock.
In the best case, you have the rmdir holding the GIL until the MDS
responds!

Fixes: https://tracker.ceph.com/issues/61869
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/osd/watch: don't print bufferlists

The bufferlists clutter the logs and don't infrom of anything useful:
```
DEBUG 2023-05-26 23:36:29,780 [shard 2] osd - send_completion sending notify replies: {notify_reply_t{watcher_gid=4220, watcher_cookie=139786421930752, bl=buffer::list(len=0,

)}, notify_reply_t{watcher_gid=4220, watcher_cookie=139786422032096, bl=buffer::list(len=0,

)}}
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/watch: Add logs around Watch/Notify

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #49044 from rzarzynski/wip-crimson-ertr-plain-rets

crimson/ertr: let ErrVisitorT return plain value if ValueFuncT returns seastar::future

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #52032 from Matan-B/wip-matanb-crimson-no-ignore

crimson/osd: don't ignore start_pg_operation returned future

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51147 from liu-chunmei/multicore-osd

Crimson/osd: support multicore osd

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51916 from cyx1231st/wip-seastar-msgr-multi-core-2

crimson/net: support connections in multiple shards

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52280 from zdover23/wip-doc-2023-07-01-radosgw-multisite

doc/radosgw: refine "Zones" in multisite.rst

doc/radosgw: refine "Zones" in multisite.rst

Describe in greater detail the function of Zones in serving S3 objects,
as requested by Alexander Patrakov here:
https://github.com/ceph/ceph/pull/49981#issuecomment-1615866374.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52277 from zdover23/wip-doc-2023-07-01-radosgw-s3select-overview

doc/radosgw: s/s3select/S3 Select/

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: s/s3select/S3 Select/

Correct several misspelling of "S3 Select". Hat tip to Anthony D'Atri,
who caught this in an earlier PR.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #47595 from 5cs/fix-lc-head-marker

rgwlc: fix lc head marker point to non-exist lc entry

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #50667 from ivancich/wip-add-reupload-testing

qa/rgw: test that multipart re-upload does not leave any orphans

Reviewed-by: Cory Snyder <csnyder@iland.com>

Merge pull request #52252 from soumyakoduri/wip-skoduri-dbstore

rgw/dbstore: Update README

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52217 from zhscn/fix-file-local-variable

crimson/os/seastore/omap_manager: correct editor settings

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #52250 from zdover23/wip-doc-2023-06-29-radosgw-s3select-basic-workflow

doc/radosgw: edit "Basic Workflow" in s3select.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit "Basic Workflow" in s3select.rst

Edit the "Basic Workflow" section in doc/radosgw/s3select.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw/dbstore: Update README

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw: d3n: fix valgrind reported leak related to libaio ops

Fixes: https://tracker.ceph.com/issues/61661
The valgrind leak indication is a false positive in this case,
it is because the libaio internal thread have not timed out yet
when radosgw is terminated.

```
man aio_init
...
aio_idle_time
This field specifies the amount of time in seconds that a worker thread
should wait for further requests before terminating, after having
completed a previous request.   The
default value is 1.
...
```
for the sake of teuthology reducing the timeout

waiting for 2 minutes for example like below would also prevent the leak report
```
❯ env
LD_LIBRARY_PATH=/mnt/nvme5n1p1/src-git/ceph--up--master-clang/build/lib/:$LD_LIBRARY_PATH
PYTHONPATH=$PYTHONPATH:/mnt/nvme5n1p1/src-git/ceph--up--master-clang/build/lib/cython_modules/lib.3
RAGWEED_CONF=$(realpath ./ragweed.conf) RAGWEED_STAGES=prepare,check tox
-- -v |& ccze -Aonolookups ;   sleep 2m | pv -t ; pkill radosgw
```

Signed-off-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #52008 from rishabh-d-dave/improvements-authmon

mon: improvements and cleanup for AuthMonitor

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd/heartbeat: relax the order of replacement reset and accept

With the new implementation in messenger, the order of replacement reset
and accept events cannot be determined because they are from different
connections.

Modify the heatbeat logic to tolerate the both cases.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

rgw: fetch_remote_obj() preserves RGW_ATTR_COMPRESSION of encrypted objects

if the source object was both compressed and encrypted, preserve its
original compression attribute so it can be decompressed the same way it
was originally compressed

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fetch_remote_obj() will never verify etags of encrypted objects

fetch_remote_obj() transfers objects in their encrypted form, so does
not have access to the decrypted data for checksum verification

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rgwx-skip-decrypt also skips decompression of encrypted objects

compression is applied before encryption. so if we skip decryption, we
can't decompress either

Fixes: https://tracker.ceph.com/issues/57905
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52157 from pkalever/log-improve

rbd-mirror: add some assorted logs

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rgw: Do not update mtime in `set_attrs`

As per https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html,
the only way for users to modify object metadata is to make a copy of the object and
set the metadata. Hence retain old mtime for any other attr changes

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #52194 from zdover23/wip-doc-2023-06-26-radosgw-s3select-overview

doc/radosgw: edit "Overview" in s3select.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit "Overview" in s3select.rst

Edit the "Overview" section in doc/radosgw/s3select.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #49971 into main

* refs/pull/49971/head:
doc/cephfs: document MDS_CLIENTS_LAGGY health warning
qa: ignore warnings
qa: add test cases to check client eviction if an OSD is laggy
mds,messages: enable beacon to report clients lagginess
mds: do not evict client on laggy osds
common: add new config option to defer client eviction
osd: add method to check for laggy osds

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52090 from xxhdx1985126/wip-fixedkv-btree-merge-fix

crimson/os/seastore/btree: should add left's size when merging levels…

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

rbd-mirror: add some logs

Added some logs as their values where not very clear while parsing though the
log files.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

crimson/os/seastore/omap_manager: correct editor settings

Signed-off-by: Zhang Song <zhangsong325@gmail.com>

Merge PR #51858 into main

* refs/pull/51858/head:
pybind/mgr/devicehealth: do not crash if db not ready

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge PR #52170 into main

* refs/pull/52170/head:
qa: assign file system affinity for replaced MDS

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #51283 from rhcs-dashboard/disable-delete

mgr/dashboard: disable delete on multisite

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

rbd-mirror: fix image replayer shut down description on force promote

On force promote if the opposite site is down then we currently show
image status description as "local image linked to unknown peer"

Previously:
----------
$ rbd --cluster=site-b mirror image status pool1/img1
img1:
  global_id:   a73341a6-8302-4c97-ac6e-278083fd347e
  state:       up+stopping_replay
  description: local image linked to unknown peer
  service:     admin on localhost.localdomain
  last_update: 2023-06-15 19:47:45
  peer_sites:
    name: site-a
    state: up+stopped
    description: local image is primary
    last_update: 2023-06-15 19:47:32
  snapshots:
    9 .mirror.primary.a73341a6-8302-4c97-ac6e-278083fd347e.1f101367-277f-42f0-8308-e51201d0529a (peer_uuids:[c46c6d97-f59b-4591-9d35-d7ff9d0d72f7])

Currently:
---------
$ rbd --cluster=site-b mirror image status pool1/img1
img1:
  global_id:   2a6d61e1-8e76-42c4-af76-8f61ce65c7e2
  state:       up+stopped
  description: orphan (force promoting)
  service:     admin on localhost.localdomain
  last_update: 2023-06-15 19:29:22
  peer_sites:
    name: site-a
    state: down+stopped
    description: local image is primary
    last_update: 2023-06-15 19:29:05
  snapshots:
    9 .mirror.primary.2a6d61e1-8e76-42c4-af76-8f61ce65c7e2.99f82a30-0241-4e51-8428-7a2376d137f6 (peer_uuids:[3150c6ef-aeee-45dc-8d0e-5dc5a53d88eb])

Fixes: https://tracker.ceph.com/issues/52913
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

qa: assign file system affinity for replaced MDS

Otherwise, the MDS that just got replaced can transition to a rank
for another file system and the test cannot deterministically infer
which MDS needs to checked.

Fixes: http://tracker.ceph.com/issues/61764
Signed-off-by: Venky Shankar <vshankar@redhat.com>

crimson/net: cleanup, rename is_fixed_cpu

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #51928 from samarahu/cmake-cleanup

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge PR #52149 into main

* refs/pull/52149/head:
cephfs-journal-tool: disambiguate usage of all keyword (in tool help).

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52155 from nbalacha/poolreplayer-logging

rbd-mirror: add additional logs to PoolReplayer

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #51417 into main

* refs/pull/51417/head:
client: only wait for write MDS OPs when unmounting
client: flush the mdlog before waiting for unsafe requests

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #50908 into main

* refs/pull/50908/head:
mon/MDSMonitor: plug paxos when maybe manipulating the osdmap
mon/MDSMonitor: force immediate propose when evicting an MDS

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #49295 into main

* refs/pull/49295/head:
mgr/snap_schedule: catch all exceptions for cli

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51845 from ssdohammer-sl/doc-update-deduplication

doc:dev update how to use deduplication

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc:dev update how to use deduplication

Add prerequisite for deduplication, example outputs, and chunk-repair operation.

Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>

cephfs-journal-tool: disambiguate usage of all keyword (in tool help).

        The fs:all for rank option description was confusing. It seemd
like the fs was optional, but it is mandatory. This change modifies the
help message to reflect the correct way to use all in the --rank option.

Fixes: https://tracker.ceph.com/issues/61753
Signed-off-by: Manish M Yathnalli <myathnal@redhat.com>

Merge pull request #52181 from zdover23/wip-doc-2023-06-25-radosgw-s3select-formatting

doc/radosgw: remove pipes from s3select.rst

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #52110 from yuvalif/wip-yuval-fix-61639

rgw/amqp: remove possible race conditions with the amqp connections

doc/radosgw: remove pipes from s3select.rst

Remove strange formatting introduced in
61d6813726fc08980903c6a6c50b0a5797ba7193.

Note to the documentation team: This is not a line-edit. This commit
includes nothing but the removal of pipes added to the left of much of
the text in this file. Several future commits will line-edit this file
and correct its formatting.

Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw/amqp: use "fmt" for converting connection_id to string

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #51980 from aisakaki/wip-fix-check_node

crimson/os/seastore: fix in check_node

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/net: dispatch the initial acceptance from the socket core

It's meaningless to dispatch the initial acceptance always from the
msgr core.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: keep the order of cross-core events in msgr v2

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: expose is_fixed_cpu configuration at Messenger level

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: misc cleanups with logs around cross-core

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: convert all interactions between protocol and io-handler to be cross-core

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: prevent racing in protocol to switch core and to call io-handler interfaces

Otherwise, calling io-handler interfaces may result in wrong core/order.

This needs to take special care to handle preemptive cases such as
closing and replacing.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: implement logic to move a io-handler to a new sid

Note that it is inevitable that the user can mark down the connection
while the protocol is trying to move the connection to another core.

In that case, the implementation should tolerate the racing, which
finally needs to cleanup resources correctly and dispatch reasonable
events.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: expose the connection working shard to users

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: notify the new connection shard upon accept/connect

Users may need to know the new connection shard prior to message
dispatching. Otherwise, there will be no chance for user to do any
related preparations.

This is still a placeholder before multi-core messegner is enabled.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: make io-handler in/out dispatching aware of being switched

To prevent the previous shard-states racing on the shared data
structures after switching to a new shard-states.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: allow io-handler to dispatch in/out independently based on ctx

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: make send/keepalive keep trying until the shard is correct

It will be possible that during cross-core send/keepalive, the
connection shard is changed on-the-fly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: factor out IOHandler::shard_states_t which will be switchable atomically

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: misc cleanups to io handler implementations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: misc cleanups to protocol v2 implementations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: notify and update io_states from io_handler to protocol asynchronously

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: introduce a simpler reset_peer_state() with cleanups

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: clear FrameAssemblerV2 at replace

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: adjust the IO path in FrameAssemblerV2 with the foreign socket

FrameAssemblerV2 and Socket may in different cores during handshake.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: allow FrameAssemblerV2 to check and shutdown a foreign socket

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: check that FrameAssemblerV2 is working in the expected core

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: use SocketFRef

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: check that Socket only works in its core

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: wrap message with foreign-ptr in the send path

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: change the static IS_FIXED_CPU to runtime

Otherwise the messenger implementation needs to be templated.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: decouple tests from SocketConnection class

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: hide seastar packet and temporary_buffer inside Socket

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: introduce IOHandler::sid and the related assertions

Note that Connection and Messenger may be destructed in any order, so
keep a copy of msgr_sid in SocketConnection.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: move socket ownership from SocketConnection to FrameAssemblerV2

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: make sure Messenger is always called in the same shard

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net/socket: extend the sockets to multiple shards

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>