crimson/net: notify the new connection shard upon accept/connect

Users may need to know the new connection shard prior to message
dispatching. Otherwise, there will be no chance for user to do any
related preparations.

This is still a placeholder before multi-core messegner is enabled.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: make io-handler in/out dispatching aware of being switched

To prevent the previous shard-states racing on the shared data
structures after switching to a new shard-states.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: allow io-handler to dispatch in/out independently based on ctx

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: make send/keepalive keep trying until the shard is correct

It will be possible that during cross-core send/keepalive, the
connection shard is changed on-the-fly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: factor out IOHandler::shard_states_t which will be switchable atomically

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: misc cleanups to io handler implementations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: misc cleanups to protocol v2 implementations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: notify and update io_states from io_handler to protocol asynchronously

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: introduce a simpler reset_peer_state() with cleanups

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: clear FrameAssemblerV2 at replace

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: adjust the IO path in FrameAssemblerV2 with the foreign socket

FrameAssemblerV2 and Socket may in different cores during handshake.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: allow FrameAssemblerV2 to check and shutdown a foreign socket

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: check that FrameAssemblerV2 is working in the expected core

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: use SocketFRef

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: check that Socket only works in its core

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: wrap message with foreign-ptr in the send path

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: change the static IS_FIXED_CPU to runtime

Otherwise the messenger implementation needs to be templated.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: decouple tests from SocketConnection class

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: hide seastar packet and temporary_buffer inside Socket

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: introduce IOHandler::sid and the related assertions

Note that Connection and Messenger may be destructed in any order, so
keep a copy of msgr_sid in SocketConnection.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: move socket ownership from SocketConnection to FrameAssemblerV2

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: make sure Messenger is always called in the same shard

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net/socket: extend the sockets to multiple shards

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson/test_socket: fix unaligned memory access

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: cleanup test_socket.cc

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: cleanups to Socket.h/cc

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: drop the template from FixedCPUServerSocket::accept()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #52167 from zdover23/wip-doc-2023-06-23-rados-operations-crush-map-edits-1-of-x

doc/rados: edit crush-map-edits.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/rados: edit crush-map-edits.rst (1 of x)

Edit doc/rados/operations/crush-map-edits.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #51974 into main

* refs/pull/51974/head:
doc: fix grammar in cephfs/standby

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #52164 from ljflores/wip-balancer-documentation-fix

doc/rados/operations: change file back to original name

doc/rados/operations: change file back to original name

The name of this file was changed in
4fcab2e7fc9f7ac170ede21cb07912b79926ccb9,
but on second thought, this could cause 404 situations.
Reverting the file name back to the original name.

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #52156 from cbodley/wip-qa-rgw-s3tests-extra-attrs

qa/rgw/s3tests: make extra_attrs additive

Merge pull request #52085 from alimaredia/wip-rgw-tempest-keystone-fix

Tempest tests Keystone Fix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa: rgw/tempest suite only runs on ubuntu latest

Signed-off-by: Ali Maredia <amaredia@redhat.com>

qa: update rgw openstack versions

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #51738 from phlogistonjohn/jjm-cephadm-new-deploy

cephadm: create a new version of the deploy command

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51499 from ljflores/wip-balancer-documentation

doc/man/8: add documentation about read balancer

doc/man/8: add documentation about read balancer

Signed-off-by: Laura Flores <lflores@redhat.com>
Co-authored-by: Josh Salomon <josh.salomon@gmail.com>

Merge pull request #51819 from mkogan1/wip-d3n-obj-w-slash

rgw: D3N: cache objects which oid contains slash

qa/s3tests: remove stale comment about rfc2616

this comment was about the old fails_strict_rfc2616 attr that was
removed in 6238eaaf0bcbcaac3c8628f68c088481f96c199a

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/s3tests: clean up extra_attrs and extra_args logic

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/s3tests: make extra_attrs additive

the s3tests.py task is filtering out several attrs by default. but
when dbstore uses `extra_attrs` to add 'not fails_on_dbstore', it
overwrites those other filters

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52129 from yuvalif/wip-yuval-kafka-memleak

rgw/kafka: fix potential memory leak when push to queue fails

Merge pull request #51961 from Matan-B/wip-matanb-crimson-single-mosdmap-handling

Crimson/osd: Disable concurrent MOSDMap handling

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #52072 from NitzanMordhai/wip-nitzan-add-ec-clay-teuthology-tests

qa: adding clay test to thrash erasure code big

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #51667 from Matan-B/wip-matanb-share-osdmap

osd/OSD: cleanup OSDService::maybe_share_map()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51450 from batrick/i59716

pybind/rados: keep byte representation if decode fails

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #50206 from TRYTOBE8TME/wip-rgw-yield-work

rgw: replace null_yields with optional ones

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/workunits/cephadm: update script use use cephadm _orch deploy

cephadm _orch deploy will eventually be the only deploy method so
we need to update the test script.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: dont set ctx.image if json deploy image is unset

If no image has been provided in the "deploy from" json do not set
ctx.image to it (empty-string or None) as we may have had a valid
value passed on the --image CLI option.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52123 from zdover23/wip-doc-2023-06-20-rados-operations-control-1-of-x

doc/rados: edit control.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit control.rst (1 of x)

Edit doc/rados/operations/control.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52109 from idryomov/wip-61616

librbd: avoid object map corruption in snapshots taken under I/O

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa/workunits/rbd: make continuous export-diff test actually work

The current version is pretty useless:

- "rbd bench" writes the same byte (0xff) over and over again, so
  almost all checksumming is in vain
- snapshots are taken in a steady state (i.e. not under I/O), so no
  race conditions can get exposed
- even with these caveats, it's not wired up into the suite

Redo this workunit to be a reliable reproducer for the issue fixed
in the previous commit and wire it up for both krbd and rbd-nbd.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

crimson/osd/osd_operations/pg_advance_map: 'from' epoch may vary between PGS explanation

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osdmap_gate: got_map() assert increase of current epoch

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #51827 from liu-chunmei/get_assert

crimson:  fix RefCountedObject::_get() assert

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/shard_services: get_or_create_pg() remove unused variable

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/shard_services: Refactor OSDSingletonState::load_map

Added logs
If case reconstructed - epoch is unsigned.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_meta: Abort instead of throwing runtime_error

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd: Add logs around handling of osdmaps

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: Fix assert of osd_debug_verify_cached_snaps

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #52104 from anthonyeleven/anthonyeleven/fixfft

doc/rados/operations: Improve formatting in crush-map.rst

doc/rados/operations: Improve formatting in crush-map.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

rgw/kafka: fix potential memory leak when push to queue fails

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge PR #51500 into main

* refs/pull/51500/head:
test/libcephfs: add test case for revoking caps
client: issue a cap release immediately if no cap exists
mds: add the revoking caps back to _revokes list
mds: move confirm_receipt() to Capability.cc

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #51655 into main

* refs/pull/51655/head:
mds: display sane hex value (0x0) for empty feature bit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

src/rgw: Additional changes based on comments

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: Changing null_yield to optional_yield

Changing null_yield in rgw_sal_rados.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: Changing null_yield to optional_yield

Changing null_yield in rgw_reshard.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: Changing null_yield to optional_yield

Passing optional_yield to rgw_rados.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rados: Changing null_yield to optional_yield

Passing optional_yield at all places in rgw_bucket.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: Changing null_yield to optional_yield

Focused on rgw_rados_operate() and passing null_yield in
rgw_lc.cc, rgw_gc.cc, rgw_object_expirer_core.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

rgw: pubsub: pass yield context

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rados: pass yield context in a few places

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #51681 from cbodley/wip-61349

librados: aio operate functions can set mtimes

Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #51737 from cbodley/wip-rocksdb-44

bump rocksdb submodule to latest ceph-reef-v7.9.2

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #51669 from ronen-fr/wip-rf-max-scrub-3

osd/scrub: increasing max_osd_scrubs to 3

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #48209 from kamoltat/wip-ksirivad-fix-tracker-57570

osd/OSDMap: Check for uneven weights & != 2 buckets post stretch mode

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rgw: D3N cache objects which oid contains slash

Fixes: https://tracker.ceph.com/issues/59739
Signed-off-by: Mark Kogan <mkogan@redhat.com>

mds: display sane hex value (0x0) for empty feature bit

Print a valid hex (0x0) during empty feature bit, so that the clients could recognize it.
When the _vec size becomes 0, print() function creates an invalid hex (0x) and 'perf stats'
crashes with the below error:
"
File "/opt/ceph/src/pybind/mgr/stats/fs/perf_stats.py", line 177, in notify_cmd
metric_features = int(metadata[CLIENT_METADATA_KEY]["metric_spec"]["metric_flags"]["feature_bits"], 16)
ValueError: invalid literal for int() with base 16: '0x'
"
This patch creates  a valid hex (0x0), when _vec size is 0.

Fixes: https://tracker.ceph.com/issues/59551
Signed-off-by: Jos Collin <jcollin@redhat.com>

librbd: stop passing IOContext to image dispatch write methods

This is a major footgun since any value passed e.g. at the API layer
may be stale by the time we get to object dispatch.  All callers are
passing the IOContext returned by get_data_io_context() for their
ImageCtx anyway, highlighting that the parameter is fictitious.

Only the read method can meaningfully take IOContext.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: use an up-to-date snap context when owning the exclusive lock

By effectively moving capturing of the snap context to the API layer,
commit 1d0a3b17f590 ("librbd: pass IOContext to image-extent IO
dispatch methods") introduced a nasty regression.  The snap context can
be captured only after exclusive lock is safely held for the duration
of dealing with the image request and even then must be refreshed if
a snapshot creation request is accepted from a peer.  This is needed to
ensure correctness of the object map in general and fast-diff states in
particular (OBJECT_EXISTS vs OBJECT_EXISTS_CLEAN) and object deltas
computed based off of them.  Otherwise the object map that is forked
for the snapshot isn't guaranteed to accurately reflect the contents of
the snapshot when the snapshot is taken under I/O (as in disabling the
object map may lead to different results being returned for reads).

The regression affects mainly differential backup and snapshot-based
mirroring use cases with object-map and/or fast-diff enabled: since
some object deltas may be incomplete, the destination image may get
corrupted.

This commit represents a reasonable minimal fix: IOContext passed
through to ImageDispatch is effected only for reads and just gets
ignored for writes.  The next commit cleans up further by undoing the
passing of IOContext through the image dispatch layers for writes.

Fixes: https://tracker.ceph.com/issues/61616
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52105 from gsantella/patch-update-typo-on-dashboard

doc/mgr/dashboard: fix a typo

doc/mgr/dashboard: fix a typo

Signed-off-by: Guido Santella <gsantella@southhills.edu>

Merge pull request #52100 from zdover23/wip-doc-2023-06-17-rados-operations-crush-map-firefly-tunables

doc/rados: edit firefly tunables section

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/rados: edit firefly tunables section

Edit the firefly tunables section in
/doc/rados/operations/crush-map.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52092 from zdover23/wip-doc-2023-06-16-rados-operations-crush-map-4-of-x

doc/rados: edit crush-map.rst (4 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit crush-map.rst (4 of x)

Edit doc/rados/operations/crush-map.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50874 from rishabh-d-dave/mds-log-variable

mds: use variable g_ceph_context directly in MDSAuthCaps

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52095 from rhcs-dashboard/jsonschema-fix

mgr/dashboard: fix the jsonschema issue in install-deps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #51549 from vedanshbhartia/coverity_1517658

rgw: modify RGWRados::cls_bucket_list_ordered to store dirent key before move

Merge pull request #51719 from vedanshbhartia/coverity_1510555

rgw: Use a unique_ptr when creating a D3nCacheAioWriteRequest instance

Merge pull request #52076 from rhcs-dashboard/cephfs-no-mds-clients

mgr/dashboard: empty list of fs clients with no mds

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52089 from samarahu/d4n-cmake-fix

RGW: Place D4N files under cmake conditional

mds: use variable g_ceph_context directly in MDSAuthCaps

Variable g_ceph_context is global, therefore use it directly instead of
passing it as a parameter to method.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52010 from pkalever/fix-ImageMap

rbd-mirror: don't refer to (remote) mirror_uuid as peer_uuid

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51132 from lxbsz/wip-59349

qa: wait for 100 seconds to make sure the quota to be enforced

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

mgr/dashboard: fix the jsonschema issue in install-deps

Fixes: https://tracker.ceph.com/issues/61690
Signed-off-by: Nizamudeen A <nia@redhat.com>