test/crimson: cleanup messenger test addresses

Consolidate the name/port/nonce of the addresses used in the messenger
unit tests.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: fix fmt related compile issue

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: drop and replace __func__

__func__ is meaningless in the lambda function with continuations.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: fixes to print exceptions and errors

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: introduce IOHandler class for message and event dispatching

IOHandler also represents the Connection as ConnectionHandler.

ProtocolV2 and IOHandler will be finally running in different cores, as
ProtocolV2 will need to call IOHandler interfaces asynchronously. And
IOHandler will also notify ProtocolV2 through HandshakeListener
asynchronously.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: cleanups to Protocol and ProtocolV2 interfaces

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: cleanup, rename out_state to io_state

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: drop Protocol::print_conn()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: move mark_down() from ProtocolV2 to Protocol

Process mark_down in Protocol rather than in ProtocolV2 to prevent
further event dispatching after mark_down is called by user. Then notify
ProtocolV2 as the IO/socket core and handshake core can be different and
the notification can be asynchronous.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: move is_connected() from ProtocolV2 to Protocol

Infer is_connected by connection events instead of the protocol V2
internals. That can save a cross-core operation for connection users
from the connection IO core to the handshake core.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: hide dispatchers from ProtocolV2

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: unshare conn for ProtocolV2 and Protocol

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: make it explict about the FrameAssemberV2 ownership

FrameAssemblerV2 is owned by ProtocolV2 during handshake, and owned by
Protocol during messaging.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: change out_state_changed to promise<> as it is not shared

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: move message read path from ProtocolV2 to Protocol

Also move socket shutdown ownership to Protocol at READY/open.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: move message write path from ProtocolV2 to Protocol

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: move dispatch ms_handle_connect() before READY

It would be too late to notify connected after the status becomes READY
and open.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: move intercept_frame() from ProtocolV2 down to FrameAssemblerV2

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: refactor socket managements

Previously, the socket state is transparent to the protocol and
implicitly handled. Move the responsibilities into protocol for finer
controls to further decouple the IO and handshake.

Also, refactor the fault handling and make the in/out message
dispatching more symmetric.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: fix execution_done wait mechanism

* prepare the execution_done before the function is invoked, not after.
* execution_done future doesn't need to be shared.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: improve logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson/test_messenger: change the logger subsys to test

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: move ms_handle_accept into the replacing gate

To prevent others to close the connection before the gate, and causing
abort without proper closing the moving socket.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: add aborts when the state is inconsistent

To prevent unexpected event dispatching and state transitions.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: move remote reset after waiting

And only dispatch the remote reset when the connection is still valid.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: introduce and integrate FrameAssemblerV2

FrameAssemblerV2 encapsulates the low-level frame processing for
protocol v2, and manages the socket instance.

FrameAssemblerV2 is supposed to be running on the socket core for
performance reasons, which will be changable.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: split gate for handshake and IO

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: move close logic from Protocol to ProtocolV2

Protocol class will be removed.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: rename in/out related members and methods

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: move IO members into Protocol class

In order to introduce the cross-core IOHandler class.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: proper group SocketConnection interfaces

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: move socket from Protocol to SocketConnection

Protocol class will be removed.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #50022 from zdover23/wip-doc-2023-02-08-multisite-maintenance

doc/radosgw: refine "Maintenance" in multisite.rst

Merge pull request #50011 from zmc/deps-broken-repos

install-deps.sh: Fail when downloading apt repos

doc/radosgw: refine "Maintenance" in multisite.rst

Refine the text in the "Maintenance" section of
doc/radosgw/multisite.rst.

https://tracker.ceph.com/issues/58632

Co-authored-by: Cole Mitchell <cole.mitchell@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #49723 from anthonyeleven/anthonyeleven/58123

ceph: Clarify usage message for admin-socket commands

install-deps.sh: Fail when downloading apt repos

More specifically, if we get e.g. a 504 while attempting to download a
.list file, fail the build and log the status code.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #49994 from zdover23/wip-doc-2023-02-05-rgw-multisite-refine-2

doc/rgw: refine ~50-~140 of multisite.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw: refine ~50-~140 of multisite.rst

Remove pleonasm from approximately line 50 to line 140 of
doc/rgw/multisite.rst.

Follows https://github.com/ceph/ceph/pull/49993.

https://tracker.ceph.com/issues/58632

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49993 from zdover23/wip-doc-2023-02-04-rgw-multisite-top-matter-refine

doc/rgw: refine 1-50 of multisite.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw: refine 1-50 of multisite.rst

Refine the English in the first fifty lines of multisite.rst. This is
part of a project whose goal is the creation of a document that will
impart a good aerial view of RGW to a first-time reader in the minimum
number of words.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #49981 from zdover23/wip-doc-2023-02-03-rgw-multisite-zones-refinement

doc/rgw: refine "Zones" in multisite.rst

doc/rgw: refine "Zones" in multisite.rst

Clean up the English under the section called "Zones". Part of a larger
project aimed at giving the reader more understanding earlier in the
documentation of how buckets and zones and zone groups and endpoints
work.

https://tracker.ceph.com/issues/58632

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49968 from zdover23/wip-doc-2023-02-02-jaegertracing-refinement

doc/jaegertracing: refine index.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49848 from ljflores/wip-tracker-58545

qa/workunits: give telemetry opt-in check more time

doc/jaegertracing: refine index.rst

Make English in jaegertracing/index.rst more elegant.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49935 from athanatos/sjust/wip-58486

crimson: fix incorrect interval check in PG::submit_transaction

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #49812 from petrutlucian94/poll_hup

msg: fix poll event driver, handling POLLHUP and POLLERR

ceph: Clarify usage message for admin-socket commands

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #48295 from NitzanMordhai/wip-nitzan-osd-start-osdmap-hang-adding-debug-msg

OSD: during test start, not all osds started due to consum map hang

Merge pull request #45247 from andriytk/update-ceph-mgw

rgw: update motr sal

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge PR #49842 into main

* refs/pull/49842/head:
Revert "cmake: do not link ceph-dencoder against global"

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #49946 from cityofships/pkg-docs

doc/cephadm: grammar / syntax in install.rst

Signed-off-by: Piotr Parczewski <piotr@stackhpc.com>

crimson/osd/pg: remove incorrect map epoch check in PG::submit_transasction

It's entirely fine for the map_epoch to change while the op is processed
as long as none of the intervening epochs caused an interval change.  In
general, the correct way to check for an interval change is with
has_reset_since.

We don't need to do this check here at all because IOInterruptCondition
will already have performed it against the captured epoch when the
continuation resumed.

Introduced: 31418020fd58
Fixes: https://tracker.ceph.com/issues/58486
Signed-off-by: Samuel Just <sjust@redhat.com>

Revert "cmake: do not link ceph-dencoder against global"

Reverting since this is causing the following ceph-dencoder exception:

        [src/tcmalloc.cc:332] Attempt to free invalid pointer 0x558f14e3a000

NOTE: I didn't debug this any more further since there is sizable backlog
of PRs that need testing/merging, but reverting the change makes it work.

Fixes: http://tracker.ceph.com/issues/58219
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49913 from neesingh-rh/wip-51459

doc: add the damage types that scrub can repair

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge PR #49048 into main

* refs/pull/49048/head:
mds: fix and skip submitting invalid osd request

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

doc: add the damage types that scrub can repair

Fixes: https://tracker.ceph.com/issues/51459
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Rollback submodules changes made by accident

Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>

Merge pull request #48541 from lxbsz/wip-qa-upgrade-fix

qa/cephfs: fix confusing config when upgrading mds nodes

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #48049 from dparmar18/wip-dparmar-qa-mixed-clients-57280

qa/fs/mixed-clients: specify distros for tests

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #48251 from dparmar18/wip-dparmar-MDS-57359

mds/Server: do not allow -ve reclaim flags to cause client eviction

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #47753 from mchangir/mon-disable-snap-id-allocation-for-fsmap-pools

mon: disable snap id allocation for fsmap pools

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #47114 from lxbsz/wip-56532

ceph_fuse: retry the test_dentry_handling if fails

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

client: switch to old method to invalidate dcache if euid != 0

Force to use the old and slow method to invalidate the dcache
if the euid is non-root, or the remount may fail with return
code 1 or 32.

https://tracker.ceph.com/issues/56532
Signed-off-by: Xiubo Li <xiubli@redhat.com>

ceph_fuse: make it to force invalidating dentries when kernel >=3.18

The remount will fail randomly for unknown reasons. And in certain
circumstance we can reprodce this very easy, which will block our
testing. Make it posible to force to old method to invalidate the
dcache when the "client_try_dentry_invalidate" option is enabled
even kernel version >= 3.18.0

Fixes: https://tracker.ceph.com/issues/56532
Signed-off-by: Xiubo Li <xiubli@redhat.com>

ceph_fuse: retry the test_dentry_handling if fails

For some unknown reason the remount will fail, we need to retry it
for the test_dentry_handling.

Fixes: https://tracker.ceph.com/issues/56532
Signed-off-by: Xiubo Li <xiubli@redhat.com>

client: move a client's option to mds-client.yaml

mds_max_retries_on_remount_failure option is used by Client.cc only.

Fixes: https://tracker.ceph.com/issues/56532
Signed-off-by: Xiubo Li <xiubli@redhat.com>

crimson/osd/osd_operations/client_request: clarify get_obc stage message

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #49906 from evershalik/main

Update README.md

Merge pull request #49905 from zdover23/wip-doc-2023-01-29-rados-config-osd-config-ref-osd-op-thread-suicide-timeout

doc/rados: document osd_op_thread_suicide_timeout

Merge pull request #49908 from Matan-B/wip-matanb-upgrade-doc

doc/dev/developer_guide: Add Upgrade Testing Documentation

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/dev/developer_guide/testing_integration_tests: Add Upgrade Testing Documentation

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

doc/rados: document osd_op_thread_suicide_timeout

Add default value for osd_op_thread_suicide_timeout configuration value
to the OSD Configuraton Reference.

https://tracker.ceph.com/issues/58590

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Update README.md

Signed-off-by: SHANKAR <98207888+evershalik@users.noreply.github.com>

Merge branch 'master' into update-ceph-mgw

crimson/osd/object_context_loader: add logging for notify_on_change

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/object_context_loader: convert to log macros

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/object_context_loader: take backend by reference

It's required and we don't check for null.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_backend: remove stopping and peering

These two state variables duplicate checks that *should* already
be handled by the IOInterruptCondition.  None of the stopping
checks should ever trigger because the caller would be in an
interruptible future context which already performed that check.

Moreover, peering doesn't really work -- it relies on the callback
firing prior the call to on_activate_complete(), and there isn't any
guarantee that will happen.  Storing the epoch from when the
callback was created as we do in IOInterruptCondition
would be required.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_interval_interrupt_condition: use new logging macros

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/replicated_backend: use new logging macros

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common: move log macros to log.h from os/seastore/logging.h

These have been handy in imposing uniformity on seastore logging,
let's start using them in PG/OSD code.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #49891 from bluikko/bluikko-doc-host-management-prompts

doc/cephadm: Fix prompts in host-management.rst

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49897 from bluikko/bluikko-doc-rados-operations-prompt

doc/rados/operations: Fix double prompt

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/cephadm: Fix example commands

Separate command output block from the command to avoid printing prompt on it.
Also remove double prompt.

Fix indentation of two example commands after prompt.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc/rados/operations: Fix double prompt

In monitoring.rst a double prompt was rendered, one non-selectable and one selectable. Remove the selectable prompt.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #49894 from bluikko/bluikko-doc-rados-operations-control-indentation

doc/rados/operations: Fix indentation

Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc/rados/operations: Fix indentation

Fix invalid indentation that caused indentation to be rendered wrong in control.rst.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #49890 from bluikko/bluikko-doc-ceph-krb-auth

doc/dev: use underscores in config vars

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/dev: use underscores in config vars

Use underscores instead of spaces in config vars in ceph_krb_auth.rst.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #49109 from NitzanMordhai/wip-nitzan-fixing-few-rados/test.sh

Wip nitzan fixing few rados/test.sh

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #49887 from cbodley/wip-qa-rgw-run-s3tests-tox

qa/workunits/rgw: run-s3tests.sh uses tox

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #49799 from phlogistonjohn/jjm-cephadm-utils-tests

cephadm/tests: add test coverage for copy_tree, copy_files, move_files, recursive_chown

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #48597 from ronen-fr/wip-rf-osd-bitfields

osd: change two ObjectContext flags from bitfields to full variables

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/workunits/rgw: run-s3tests.sh uses tox

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49624 from ronen-fr/wip-rf-unique-formatter

common: extend Formatter API with a unique_ptr-returning static function

Merge pull request #49537 from rzarzynski/wip-bug-57546

Revert "osd/PeeringState: proc_lease_ack break once found from OSD"

Merge pull request #48854 from ifed01/wip-ifed-small-chunk-bluefs

os/bluestore: enable 4K allocation unit for BlueFS