crimson/os: add junction between OSD's ops and Seastore's OrderingHandle.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate AggregateBlocker-related ops to new tracking infra

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: bring AggregateBlockingEvent.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate PeeringEvent to new tracking infra, part 2

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: add BlockingEvent-aware variant of OSD::get_or_create_pg

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate InternalClientRequest to new tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate RecoverySubRequest to new tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate RepRequest to new tracking infra

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate PGAdvanceMap to new tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: reduce unnecessary indentation level in PGAdvanceMap::start()

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate PeeringEvent to new tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate BackgroundRecovery to new tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate BackfillRecoveryPipeline to new tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: CompoundPeeringRequest tracks its start and completion too

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: migrate CompoundPeeringRequest into new tracking infra

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: move SubOpBlocker from .cc to header

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: PGActivationBlocker does support new tracking infra

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: dissect PG::WaitForActiveBlocker into PGActivationBlocker

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: PGMap blocker does support new tracking infra

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: make PhaseOperationT able to deal with interruptors

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: pg_map.h doesn't pull pg.h anymore

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: convert entire ClientRequest to the new infra

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: slightly optimize OperationRegistryT::do_register()

There is no need to call the virtual op::get_type() twice.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: fix an assertion failure in Formatter when dumping ops

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: implement dump_ops_in_flight

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: track start and completion of ClientRequest

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: bring boilerplate for LTTng-based external tracking backend

At the moment it's mostly for verification & demonstration purposes.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: drop forward declaration of OSDMap osd.h

Having it doesn't make sense as we already include the header.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: add registry of external handlers of tracking events

This commit intentionally fails the build. It does that to verify
the static assertion on per-op-type lookup of event registry.

The concrete assertion is the one about `ClientRequest` as we
already moved two of its blocker into the new op tracking infra.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: switch one OSDMapGate of ClientRequest to new blocking infra

This time the demo of `BlockerT::BlockingEvent::TriggerI` which
is supposed to handle situations where:

1) `BlockerT` is separated from `TrackableOperationT` by many
intermediaries while
2) we don't want to templatize them all.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: switch a stage of ClientRequest to new blocking infra

This is a debut of the new blocking & tracking infrastructure
using the type-aware `BlockerT::BlockingEvent::Trigger<T>`.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: introduce PhasedOperationT to deduplicate PipelineHandles

This new class in the hierarchy will be handy when it will come to
introduce helpers for deailing with entering a pipeline stage.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: switch all OSD operations to TrackableOperationT

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: get rid of unnecessary private in ClientRequest

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: switch all ClientRequest-related pipeline stages to the new infra

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: pipeline stage classes can be bases for CRTP now

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: pipeline stage classes derive from BlockerT now

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: document OperationThrottler's behavior towards op's blocker list.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: simplify OperationThrottler::with_throttle()

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: move with_blocking_future from Operation to OperationT.

Buildability of this commit proves all users of `with_blocking_future()`
have the concrete operation type which is necessary on the the way
to the compile-time defined op's blocker registry.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/os/seastore: migrate OrderingHandle away from blocking_future

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: OperationThrottler derives from BlockerT instead of Blocker.

One another step closer. See the previous' commit description.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: make OperationThrottler aware about exact type of operation

This is a step towards making op's blocker registry static and
compile-time checked.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: make the OperationThrottler::with_throttle private

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd: add types PG and OSD-hosted OSDMapGate instances.

`ClientRequest` is going to hold all tracking events as a `std::tuple`
instance and access them through the type-related variant of `std::get`.
As two instances of `OSDMapGate` are used there, a need to differentiate
at the type level arises.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/common: add basic infrastructure for op tracking

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/common: abstract from staticness of BlockerT::type_name.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #37655 from rishabh-d-dave/vr-run-multiple-cmds

vstart_runner: accepts multiple commands as str too

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45473 from neesingh-rh/fix_50470

cephfs-top: Multiple filesystem support

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45776 from neesingh-rh/fix_48863

cephfs-shell: chmod should be allowed for changing all mode bits.

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45974 from rishabh-d-dave/shell-unrecognized-cmds

cephfs-shell: exit with non-zero on getting unrecognized cmd

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46135 from athanatos/sjust/wip-vstart-asok

vstart.sh: default CEPH_ASOK_DIR to out/

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #46137 from athanatos/sjust/wip-crimson-typos

crimson: misc typos

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #46134 from athanatos/sjust/wip-55326

crimson/common/format: remove ceph::coarse_real_clock format speciali…

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson/common/format: remove ceph::coarse_real_clock format specialization

...
..., value=..., loc=...) at /usr/include/fmt/ostream.h:114
...

Introduced: e9784125241064709d25e9c64d16e1c72b4d7bed
Fixes: https://tracker.ceph.com/issues/55326
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #46128 from vumrao/wip-vumrao-quincy-release-note-update

docs/releases: update quincy release notes tracker 55383 known issue

docs/releases: update quincy release notes tracker 55383 known issue

Signed-off-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge pull request #46079 from ronen-fr/wip-rf-log-mock

common: make LogChannel mockable

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #46115 from dang/wip-dang-split-init

RGW - Split RGWRados initialization

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Soumya Koduri <skaduri@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #46066 from bluikko/patch-4

doc/cephadm/services: Add missing ceph command to orch apply commands

Merge pull request #46109 from bluikko/pgcalc_links

doc/rados: fix pgcalc links

Reviewed-by: Zac Dover <zac.dover@gmail.com>

vstart.sh: default CEPH_ASOK_DIR to out/

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: fix typo

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/operation: fix comment formatting

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/admin/osd_admin: fix comment spacing

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #44494 from mdw-at-linuxbox/wip-master-mankey

rgw/crypt: sse-s3 changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #45843 from ajarr/wip-mgr-nfs-output-message

mgr/nfs: clarify in the output message

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #46095 from aclamk/wip-aclamk-unbounded-wholespace-iterator

kv/RocksDBStore: Remove ability to bound WholeSpaceIterator

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Cory Snyder <csnyder@iland.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

RGW - Split RGWRados initialization

The inter-connectedness of RadosStore and RGWRados resulted in a
segfault during RGWRados::init_complete due to the rados pointer not
being set in RadosStore yet.

Split the calls to RGWRados::initialize and RGWRados::init_complete, so
that we can set up RadosStore between them, allowing the services
created in RGWRados::init_complete to access the RadosStore.

Fixes: https://tracker.ceph.com/issues/55512
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

qa/vstart_runner: add a helper method to run commands

It's more convenient to not to pass stdout, stdin, and stderr every
single time.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner: don't use os.kill for passing SIGKILL

vstart_runner.py is written assuming that it can run commands with
superuser privileges whenever possible and vstart_runner.py is meant to
be executed without sudo.

So, it's better kill a process using "sudo kill -9 <PID>", instead of
using os.kill() because os.kill() can't kill process launched with
superuser privileges.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner.py: add a note about omitting sudo

About the commit date: this commit got dropped from the patch series
during some PR branch update but is added back now.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner: get rid of "exec sudo" from "ceph -w" cmd args

Passing "exec sudo" to "ceph -w" caused "Ceph API test" CI job to fail.
Error was not related to this tracker issue but the code added for it
is reversed now in this commit. The tracker issue -
https://tracker.ceph.com/issues/49644.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner: keep cmd args as passed by user as well

We convert all cmd args to str and pass bash functions along to override
certain arguments in those command arguments. Let's save cmd args
without those bash functions since they can be useful later (for
example, printing cmd args in logs, which is the case in this patch.)

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner: copy few more note points from dev guide

The intention behind copying these note points is to document the
behaviour of vstart_runner.py inside vstart_runner.py as well so that
developer don't miss it out while working on it.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner: override adjust-ulimits, ceph-coverage & sudo

Overridding commands is much better than deleting these commands from
command argument string using Python since, unlike deleting, overridding
doesn't require parsing. A note has been added for this to
vstart_runner.py's module docstring and to Ceph Developer's Guide
document.

Since functions don't work with sh shell, to make overriding work
vstart_runner.py will use bash shell here onwards.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner: accept multiple commands as str

Convert all command arguments to str from list, update checks and
adjustments performed on command arguments accordingly and update
documentation to include warnings about some critical parts of
vstart_runner.py and update tasks.cephfs.mount.MountCephFS.run_shell().

Fixes: https://tracker.ceph.com/issues/47849
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #45842 from yuvalif/wip-yuval-const-notifications

rgw/notifications: make the interfaces const when possible

Merge pull request #46100 from ronen-fr/wip-rf-80cls-v1

osd/scrub: reformat scrub files to 80 cols

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>

Merge pull request #45889 from lxbsz/wip-mds-clienup

mds: remove duplicated code

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45983 from lxbsz/client_warning

client: fix compile warning

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/radosgw: fix pgcalc link

The pgcalc tool has moved to the "old" ceph site so update
the link to avoid a 404.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc/rados/operations: fix pgcalc link

The pgcalc tool has moved to the "old" ceph site so update
the link to avoid a 404.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #46051 from Matan-B/wip-matanb-ipv6-mount

mount/conf: Fix IPv6 parsing

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #46102 from bluikko/ioctx_get_stats_docs

doc/rados/api: fix invalid RST syntax and don't sudo for vim

Merge pull request #44305 from kotreshhr/vstart-fixes

qa: Fix a few tracebacks in vstart_runner

Reviewed-by: Venky Shankar <vshankar@redhat.com>

pybind/rados: fix typo in get_stats documentation

Ioctx.get_stats() had a typo in return value documentation
block so add the "c" in "objects".

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc/rados/api: fix invalid RST syntax and don't sudo for vim

Fix invalid syntax where "linenos:" was printed in the
final rendered documentation instead of being used as
formatting syntax.

There is no need to use sudo for editing a source file
so run vim without sudo.

Change the whole block to use bash prompt since it is
a command.

Also modify the preceding text to hopefully not bold it.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #45794 from Matan-B/wip-matanb-c-submit_txn

crimson/osd: submit_transaction() refactoring

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: submit_transaction() refactoring

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #45810 from zenomri/wip-rgw-lua-bucketname

rgw: support bucket name in lua pre request execution

mount/conf: Fix IPv6 parsing

Fixes: https://tracker.ceph.com/issues/47300
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

rgw: support bucket name in pre request context

because bucket object is created only after authentication,
if bucket object is null upon accessing Request.Bucket.Name, we return req_state->init_state.url_bucket

Signed-off-by: Omri Zeneva <ozeneva@redhat.com>

doc/cephadm/services: Add missing ceph command to orch apply

In cephadm service management documentation several of the
ceph orch commands are missing the ceph part, mostly in
ceph orch apply commands but not all of them.

Add ceph in the front of the command to make them consistent
with all other commands.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #46030 from ivancich/wip-fix-bucket-index-new

rgw: remove entries from bucket index shards directly in limited cases

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45990 from cbodley/wip-55339

rgw: RGWPutLC does not require Content-MD5

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #46098 from anthonyeleven/anthonyeleven/submitting_patches_improvement

SubmittingPatches: Improve SubmittingPatches example

Merge pull request #45873 from BenoitKnecht/rgw-opa-segv

rgw: Avoid segfault when OPA authz is enabled

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #45882 from ivancich/wip-index-completion-mgr-crash

rgw: address crash and race in RGWIndexCompletionManager

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

common: make LogChannel mockable

Adding LoggerSinkSet, an abstract API that enables unit-testing
of OSD components that emit cluster-log messages, and trapping the
logs.

First usage example is in the Scrubber backend unit-tests.
The log "implementation" there (note the error counter that
can be used to determine tests success):

 class MockLog : public LoggerSinkSet {
 public:
  void warn(std::stringstream& s) override
  {
    std::cout << "\n<<warn>> " << s.str() << std::endl;
  }
  void error(std::stringstream& s) override
  {
    err_count++;
    std::cout << "\n<<error>> " << s.str() << std::endl;
  }
  ...

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>