crimson/osd: forward declare OSDOp and ObjectState

simpler this way, also, silence clang warning, as OSDOp is defined as
struct, let's declare it as struct as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: construct future state in-place

instead of constructing the value of state, pass the parameters for
building it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40646 from rzarzynski/wip-crimson-watch-timeout

crimson/osd: implement timeout support for watches

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40608 from tschoonj/etag-spelling

rgw: fix spelling of eTag in S3 message structure

Merge pull request #41245 from tchaikov/wip-doc-confval-ext

doc/_ext: fixes related to mgr option rendering

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41309 from yuriw/wip-yuriw-crontab-master

qa/tests: added pacific-p2p

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41303 from tchaikov/wip-crimson-os-cleanups

crimson/os/seastore: do not capture unused variables

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41261 from cbodley/wip-rgw-beast-sync

rgw: add config option to disable beast's async process_request()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

crimson/osd: unify the interruption handling between {Internal,}ClientRequest.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: share do_recover_missing() between {Internal,}ClientRequest.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tests: added pacific-p2p

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #41094 from dvanders/negative_progress

mgr/progress: ensure progress stays between [0,1]

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41092 from ifed01/wip-ifed-fix-alloc-init-add-free-0-len

os/bluestore: tolerate zero length for allocators' init_[add/rm]_free()

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40864 from wjwithagen/wjw-fix-KernelDevice-locking

include, blk: fallback to basic locking if a OS does not have F_OFD_SETLK

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/common: use parameter pack for building future<>

it mirrors the way how seastar::make_ready_future() works. and more
importantly, it allows us to build interruptible_future in the same
way as we build plain seastar::future<> in-place. so we can, for
instance, create a future<tuepl<int,int>> using:

make_ready_future<tuple<int,int>>(1, 2)

instead of using

make_ready_future<tuple<int,int>>(make_tuple(1, 2))

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not capture unused variables

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: ClientRequest::do_recover_missing doesn't depend on OSD anymore.

This commit enables the unification of missing objects between
`ClientRequest` and `InternalClientRequest`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: sending EVENT_DISCONNECT becomes implementation detail of Watch.

In contrast to ceph-osd crimson sends CEPH_WATCH_EVENT_DISCONNECT directly
from the timeout handler and after CEPH_WATCH_EVENT_NOTIFY_COMPLETE.
This simplifies the Watch::remove() interface as callers aren't obliged
anymore to decide whether EVENT_DISCONNECT needs to be send or not -- it
becomes an implementation detail of Watch.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: wire up handling of watch timeouts.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: s/do_timeout/do_notify_timeout/ per the upcoming do_watch_timeout().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: introduce the InternalClientRequest infrastructure.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: PG::with_locked_obc() doesn't depend on MOSDOp anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41161 from rhcs-dashboard/ingress-creation-follow-up

mgr/dashboard: ingress service creation follow-up

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41174 from pritha-srivastava/wip-rgw-sts-cert-url-fix

rgw/sts: code to fetch certs using .well-known/openid-configuration url

Merge pull request #41221 from rhcs-dashboard/50686-fix-osd-host-grafana

mgr/dashboard: fix OSDs Host details/overview grafana graphs

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: ingress service creation follow-up

Fixes: https://tracker.ceph.com/issues/50568
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Pre-populating the service id(read-only) with the value same as backend service.

Merge pull request #41103 from navinbarnwal/RenameFilesystems

mgr/dashboard: fix typo: Filesystems to File Systems

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41023 from rhcs-dashboard/bucket-name-async-validator-performance

 mgr/dashboard: RGW buckets async validator performance enhancement and name constraints

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #41196 from rhcs-dashboard/add-services-e2e

mgr/dashboard: add Services e2e tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41292 from tchaikov/wip-crimson-alienstore-cleanup

crimson/os/alienstore: cleanups

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #41291 from athanatos/sjust/wip-store-nbd-fs

crimson: add seastore and bluestore to crimson-store-nbd

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: ensure RequestWriter outlives any requests on error

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore.cc: swap stop and umount

start/stop seem to be intended for things like thread pools, etc that
seastore doesn't have.  Umount maps more cleanly here.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd: add FuturizedStore driver

Allows usage of bluestore and seastore.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/alienstore: create tuple in-place

no need to use make_tuple<> when constructing a future whose value is
available. as future<> can be constructed by perfect forwarding the
parameters to its state constructor.

also, wrap the lines whose length is over 80 chars.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/alienstore: add curly brackets

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/alienstore: use get_conf() helper

do not use cct->_conf.get_val<>, unless we are in alien threads.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: add debugging on shutdown for cache, pin contents

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store-nbd: break into multiple files

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #41271 from cyx1231st/wip-seastore-onode-tree-logger

crimson/onode-staged-tree: switch to seastore logger macros

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41256 from tchaikov/wip-doc-rbd-confval

doc/rbd/rbd-config-ref: use confval directive to define options

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: fix typo: Filesystems to File Systems

Fixes: https://tracker.ceph.com/issues/50341
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>

mgr/dashboard: fix typo: Filesystems to File Systems

Fixes: https://tracker.ceph.com/issues/50341
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>

Merge pull request #41226 from yuriw/wip-yuriw-crontab-master

qa/tests: removed most of runs for nautilus as it's almost EOL

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Detailed error messages in rgw bucket name validation

Explain the rgw bucket name constrains for each bucket name validation
errors.

Fixes: https://tracker.ceph.com/issues/50516
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix typo: Filesystems to File Systems

Fixes: https://tracker.ceph.com/issues/50341
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>

mgr/dashboard: add Services e2e tests

Fixes: https://tracker.ceph.com/issues/50567
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Introducing e2e tests for service creation for Ingress and RGW service types.

qa/tests: removed most of runs for nautilus as it's almost EOL

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #41272 from Aran85/fix-assert

crimson/os: cleanup wrongly assertion

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41274 from tchaikov/wip-crimson-os-cleanups

crimson/os/seastore: do not capture unused variables

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: RGW buckets async validator performance enhancement

The rgw bucket creation form has the Name field which have an async
validator. The validator calls all the bucket name and check if the
entered name is unique or not. This happens on every keystroke. So if
100 or more buckets are there, then the async validation can be real
    slow and causes misvalidations in different fields.

I changed the validation logic and did some cleanups to improve the
performance of the async validation.

Fixes: https://tracker.ceph.com/issues/50514
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41193 from sebastian-philipp/cephadm-cephfs-mirror-key-profile-cephfs-mirror

mgr/cephadm: Use `cephfs-mirror` caps profile

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #41181 from adk3798/iscsi-timeout

mgr/cephadm: add timeout when removing iscsi gateway.conf

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #41175 from mgfritch/cephadm-autopep8

cephadm: introduce autopep8

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #41250 from VasishtaShastry/tune_err_msg

mgr: Fix orch osd rm stop help message

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41240 from tchaikov/wip-debian-install-cleanup

debian/ceph-common.postinst: fix indent and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #41276 from tchaikov/wip-docs-responsive-button

doc/_theme: show the menu button

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #41265 from zdover23/wip-doc-config-ssl-tls-for-grafana-2021-05-11

doc/cephadm: rewrite "config ssl/tls f. grafana"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41194 from idryomov/wip-rbd-pwl-status

rbd: don't attempt to interpret image cache state json

Reviewed-by: Yin Congmin <congmin.yin@intel.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

mgr/dashboard: fix typo: Filesystems to File Systems
Added breadcrumbs and edited Filesystem -> File Systems in dashboard
Navigation.

Fixes: https://tracker.ceph.com/issues/50341
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>

doc/_theme: show the menu button

because we have a top-nav bar, which is setting on top of the bar
containing the menu button when the docs is displayed wit a device with
smaller width. in this change, the container of the menu button is moved
down a little bit, so it is visible again.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: do not capture unused variables

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41100 from navinbarnwal/MinorIssueFeature

mgr/dashboard: fix rename inventory to disks

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

crimson/os: cleanup wrongly assertion

Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

doc/mgr/prometheus: use confval directive to define options

less repeating this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: print "default" using "literal" filter

there is chance that the default value of an option is empty, before
this change the rendered result would be "````", and sphinx is confused
at seeing this:

Unexpected section title or transition.

as it takes it as a notion of section title. after this change,
the "literal" filter is always applied to the default value, so an empty
string is rendered like "<empty string>" instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: define a helper for current_module()

less repeating this way, this change also addresses the missing option
issue if the option to be rendered is the first option in a document,
and there is no "mgr_module" directive before it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: switch to use seastore logger macros

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore: remove duplicated log prefix

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: change to test logger in TreeBuilder

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/_ext: do not require "mgr_module" at end of document

before this change,

.. mgr_module:: None

is required at end of a mgr module document to reset the context, so
the builder env is not polluted when processing the next document.

after this change, this ending directive is not needed anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: do not load all modules for a certain module

instead of collecting all options from all modules, just load the
options from the specified module, this addresses the issue where
options with the same name are overriden by the options defined by
another mgr module.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: add "module" option to confval

in addition to "mgr_module" directive, add the "module" option to
confval. this allows a certain option to specify its own module without
being nested in a "mgr_module" directive pair.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: use str for mgr module options by default

sometimes, they don't specify the option type and just default to "str".

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/prometheus: assign default values for server_[addr,port]

and add desc to them, so these metadata can be extracted by the sphinx extension.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: compose name using cur_module and option name

so a mgr option can be referenced using
`mgr/<module_name>/<option_name>`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41269 from tchaikov/wip-mgr-max_misplaced

doc/rados/operations: s/max_misplaced/target_max_misplaced_ratio/

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41075 from cyx1231st/wip-seastore-onode-mgr

crimson/seastore: integrate and test onode-erase/list features

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/rados/operations: s/max_misplaced/target_max_misplaced_ratio/

max_misplaced with replaced by in target_max_misplaced_ratio
edbd592ee44e02a5328e1510879555c2f9dcfc9e, but the document was not
sync'ed. let's update it accordingly.

Fixes: https://tracker.ceph.com/issues/50745
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41207 from paulreece42/wip-doc-balancer-options

doc: added doc on more throttling options for the PG balancer module

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rbd/rbd-config-ref: use confval directive to define options

less repeating this way.

also fix a typo of "rbd_qos_writ_bps_limit", it should be
"rbd_qos_write_bps_limit".

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: rewrite "config ssl/tls f. grafana"

This PR streamlines the grammar in the subsection
called "Configuring SSL/TLS for Grafana" in the
monitoring.rst file. It also corrects the prompt
rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/osd: PG::get_oid() doesn't depend on MOSDOp anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: introduce OpInfo filling from a vector of OSDOps.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: expose the non-MOSDOp-taking variant of do_osd_ops() externally.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: PG::do_osd_ops_execute() doesn't depend on MOSDOp anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: pass std::vector<OSDOp>& to PG::submit_transaction().

This will allow in a moment to get rid of the dependency on
`MOSDOp` on all paths of `PG::do_osd_ops_execute()`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: PG::do_osd_ops_execute() doesn't directly takes ObjectContextRef.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: PG::repair_object() doesn't depend on MOSDOp anymore.

Before this commit the method was depending on `MOSDOp::get_min_epoch()`
to start an `UrgentRecovery`. However, it seems `PG::get_osdmap_epoch()`
would be sufficient here as the very early stages of the processing
in `ClientRequest` ensure the PG fits the `get_min_epoch()` requirement.

In the classical OSD the counterpart code looks like below:

```
int PrimaryLogPG::rep_repair_primary_object(const hobject_t& soid, OpContext *ctx)
{
  // ...
  queue_peering_event(
      PGPeeringEventRef(
        std::make_shared<PGPeeringEvent>(
        get_osdmap_epoch(),
        get_osdmap_epoch(),
        PeeringState::DoRecovery())));

  return -EAGAIN;
}
```

In addition to the dependency minimalisation, the commits reformats
the code around `PG::repair_object()` to fit our guidelines.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: reload obc also when handling ct_error::object_corrupted.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: introduce RollbackOrchestrator to OpsExecuter.

If the execution of an `OSDOp` fails, we're left with potentially
altered `ObjectContext`. We deal with that by reloading `obc` if
there was any modification to it. To figure this out, `has_seen_write()`
on `OpsExecuter` is being called. Unfortunately, the current impl.
has following drawbacks:

 * `has_seen_write()` can be called after `std::move(ox).flush_...()`
    which is very inelegant;
 * it requires catching both `ObjectContext` and `OpsExecuter` while
   the latter already references the former;
 * there is no explicitly given reason in the header for justifying
   the presence of `has_seen_writes()`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: split PG::do_osd_ops() to facilitate InternalClientRequest.

This commit brings `PG::do_osd_ops_execute()` a subset of
`PG::do_osd_ops()`; it handles the ops execution through
`OpsExecuter` and the `submit_transaction()` but it stays
indepedent from `MOSDOp` and `MOSDOpReply`. This trait
facilitates the `InternalClientRequest`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: erase the message type in OpsExecuter.

THe reason is unification of infrastructure between external
client requests (everything represented by the `ClientRequest`)
and internal requests.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: drop namespace for arg in PG::with_locked_obc().

It's unnecessary.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: split ClientRequest::PGPipeline into CommonPGPipeline.

This is another step towards the `InternalClientRequst`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: the ClientRequest::do_recover_missing() takes oid externally.

This refactor is a first step towards sharing the recovery bits
with `InternalClientRequest`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement ObjectContext relocking.

This commit introduces a `ObjectContext`-taking variant of
`PG::with_locked_obc()`. The upcoming internal counterpart
for the `ClientRequest` is the intended audience.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: ObjectContext allows the hobject_t to be std::moved in ctor.

Taken with "crimson/osd: use obc->get_oid() instead of passing
hobject_t around" and enriched with the move-constructing down
the `ObjectState` path this should allows to save some work in
e.g. `std::string` instances that are part of the `hobject_t`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: OpsExecuter retrieves PG when doing op effects.

This will necessary to spawn the upcoming `InternalClientRequest`
from the `Watch`'s timeout handler.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw: add config option to disable beast's async process_request()

when disabled, a null_yield is given to process_request() so that it
runs synchronously. mostly intended for debugging

Signed-off-by: Casey Bodley <cbodley@redhat.com>