os/bluestore: Fix code rot in ZonedAllocator.

Signed-off-by: Abutalib Aghayev <agayev@psu.edu>

Merge pull request #38010 from cbodley/wip-rgw-doc-reshard-syncstop

doc/rgw: update reshard design regarding 'bucket sync disable'

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #38585 from tchaikov/wip-doc-dev-errorator

doc/dev/crimson: highlight code blocks using C++ lexer

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38582 from tchaikov/wip-crimson-cleanup

crimson/osd: cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/dev/crimson: quote inline codeblock using "``"

backquote is used for quoting italic typed text.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev/crimson: highlight code blocks using C++ lexer

also remove redundant ":", normally "::" would suffice to start a
codeblock.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle error using handle_error()

no need to use safe_then() for this purpose.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not use __func__ in lambda

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: drop log messages only for debugging

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not capture dup variable

first can be deduced by variable already captured, so store it as a
local variable. it's less expensive than storing it as a captured
variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not use do_with() unless necessary

in this change, a seastar::do_with() is removed, as the captured
variables are passed to submit_push_data() by value. so no need to keep
them alive after the call.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: let trim_pushed_data() return extents

trim_pushed_data() is responsible for building up the extents to be used
for pushing. so let it return the extents.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use range-based for loop

also add comment explaining
ReplicatedRecoveryBackend::trim_pushed_data()

and move the definition of local variable closer to where it is used.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38539 from dillaman/wip-48526

librbd/object_map: don't assert if a snapshot doesn't exist

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38570 from rzarzynski/wip-crimson-recover_missing-noopt

crimson/osd: recover_missing() doesn't return std::optional anymore.

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38569 from tchaikov/wip-crimson-cleanups

crimson/osd: cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #38525 from varshar16/wip-ls-nfs-services

mgr/rook: Fix various 'orch ls' issues

Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #38557 from pcuzner/bootstrap-allocate-output-dir

cephadm: create default output dir during bootstrap

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #38577 from athanatos/sjust/wip-fix-crimson-nbd

crimson/tools: fix bugs introduced in 3333ff

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/tools: fix bugs introduced in 3333ff

You can't actually mix buffered and direct io on a stream.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #38572 from rzarzynski/wip-crimson-obc-not-blocker

crimson/osd: ObjectContext is not a Blocker anymore.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38563 from tchaikov/wip-admin-breathe

admin/doc-requirements.txt: require breathe >= 4.20

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38567 from rzarzynski/wip-doc-dev-errorator

doc/dev/crimson: exemplify errorator usage in error-handling.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

messages,osd,crimson: let MOSDPGPull::take_pulls() return pulls

so the caller can grab the pulls without creating a pull vector first.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: let build_push_op() return PushOp

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38111 from ronen-fr/wip-ronenf-scrub-v2

osd: Extracting scrub-related code from the PG & friends

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38566 from tchaikov/crimson-tools

crimson/tools: use helpers from byteorder.hh

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38565 from tchaikov/wip-crimson-cmake

cmake: add alien::cflags and move SubProcess.cc into crimson-alien-common

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

cephadm: update permissions on newly created dirs

Use 755 instead of 700 on new directories created during
bootstrap

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

crimson/osd: ObjectContext is not a Blocker anymore.

This is a follow-up to the c2dc437f78cd8be393c4d2eb8d79ce9c351ec4de
which removes unused OBC mebers after the with_lock() transition.

I'm not sure there is a reason to keep the `Blocker` bits as all
`ObjectContext::dump_detail()` does is just dumping `object_state_t`
while entering the OBC stage is already denoted by `PGPipeline::get_obc`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: recover_missing() doesn't return std::optional anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: add "alien::cflags" library

which exposes the definitions needed for building alienized bluestore
and the shared facilities compiled with -DWITH_ALIEN and -DWITH_SEASTAR

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move SubProcess.cc from crimson-common to crimson-alien-common

SubProcess is used by blkdev.cc only

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: let prep_push() return a map of PushOp

ReplicatedRecoveryBackend::prep_push() is responsible for building
PushOps, so it's more natural to let it return a map of PushOp.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: move unrelated continuation out of do_with()

if the variables held by do_with() is not used by the continuation, and
the continuation does not depend on the other continuations called in
the do_with(), let's move it out for smaller lexical scope. and hence
better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark const method "const"

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: pass vector<pg_shard_t> for recovery

std::list<std::map<pg_shard_t, pg_missing_t>::const_iterator> is
difficult to parse and understand. and what we need is no more than a
collection of pg_shard_t here. we do the look in
ReplicatedRecoveryBackend::prep_push() again anyway.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use range-based for loop

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not return unused value

do not return a bool from
`ReplicatedRecoveryBackend::maybe_pull_missing_obj()` anymore, as the
caller does not check it at all.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36146 from rhcs-dashboard/wip-46577-master

mgr: add types to MODULE_OPTIONS

Reviewed-by: p-se <pseidensal@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: sebastian-philipp <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38406 from votdev/issue_48438_drop_rgw_instance

mgr/dashboard: Drop invalid RGW client instances, improve logging

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/dev/crimson: exemplify errorator usage in error-handling.rst

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/dev/crimson: add error-handling.rst

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37642 from alimaredia/wip-rgw-apache-log-timestamp-change

rgw: add request time into beast log instead of current time

Merge pull request #38559 from xxhdx1985126/wip-crimson-bug-fix

crimson/osd: fix two bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/tools: use helpers from byteorder.hh

simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: recovery read lock should be held across the hole object push phase

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

mgr/dashboard: Drop invalid RGW client instances, improve logging

* Log RGW settings for easier problem tracking.
* Drop RGW client instances that do not exist anymore. This happens when a RGW service is removed via the orchestrator. The Dashboard tries to access the RGW client using the previous settings which leads to an error which might be confusing. Without dropping the error message 'RGW REST API failed request ...' is displayed instead of the correct one 'No RGW found'. Dropping the RGW client instance will produce correct error messages the next time the backend tries to establish a new connection.

Fixes: https://tracker.ceph.com/issues/48586
Signed-off-by: Volker Theile <vtheile@suse.com>

mgr: add types to MODULE_OPTIONS

Uses a typed Option class for MODULE_OPTIONS and replaces those in the
modules that are checked by mypy.

Fixes: https://tracker.ceph.com/issues/46577
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

admin/doc-requirements.txt: require breathe >= 4.20

to be compatible with Sphinx 3.2
see https://github.com/michaeljones/breathe/tree/v4.20.0#change-log

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37047 from sebastian-philipp/doc-cephadm-rm-host

doc/cephadm: Add remove host to operations

Reviewed-by: Zac Dover <zac.dover@gmail.com>

crimson/osd: allow obc to promote lock state from none to excl

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

cephadm: create default output dir during bootstrap

If the output dir doesn't exist the old behaviour was to
abort bootstrap. Since we're running with UID=0, this
patch will create the missing dirs for the user to keep the
bootstrap process going

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #38545 from tchaikov/wip-crimson-test-errorator

test/crimson: add a test to verify that errorator does not copy

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/crimson: add a test to verify that errorator does not copy

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: move member variable into test

so the test is more self-contained.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: s/do_until_test_t/errorator_test_t/

so we can reuse this test suite for more errorator related tests.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38554 from tchaikov/wip-admin-build-doc

admin/build-doc: consolidate the build deps into admin/doc-requirements.txt

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38547 from varshar16/wip-kubejacker-updates

script/kubejacker: Add python_common library and misc updates

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

admin/build-doc: drop comments for building dummy C bindings

we don't build dummy C bindings since
2235c2db0ecc7a8fbe165aba2ea8aaf8189d349c

Signed-off-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: revert "upgrade pip setuptools first"

per Radoslow Zarzynski, the build failure was due to missing Python.h
for the python version he was using on the system.

This reverts commit bc73204812907b1e55f0f4b8ad977bcce8834c6f.

Signed-off-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: stop passing --use-feature=2020-resolver to pip

to silence the warning of

WARNING: --use-feature=2020-resolver no longer has any effect, since it is now the default dependency resolver in pip. This will become an error in pip 21.0.

Signed-off-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: consolidate the build deps into admin/doc-pybind.txt

since all the python bindings are required for building the doc, extract
them into admin/doc-pybind.txt.

because the pybind python extensions require Cython python module to build,
we have to move them  into another requirement file, and install them after
Cython is installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37433 from aclamk/wip-rocksdb-better-options

kv/RocksDBStore: Added ability to set compound options

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #38020 from mdw-at-linuxbox/teuth-fix-x509-san

qa/tasks/openssl_keys.py: add subjectAltName to certificates

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38282 from vosdev/ceph-pool-alert

mgr/prometheus: Fix 'pool filling up' with >50% usage

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #38290 from athanatos/sjust/wip-transaction-manager-nbd

crimson/seastore: add journaling checksumming and nbd based simple perf tester

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38322 into master

* refs/pull/38322/head:
test_cephfs_shell: add tests for ls command

Reviewed-by: Rishabh Dave <ridave@redhat.com>

crimson/os/seastore/journal: fix typo in comment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager: allocate page aligned in read helper

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cached_extent: always allocate page aligned

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: remove now unused scan_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: rework jorunal::scan_extents to use scan_valid_records

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: replace scan_segment with scan_valid_records in replay_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add scan_valid_records

scan_valid_records validates record checksums and uses last_comitted
to avoid checking data crcs except on the final few records.  Will
replace scan_segment in a few patches.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #38444 from tchaikov/wip-msgr-string-view

msg/async: drop unused member variable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38439 from adk3798/iscsi-stuff

mgr/cephadm: improve iscsi dashboard config

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38533 from sebastian-philipp/cephadm-mypy-no-untyped

mgr/cephadm: disallow_untyped_defs=True

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38544 from tchaikov/wip-build-doc-auto-build

admin/build-doc: add livehtml target

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/seastore/journal: try_decode* pass bl as const &

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: validate header crc on read

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree/btree_range_pin: remove obsolete assert

In the event that the only two children of the root are merged, they
as well as the old root will be retired meaning that check_parent
will find no parent.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: maintain committed_to on each record

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use specified size types for header

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix debugging lines

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add data and metadata checksums

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node: collapse root on merge if down to one element

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: fix merge_entry return values

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: check segment nonce in scan_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: factor out read_segment_header

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: propogate segment headers with replay segments

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: record segment/record nonces based on seastore_id

Signed-off-by: Samuel Just <sjust@redhat.com>

include/uuid: make bytes() return const *

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager: add seastore_meta_t with uuid

Signed-off-by: Samuel Just <sjust@redhat.com>

include/uuid: convert to denc

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manger/ephemeral: add final annotations

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: use unqualified en/decode for denc types

Otherwise, if definitions from encoding.h are in scope the templated
denc one won't be considered.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix error returns in replay

SegmentManager is allowed to return an enoent if the segment isn't
readable.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../ephemeral: introduce pauses for testing

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: send aligned buffer in initialize_segment

Signed-off-by: Samuel Just <sjust@redhat.com>