crimson/net: assert connection gate is not unexpectedly closed

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: fix test error report

Report unexpected exceptions before destructing the tests.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38572 from rzarzynski/wip-crimson-obc-not-blocker

crimson/osd: ObjectContext is not a Blocker anymore.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38563 from tchaikov/wip-admin-breathe

admin/doc-requirements.txt: require breathe >= 4.20

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38567 from rzarzynski/wip-doc-dev-errorator

doc/dev/crimson: exemplify errorator usage in error-handling.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38111 from ronen-fr/wip-ronenf-scrub-v2

osd: Extracting scrub-related code from the PG & friends

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38566 from tchaikov/crimson-tools

crimson/tools: use helpers from byteorder.hh

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38565 from tchaikov/wip-crimson-cmake

cmake: add alien::cflags and move SubProcess.cc into crimson-alien-common

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: ObjectContext is not a Blocker anymore.

This is a follow-up to the c2dc437f78cd8be393c4d2eb8d79ce9c351ec4de
which removes unused OBC mebers after the with_lock() transition.

I'm not sure there is a reason to keep the `Blocker` bits as all
`ObjectContext::dump_detail()` does is just dumping `object_state_t`
while entering the OBC stage is already denoted by `PGPipeline::get_obc`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: add "alien::cflags" library

which exposes the definitions needed for building alienized bluestore
and the shared facilities compiled with -DWITH_ALIEN and -DWITH_SEASTAR

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move SubProcess.cc from crimson-common to crimson-alien-common

SubProcess is used by blkdev.cc only

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36146 from rhcs-dashboard/wip-46577-master

mgr: add types to MODULE_OPTIONS

Reviewed-by: p-se <pseidensal@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: sebastian-philipp <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38406 from votdev/issue_48438_drop_rgw_instance

mgr/dashboard: Drop invalid RGW client instances, improve logging

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/dev/crimson: exemplify errorator usage in error-handling.rst

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/dev/crimson: add error-handling.rst

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37642 from alimaredia/wip-rgw-apache-log-timestamp-change

rgw: add request time into beast log instead of current time

Merge pull request #38559 from xxhdx1985126/wip-crimson-bug-fix

crimson/osd: fix two bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/tools: use helpers from byteorder.hh

simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: recovery read lock should be held across the hole object push phase

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

mgr/dashboard: Drop invalid RGW client instances, improve logging

* Log RGW settings for easier problem tracking.
* Drop RGW client instances that do not exist anymore. This happens when a RGW service is removed via the orchestrator. The Dashboard tries to access the RGW client using the previous settings which leads to an error which might be confusing. Without dropping the error message 'RGW REST API failed request ...' is displayed instead of the correct one 'No RGW found'. Dropping the RGW client instance will produce correct error messages the next time the backend tries to establish a new connection.

Fixes: https://tracker.ceph.com/issues/48586
Signed-off-by: Volker Theile <vtheile@suse.com>

mgr: add types to MODULE_OPTIONS

Uses a typed Option class for MODULE_OPTIONS and replaces those in the
modules that are checked by mypy.

Fixes: https://tracker.ceph.com/issues/46577
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

admin/doc-requirements.txt: require breathe >= 4.20

to be compatible with Sphinx 3.2
see https://github.com/michaeljones/breathe/tree/v4.20.0#change-log

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37047 from sebastian-philipp/doc-cephadm-rm-host

doc/cephadm: Add remove host to operations

Reviewed-by: Zac Dover <zac.dover@gmail.com>

crimson/osd: allow obc to promote lock state from none to excl

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #38545 from tchaikov/wip-crimson-test-errorator

test/crimson: add a test to verify that errorator does not copy

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/crimson: add a test to verify that errorator does not copy

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: move member variable into test

so the test is more self-contained.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: s/do_until_test_t/errorator_test_t/

so we can reuse this test suite for more errorator related tests.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38554 from tchaikov/wip-admin-build-doc

admin/build-doc: consolidate the build deps into admin/doc-requirements.txt

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38547 from varshar16/wip-kubejacker-updates

script/kubejacker: Add python_common library and misc updates

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

admin/build-doc: drop comments for building dummy C bindings

we don't build dummy C bindings since
2235c2db0ecc7a8fbe165aba2ea8aaf8189d349c

Signed-off-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: revert "upgrade pip setuptools first"

per Radoslow Zarzynski, the build failure was due to missing Python.h
for the python version he was using on the system.

This reverts commit bc73204812907b1e55f0f4b8ad977bcce8834c6f.

Signed-off-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: stop passing --use-feature=2020-resolver to pip

to silence the warning of

WARNING: --use-feature=2020-resolver no longer has any effect, since it is now the default dependency resolver in pip. This will become an error in pip 21.0.

Signed-off-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: consolidate the build deps into admin/doc-pybind.txt

since all the python bindings are required for building the doc, extract
them into admin/doc-pybind.txt.

because the pybind python extensions require Cython python module to build,
we have to move them  into another requirement file, and install them after
Cython is installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37433 from aclamk/wip-rocksdb-better-options

kv/RocksDBStore: Added ability to set compound options

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #38020 from mdw-at-linuxbox/teuth-fix-x509-san

qa/tasks/openssl_keys.py: add subjectAltName to certificates

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38282 from vosdev/ceph-pool-alert

mgr/prometheus: Fix 'pool filling up' with >50% usage

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #38290 from athanatos/sjust/wip-transaction-manager-nbd

crimson/seastore: add journaling checksumming and nbd based simple perf tester

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38322 into master

* refs/pull/38322/head:
test_cephfs_shell: add tests for ls command

Reviewed-by: Rishabh Dave <ridave@redhat.com>

crimson/os/seastore/journal: fix typo in comment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager: allocate page aligned in read helper

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cached_extent: always allocate page aligned

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: remove now unused scan_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: rework jorunal::scan_extents to use scan_valid_records

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: replace scan_segment with scan_valid_records in replay_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add scan_valid_records

scan_valid_records validates record checksums and uses last_comitted
to avoid checking data crcs except on the final few records.  Will
replace scan_segment in a few patches.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #38444 from tchaikov/wip-msgr-string-view

msg/async: drop unused member variable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38439 from adk3798/iscsi-stuff

mgr/cephadm: improve iscsi dashboard config

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38533 from sebastian-philipp/cephadm-mypy-no-untyped

mgr/cephadm: disallow_untyped_defs=True

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38544 from tchaikov/wip-build-doc-auto-build

admin/build-doc: add livehtml target

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/seastore/journal: try_decode* pass bl as const &

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: validate header crc on read

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree/btree_range_pin: remove obsolete assert

In the event that the only two children of the root are merged, they
as well as the old root will be retired meaning that check_parent
will find no parent.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: maintain committed_to on each record

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use specified size types for header

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix debugging lines

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add data and metadata checksums

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node: collapse root on merge if down to one element

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: fix merge_entry return values

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: check segment nonce in scan_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: factor out read_segment_header

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: propogate segment headers with replay segments

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: record segment/record nonces based on seastore_id

Signed-off-by: Samuel Just <sjust@redhat.com>

include/uuid: make bytes() return const *

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager: add seastore_meta_t with uuid

Signed-off-by: Samuel Just <sjust@redhat.com>

include/uuid: convert to denc

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manger/ephemeral: add final annotations

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: use unqualified en/decode for denc types

Otherwise, if definitions from encoding.h are in scope the templated
denc one won't be considered.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix error returns in replay

SegmentManager is allowed to return an enoent if the segment isn't
readable.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../ephemeral: introduce pauses for testing

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: send aligned buffer in initialize_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/: add really simple nbd driver for TransactionManager

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #38413 from pcuzner/fix-yaml-error

orchestrator: validate osd yaml

rgw: add request time into beast log instead of current time

Change time format on the line that prints a
summary of the request in apache common log format
to match the format suggested by apache and encapsulate
apache time formatting in stream output operator.

For more information visit:
https://httpd.apache.org/docs/current/logs.html#common

Signed-off-by: Ali Maredia <amaredia@redhat.com>
Co-authored-by: Casey Bodley <cbodley@redhat.com>

doc/dev/kubernetes: Add minikube tips and misc updates

Signed-off-by: Varsha Rao <varao@redhat.com>

script/kubejacker: Add python_common library

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge PR #38511 into master

* refs/pull/38511/head:
qa/tasks/cephfs/nfs: Check if host ip is in cluster info output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

admin/build-doc: upgrade pip setuptools first

on certain distros, buggy pip and setuptools are used, so we need to
upgrade them first.

see also: googleapis/google-cloud-python#2990 and pypa/setuptools#885

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38519 from rhcs-dashboard/fix-47727-master

mgr/dashboard: RGW User Form is validating disabled fields

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38125 from rhcs-dashboard/improve-CRUSH-info-columns

mgr/dashboard: Replace Replica size and Erasure code profile with a single column

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38494 from dillaman/wip-45694

librbd: fix sporadic failures in TestMigration.StressLive

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38515 from tchaikov/wip-script-run-cbt

script/run-cbt.sh: drop bashism

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

admin/build-doc: add livehtml target

it is time-consuming to rebuild the python bindings every time we
rebuild the document, it'd be ideal if we could just build document.

in this change, in addition to "html" and "doc", "livehtml" argument is
now supported by build-doc script, so one can just use

./build-doc livehtml

to build and start a web server. whenever a change in doc/ is detected,
the document is rebuilt. for more details, see
https://pypi.org/project/sphinx-autobuild/

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd/deep_copy: added new migrating flag to object copy

The migration operation and the copyup state machine will set
this flag when attempting to perform a deep-copy due to a
live-migration.

This flag will prevent a possible race condition between the
start of the object deep-copy when migration was enabled and
the writing portion of the deep-copy when migration might
have completed via external means.

Fixes: https://tracker.ceph.com/issues/45694
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/deep_copy: added bitwise flag parameter to object copy

This initial version subsumes the original "flatten" boolean flag.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

script/kubejacker: Add podman push example for plain HTTP registry

If the image is pushed to a plain HTTP registry then tls-verify needs to be set
to false. This patch adds an example for it.

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #38486 from Rjerk/fix-mds-conf-typo

doc: fix a couple of typos

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38542 from SMIL-Infra/wip-doc-conf-typo

doc: Fixes a typo in mon_dns_srv_name.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38517 from dillaman/wip-48525

rbd-mirror: bad state and crashes in snapshot-based mirroring

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38492 from dillaman/wip-librbd-io-tweaks

librbd: minor fix to IO shut-down and increased IO logging

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc: Fixes a typo in mon_dns_srv_name.

Signed-off-by: 胡玮文 <huww98@outlook.com>

Merge pull request #38480 from yician1000/fix_48488

mgr/restful: fix TypeError occurring in _gather_osds()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38384 from bk201/wip-48406

mgr/dashboard: fix test_dashboard_e2e.sh errors

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37785 from lxbsz/conf1

common/config: do not expand the pid for socket admin

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38417 from changchengx/osdmap

osd/OSDMonitor: correct duplicated output info

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38453 from b-ranto/wip-crash-serial

mgr/crash: Serialize command handling

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38501 from tchaikov/wip-cmake-crimson-test

test/crimson: do not link against crimson-{os,common}

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38516 from tchaikov/wip-cmake-rocksdb

cmake: stop rebuilding rocksdb everytime

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

orchestrator: drop redundant try/except

YAML is validated up front, so the additional try/except
clause is not needed.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #35314 from zdover23/wip-doc-elasticsearch-7-warning

doc/radosgw: Note that ElasticSearch 7 doesn't work. Just 6 and lower.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>