doc: Fix mds capabilities for manual deployment

Now matches auth-config-ref.rst, and actually works.

Signed-off-by: Ivan Jager <aij+git@mrph.org>

Merge pull request #37433 from aclamk/wip-rocksdb-better-options

kv/RocksDBStore: Added ability to set compound options

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #38020 from mdw-at-linuxbox/teuth-fix-x509-san

qa/tasks/openssl_keys.py: add subjectAltName to certificates

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38282 from vosdev/ceph-pool-alert

mgr/prometheus: Fix 'pool filling up' with >50% usage

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #38290 from athanatos/sjust/wip-transaction-manager-nbd

crimson/seastore: add journaling checksumming and nbd based simple perf tester

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38322 into master

* refs/pull/38322/head:
test_cephfs_shell: add tests for ls command

Reviewed-by: Rishabh Dave <ridave@redhat.com>

crimson/os/seastore/journal: fix typo in comment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager: allocate page aligned in read helper

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cached_extent: always allocate page aligned

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: remove now unused scan_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: rework jorunal::scan_extents to use scan_valid_records

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: replace scan_segment with scan_valid_records in replay_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add scan_valid_records

scan_valid_records validates record checksums and uses last_comitted
to avoid checking data crcs except on the final few records.  Will
replace scan_segment in a few patches.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #38444 from tchaikov/wip-msgr-string-view

msg/async: drop unused member variable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38439 from adk3798/iscsi-stuff

mgr/cephadm: improve iscsi dashboard config

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38533 from sebastian-philipp/cephadm-mypy-no-untyped

mgr/cephadm: disallow_untyped_defs=True

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38544 from tchaikov/wip-build-doc-auto-build

admin/build-doc: add livehtml target

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/seastore/journal: try_decode* pass bl as const &

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: validate header crc on read

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree/btree_range_pin: remove obsolete assert

In the event that the only two children of the root are merged, they
as well as the old root will be retired meaning that check_parent
will find no parent.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: maintain committed_to on each record

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use specified size types for header

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix debugging lines

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add data and metadata checksums

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node: collapse root on merge if down to one element

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: fix merge_entry return values

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: check segment nonce in scan_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: factor out read_segment_header

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: propogate segment headers with replay segments

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: record segment/record nonces based on seastore_id

Signed-off-by: Samuel Just <sjust@redhat.com>

include/uuid: make bytes() return const *

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager: add seastore_meta_t with uuid

Signed-off-by: Samuel Just <sjust@redhat.com>

include/uuid: convert to denc

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manger/ephemeral: add final annotations

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: use unqualified en/decode for denc types

Otherwise, if definitions from encoding.h are in scope the templated
denc one won't be considered.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix error returns in replay

SegmentManager is allowed to return an enoent if the segment isn't
readable.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../ephemeral: introduce pauses for testing

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: send aligned buffer in initialize_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/: add really simple nbd driver for TransactionManager

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #38413 from pcuzner/fix-yaml-error

orchestrator: validate osd yaml

Merge PR #38511 into master

* refs/pull/38511/head:
qa/tasks/cephfs/nfs: Check if host ip is in cluster info output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

admin/build-doc: upgrade pip setuptools first

on certain distros, buggy pip and setuptools are used, so we need to
upgrade them first.

see also: googleapis/google-cloud-python#2990 and pypa/setuptools#885

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38519 from rhcs-dashboard/fix-47727-master

mgr/dashboard: RGW User Form is validating disabled fields

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38125 from rhcs-dashboard/improve-CRUSH-info-columns

mgr/dashboard: Replace Replica size and Erasure code profile with a single column

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38494 from dillaman/wip-45694

librbd: fix sporadic failures in TestMigration.StressLive

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38515 from tchaikov/wip-script-run-cbt

script/run-cbt.sh: drop bashism

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

admin/build-doc: add livehtml target

it is time-consuming to rebuild the python bindings every time we
rebuild the document, it'd be ideal if we could just build document.

in this change, in addition to "html" and "doc", "livehtml" argument is
now supported by build-doc script, so one can just use

./build-doc livehtml

to build and start a web server. whenever a change in doc/ is detected,
the document is rebuilt. for more details, see
https://pypi.org/project/sphinx-autobuild/

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd/deep_copy: added new migrating flag to object copy

The migration operation and the copyup state machine will set
this flag when attempting to perform a deep-copy due to a
live-migration.

This flag will prevent a possible race condition between the
start of the object deep-copy when migration was enabled and
the writing portion of the deep-copy when migration might
have completed via external means.

Fixes: https://tracker.ceph.com/issues/45694
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/deep_copy: added bitwise flag parameter to object copy

This initial version subsumes the original "flatten" boolean flag.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38486 from Rjerk/fix-mds-conf-typo

doc: fix a couple of typos

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38542 from SMIL-Infra/wip-doc-conf-typo

doc: Fixes a typo in mon_dns_srv_name.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38517 from dillaman/wip-48525

rbd-mirror: bad state and crashes in snapshot-based mirroring

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38492 from dillaman/wip-librbd-io-tweaks

librbd: minor fix to IO shut-down and increased IO logging

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc: Fixes a typo in mon_dns_srv_name.

Signed-off-by: 胡玮文 <huww98@outlook.com>

Merge pull request #38480 from yician1000/fix_48488

mgr/restful: fix TypeError occurring in _gather_osds()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38384 from bk201/wip-48406

mgr/dashboard: fix test_dashboard_e2e.sh errors

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37785 from lxbsz/conf1

common/config: do not expand the pid for socket admin

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38417 from changchengx/osdmap

osd/OSDMonitor: correct duplicated output info

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38453 from b-ranto/wip-crash-serial

mgr/crash: Serialize command handling

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38501 from tchaikov/wip-cmake-crimson-test

test/crimson: do not link against crimson-{os,common}

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38516 from tchaikov/wip-cmake-rocksdb

cmake: stop rebuilding rocksdb everytime

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

orchestrator: drop redundant try/except

YAML is validated up front, so the additional try/except
clause is not needed.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #35314 from zdover23/wip-doc-elasticsearch-7-warning

doc/radosgw: Note that ElasticSearch 7 doesn't work. Just 6 and lower.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38523 from tchaikov/wip-gcc-toolset-9

install-deps.sh: do not use gcc-toolset-9 from copr

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38157 from sebastian-philipp/labeler-orchestrator

github/labeler: Add some more to cephadm, add orchestrator

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38408 from trociny/wip-rbd-group-snap-create2

librbd: extend group snap create API to support flags

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38518 from trociny/wip-rbd-test-memleak

test/librbd: fix memory leak in TestLibRBD.ConcurentOperations

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38490 from sebastian-philipp/mypy-0.790

src,qa,dashboard: Upgrade to mypy 0.790

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38075 from joke-lee/s3-auth-date-fix

rgw: fix RequestTimeTooSkewed when Date contain RFC-822/ISO 8601 standard timezone

Merge pull request #37926 from BryceCao/wip-catch-non-int-exception

rgw : catch non int exception

Merge pull request #37909 from alexandrsushko/prometheus-pg-laggy-46142

mgr: Fix for dashboard/prometheus failure due to laggy pg state

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #38131 from linuxbox2/wip-lc-ix

rgwlc: correctly dimension lc shard index vector

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mgr/cephadm: disallow_untyped_defs=True

simplifies mypy.ini

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

github/labeler: Add some more to cephadm, add orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38524 from tchaikov/wip-crimson-qa-rbd

qa/suites: add crimson-rados/rbd test suites

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38470 from sebastian-philipp/cephadm-command-unit-verbose

cephadm: command_unit: call systemctl with verbose=True

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38009 from ideepika/wip-48058-cephadm

qa/tasks/cephadm: update_info_yam() only if ctx.archive is valid

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36551 from CongMinYin/immutable_object_cache_throttle

tools: add throttle mechanism to immutable object cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: do not attempt to unlink from more recent snapshots

The snapshot-based mirroring replayer should only attempt to unlink
from any snapshots that are older than the end remote snapshot id to
prevent the remote side from incorrectly deleted the snapshot.

Fixes: https://tracker.ceph.com/issues/48527
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/mirror: unlink peer might recursively loop

If the mirror peer set is (incorrectly) empty, it's not currently
possible for the unlink peer state machine to properly delete the
snapshot. This can result in a recursive loop between the create
primary snapshot state machine and the unlink peer state machine
until the stack depth grows too large.

Fixes: https://tracker.ceph.com/issues/48525
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites: add crimson-rados/rbd test suites

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38435 from votdev/issue_48449_test_standby

mgr/dashboard: test_standby* (tasks.mgr.test_dashboard.TestDashboard) failed locally

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38441 from bk201/wip-48455

mgr/dashboard: adjust refresh intervals of Services and Daemons

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38212 from sebastian-philipp/mgr-cephadm-type-annotations-mandatory-module

mgr/cephadm: make type annotations for module.py mandatory

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

install-deps.sh: do not use gcc-toolset-9 from copr

since both CentOS and RHEL now include the updated
gcc-toolset-9-gcc-9.2.1-2.3.el8.

see also https://bugzilla.redhat.com/show_bug.cgi?id=1853900

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use correct fmt string in logger().debug(...)

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: exclude unit tests from the "all" target

no need to build unit tests when "make all", unless required explicitly,
like "make tests".

Signed-off-by: Kefu Chai <kchai@redhat.com>

src/*: do not pass cct to ceph_version_to_str()

in e5b1ae5554c4d8a20f9f0ff562b231ad0b0ba0ab, a new option named
"debug_version_for_testing" is introduced to override the version so
we can test version check.

in crimson, we have two families of shared functions.

- one of them is used by alien store. they are compiled with
  -DWITH_SEASTAR and -DWITH_ALIEN, to enable the shim code between
  seastar and POSIX thread.
- another is used by crimson in general. where no lock is allowed.

currently, we use the "crimson" and "ceph" namespace to differentiate
these two families of functions, so they can colocate in the same
executable without violating the ODR. see src/include/common_fwd.h for
more details.

the functions defined in src/common/version.cc are also shared by
alien store and crimson code. and because we have different
implementations of `CephContext` in crimson and in classic OSD (i.e.
alienstore), we have to have different implementations of this function
as well, if we follow the same approach. but since these functions are
very simple and are non-blocking, there is not much value in
differentiating them, it is better to inject the test settings using
environment variable instead of using ceph option subsystem.

in this change, "ceph_debug_version_for_testing" environment variable is
checked instead, so that crimson and alienstore can share the same
compilation unit of version.cc. and "debug_version_for_testing" option
is removed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: reorder linked libraries of crimson-alienstore

so the libraries like libkv can access the symbols exposed by
crimson-alien-common.

this change should address the link failures like:

/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.0.0-8049-g1ab93e4d/rpm/el8/BUILD/ceph-16.0.0-8049-g1ab93e4d/src/common/PriorityCache.cc:175:
undefined reference to `ceph::common::PerfCountersBuilder::~PerfCountersBuilder()'

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link crimson-alienstore against crimson-os

crimson-os contains crimson-alienstore, we should not link the latter
against the former. this change partially reverts
490b6322fbbece053f1d92b29ae101bfb0976007

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link unittest-seastar-alienstore-thread-pool against crimson-alienstore

no need to link it against crimson-os in a wholesale, crimson-alienstore
would be enough.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: do not link against crimson-{os,common}

this change partially reverts 652dbacc7424efbd3c3175de8ba79ed29edd55c8
quite a few test does not use crimson-os at all, so no need to link
against this library.

even worse is that crimson-os contains crimson-seastore *and*
crimson-alienstore. this introduces cyclic references.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd: make "group snap create" support quiesce options

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: extend group snap create API to support flags

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: RGW User Form is validating disabled fields

When you try to submit a non valid form, angular is checking all the fields of the form, resulting in some wrong validations.

Fixes:https://tracker.ceph.com/issues/47727
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

test/librbd: fix memory leak in TestLibRBD.ConcurentOperations

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr/prometheus/module.py: defaultdict for num_by_state

num_by_state[state] += count in get_pg_status method raises KeyError
if pg state is not in PG_STATES list. PG_STATES should be synced with
osd_types.cc:pg_state_string(). But sometimes it is not. After the
KeyError raise mgr metrics are not available at all.

Fixes: https://tracker.ceph.com/issues/46142
Signed-off-by: Alexander Sushko <alexandrsushko@gmail.com>

cmake: stop rebuilding rocksdb everytime

this change was originally introduced as a part of
418bfd7bb5ec1dcec2b011e9df118c33ce38d398, and latter migrated / changed
in the current form. but the idea is the same: to rebuild rocksdb even
if the stamp file shows that it has been built. there is no need to do
so, as we don't hack RocksDB as we used to. also,it is distracting to
check this log message when rebuilding the tree. so drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38431 from tchaikov/wip-rtd-doc

pybind,doc: build stub using cython and build API doc with RTD

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@hotmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

script/run-cbt.sh: drop bashism

* "local" is not supported by POSIX shell, so drop it
* assign "$*" to a non-array variable, see
  https://github.com/koalaman/shellcheck/wiki/SC2124
* fail early if "cd" fails, see
  https://github.com/koalaman/shellcheck/wiki/SC2164
* quote "$(...)" with quotes to avoid string split

Signed-off-by: Kefu Chai <kchai@redhat.com>