mgr/dashboard: Drop invalid RGW client instances, improve logging

* Log RGW settings for easier problem tracking.
* Drop RGW client instances that do not exist anymore. This happens when a RGW service is removed via the orchestrator. The Dashboard tries to access the RGW client using the previous settings which leads to an error which might be confusing. Without dropping the error message 'RGW REST API failed request ...' is displayed instead of the correct one 'No RGW found'. Dropping the RGW client instance will produce correct error messages the next time the backend tries to establish a new connection.

Fixes: https://tracker.ceph.com/issues/48586
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #37047 from sebastian-philipp/doc-cephadm-rm-host

doc/cephadm: Add remove host to operations

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38545 from tchaikov/wip-crimson-test-errorator

test/crimson: add a test to verify that errorator does not copy

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/crimson: add a test to verify that errorator does not copy

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: move member variable into test

so the test is more self-contained.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: s/do_until_test_t/errorator_test_t/

so we can reuse this test suite for more errorator related tests.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38554 from tchaikov/wip-admin-build-doc

admin/build-doc: consolidate the build deps into admin/doc-requirements.txt

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38547 from varshar16/wip-kubejacker-updates

script/kubejacker: Add python_common library and misc updates

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

admin/build-doc: drop comments for building dummy C bindings

we don't build dummy C bindings since
2235c2db0ecc7a8fbe165aba2ea8aaf8189d349c

Signed-off-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: revert "upgrade pip setuptools first"

per Radoslow Zarzynski, the build failure was due to missing Python.h
for the python version he was using on the system.

This reverts commit bc73204812907b1e55f0f4b8ad977bcce8834c6f.

Signed-off-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: stop passing --use-feature=2020-resolver to pip

to silence the warning of

WARNING: --use-feature=2020-resolver no longer has any effect, since it is now the default dependency resolver in pip. This will become an error in pip 21.0.

Signed-off-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: consolidate the build deps into admin/doc-pybind.txt

since all the python bindings are required for building the doc, extract
them into admin/doc-pybind.txt.

because the pybind python extensions require Cython python module to build,
we have to move them  into another requirement file, and install them after
Cython is installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37433 from aclamk/wip-rocksdb-better-options

kv/RocksDBStore: Added ability to set compound options

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #38020 from mdw-at-linuxbox/teuth-fix-x509-san

qa/tasks/openssl_keys.py: add subjectAltName to certificates

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38282 from vosdev/ceph-pool-alert

mgr/prometheus: Fix 'pool filling up' with >50% usage

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #38290 from athanatos/sjust/wip-transaction-manager-nbd

crimson/seastore: add journaling checksumming and nbd based simple perf tester

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38322 into master

* refs/pull/38322/head:
test_cephfs_shell: add tests for ls command

Reviewed-by: Rishabh Dave <ridave@redhat.com>

crimson/os/seastore/journal: fix typo in comment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager: allocate page aligned in read helper

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cached_extent: always allocate page aligned

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: remove now unused scan_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: rework jorunal::scan_extents to use scan_valid_records

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: replace scan_segment with scan_valid_records in replay_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add scan_valid_records

scan_valid_records validates record checksums and uses last_comitted
to avoid checking data crcs except on the final few records.  Will
replace scan_segment in a few patches.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #38444 from tchaikov/wip-msgr-string-view

msg/async: drop unused member variable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38439 from adk3798/iscsi-stuff

mgr/cephadm: improve iscsi dashboard config

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38533 from sebastian-philipp/cephadm-mypy-no-untyped

mgr/cephadm: disallow_untyped_defs=True

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38544 from tchaikov/wip-build-doc-auto-build

admin/build-doc: add livehtml target

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/seastore/journal: try_decode* pass bl as const &

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: validate header crc on read

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree/btree_range_pin: remove obsolete assert

In the event that the only two children of the root are merged, they
as well as the old root will be retired meaning that check_parent
will find no parent.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: maintain committed_to on each record

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use specified size types for header

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix debugging lines

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add data and metadata checksums

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node: collapse root on merge if down to one element

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: fix merge_entry return values

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: check segment nonce in scan_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: factor out read_segment_header

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: propogate segment headers with replay segments

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: record segment/record nonces based on seastore_id

Signed-off-by: Samuel Just <sjust@redhat.com>

include/uuid: make bytes() return const *

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager: add seastore_meta_t with uuid

Signed-off-by: Samuel Just <sjust@redhat.com>

include/uuid: convert to denc

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manger/ephemeral: add final annotations

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: use unqualified en/decode for denc types

Otherwise, if definitions from encoding.h are in scope the templated
denc one won't be considered.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix error returns in replay

SegmentManager is allowed to return an enoent if the segment isn't
readable.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../ephemeral: introduce pauses for testing

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: send aligned buffer in initialize_segment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/: add really simple nbd driver for TransactionManager

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #38413 from pcuzner/fix-yaml-error

orchestrator: validate osd yaml

doc/dev/kubernetes: Add minikube tips and misc updates

Signed-off-by: Varsha Rao <varao@redhat.com>

script/kubejacker: Add python_common library

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge PR #38511 into master

* refs/pull/38511/head:
qa/tasks/cephfs/nfs: Check if host ip is in cluster info output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

admin/build-doc: upgrade pip setuptools first

on certain distros, buggy pip and setuptools are used, so we need to
upgrade them first.

see also: googleapis/google-cloud-python#2990 and pypa/setuptools#885

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38519 from rhcs-dashboard/fix-47727-master

mgr/dashboard: RGW User Form is validating disabled fields

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38125 from rhcs-dashboard/improve-CRUSH-info-columns

mgr/dashboard: Replace Replica size and Erasure code profile with a single column

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38494 from dillaman/wip-45694

librbd: fix sporadic failures in TestMigration.StressLive

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38515 from tchaikov/wip-script-run-cbt

script/run-cbt.sh: drop bashism

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

admin/build-doc: add livehtml target

it is time-consuming to rebuild the python bindings every time we
rebuild the document, it'd be ideal if we could just build document.

in this change, in addition to "html" and "doc", "livehtml" argument is
now supported by build-doc script, so one can just use

./build-doc livehtml

to build and start a web server. whenever a change in doc/ is detected,
the document is rebuilt. for more details, see
https://pypi.org/project/sphinx-autobuild/

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd/deep_copy: added new migrating flag to object copy

The migration operation and the copyup state machine will set
this flag when attempting to perform a deep-copy due to a
live-migration.

This flag will prevent a possible race condition between the
start of the object deep-copy when migration was enabled and
the writing portion of the deep-copy when migration might
have completed via external means.

Fixes: https://tracker.ceph.com/issues/45694
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/deep_copy: added bitwise flag parameter to object copy

This initial version subsumes the original "flatten" boolean flag.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

script/kubejacker: Add podman push example for plain HTTP registry

If the image is pushed to a plain HTTP registry then tls-verify needs to be set
to false. This patch adds an example for it.

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #38486 from Rjerk/fix-mds-conf-typo

doc: fix a couple of typos

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38542 from SMIL-Infra/wip-doc-conf-typo

doc: Fixes a typo in mon_dns_srv_name.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38517 from dillaman/wip-48525

rbd-mirror: bad state and crashes in snapshot-based mirroring

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38492 from dillaman/wip-librbd-io-tweaks

librbd: minor fix to IO shut-down and increased IO logging

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc: Fixes a typo in mon_dns_srv_name.

Signed-off-by: 胡玮文 <huww98@outlook.com>

Merge pull request #38480 from yician1000/fix_48488

mgr/restful: fix TypeError occurring in _gather_osds()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38384 from bk201/wip-48406

mgr/dashboard: fix test_dashboard_e2e.sh errors

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37785 from lxbsz/conf1

common/config: do not expand the pid for socket admin

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38417 from changchengx/osdmap

osd/OSDMonitor: correct duplicated output info

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38453 from b-ranto/wip-crash-serial

mgr/crash: Serialize command handling

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38501 from tchaikov/wip-cmake-crimson-test

test/crimson: do not link against crimson-{os,common}

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38516 from tchaikov/wip-cmake-rocksdb

cmake: stop rebuilding rocksdb everytime

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

orchestrator: drop redundant try/except

YAML is validated up front, so the additional try/except
clause is not needed.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #35314 from zdover23/wip-doc-elasticsearch-7-warning

doc/radosgw: Note that ElasticSearch 7 doesn't work. Just 6 and lower.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38523 from tchaikov/wip-gcc-toolset-9

install-deps.sh: do not use gcc-toolset-9 from copr

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38157 from sebastian-philipp/labeler-orchestrator

github/labeler: Add some more to cephadm, add orchestrator

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38408 from trociny/wip-rbd-group-snap-create2

librbd: extend group snap create API to support flags

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38518 from trociny/wip-rbd-test-memleak

test/librbd: fix memory leak in TestLibRBD.ConcurentOperations

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38490 from sebastian-philipp/mypy-0.790

src,qa,dashboard: Upgrade to mypy 0.790

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38075 from joke-lee/s3-auth-date-fix

rgw: fix RequestTimeTooSkewed when Date contain RFC-822/ISO 8601 standard timezone

Merge pull request #37926 from BryceCao/wip-catch-non-int-exception

rgw : catch non int exception

Merge pull request #37909 from alexandrsushko/prometheus-pg-laggy-46142

mgr: Fix for dashboard/prometheus failure due to laggy pg state

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #38131 from linuxbox2/wip-lc-ix

rgwlc: correctly dimension lc shard index vector

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mgr/cephadm: disallow_untyped_defs=True

simplifies mypy.ini

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

github/labeler: Add some more to cephadm, add orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38524 from tchaikov/wip-crimson-qa-rbd

qa/suites: add crimson-rados/rbd test suites

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38470 from sebastian-philipp/cephadm-command-unit-verbose

cephadm: command_unit: call systemctl with verbose=True

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38009 from ideepika/wip-48058-cephadm

qa/tasks/cephadm: update_info_yam() only if ctx.archive is valid

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36551 from CongMinYin/immutable_object_cache_throttle

tools: add throttle mechanism to immutable object cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: do not attempt to unlink from more recent snapshots

The snapshot-based mirroring replayer should only attempt to unlink
from any snapshots that are older than the end remote snapshot id to
prevent the remote side from incorrectly deleted the snapshot.

Fixes: https://tracker.ceph.com/issues/48527
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/mirror: unlink peer might recursively loop

If the mirror peer set is (incorrectly) empty, it's not currently
possible for the unlink peer state machine to properly delete the
snapshot. This can result in a recursive loop between the create
primary snapshot state machine and the unlink peer state machine
until the stack depth grows too large.

Fixes: https://tracker.ceph.com/issues/48525
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites: add crimson-rados/rbd test suites

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38435 from votdev/issue_48449_test_standby

mgr/dashboard: test_standby* (tasks.mgr.test_dashboard.TestDashboard) failed locally

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38441 from bk201/wip-48455

mgr/dashboard: adjust refresh intervals of Services and Daemons

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38212 from sebastian-philipp/mgr-cephadm-type-annotations-mandatory-module

mgr/cephadm: make type annotations for module.py mandatory

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

install-deps.sh: do not use gcc-toolset-9 from copr

since both CentOS and RHEL now include the updated
gcc-toolset-9-gcc-9.2.1-2.3.el8.

see also https://bugzilla.redhat.com/show_bug.cgi?id=1853900

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use correct fmt string in logger().debug(...)

Signed-off-by: Kefu Chai <kchai@redhat.com>