qa/cephfs: make filelock_interrupt.py work with python3

Fixes: https://tracker.ceph.com/issues/43513
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge PR #32631 into master

* refs/pull/32631/head:
doc: add mds map to list of monitor assets

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32642 into master

* refs/pull/32642/head:
qa: save MDS epoch barrier
qa: prefer rank_asok

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32667 into master

* refs/pull/32667/head:
mds: track high water mark for purges
qa: use correct variable for exception debug
mds: mark purge queue protected members private

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: save MDS epoch barrier

Rather than verify the latest OSDMap is the same as the rank's
osdmap_epoch_barrier, just use the rank's version. The OSDMap may change
out-of-band with the test startup and thus the epoch's would diverge.
The file system and rank is fresh for each test so there's no reason to
care if the MDS barrier is one epoch behind the latest.

Fixes: https://tracker.ceph.com/issues/43554
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: prefer rank_asok

This is a trivial refactor.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32629 from tchaikov/wip-cephfs.pyx

libcephfs: add missing declaration of ceph_getaddrs()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: track high water mark for purges

This makes the corresponding test not racy.

Fixes: https://tracker.ceph.com/issues/16881
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use correct variable for exception debug

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: mark purge queue protected members private

Nothing inherits from PQ.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32652 from rhcs-dashboard/fix-43607-master

mgr/dashboard: fix improper URL checking

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #32524 into master

* refs/pull/32524/head:
qa/xfstests_dev: change deps for xfstests-dev on ubuntu
qa/cephfs: change deps for xfstests-dev on centos8
vstart_runnner: add sh method to LocalRemote

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32637 into master

* refs/pull/32637/head:
client: Add is_dir check before changing directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #32676 into master

* refs/pull/32676/head:
mds: don't respond getattr with -EROFS when mds is readonly

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32639 into master

* refs/pull/32639/head:
qa: rename kcephfs distro overrides

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #32696 into master

* refs/pull/32696/head:
mgr/volumes: fail removing subvolume with snapshots

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32645 into master

* refs/pull/32645/head:
mgr/volumes: remove stale subvolume module

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32659 into master

* refs/pull/32659/head:
mds: do not include metric_spec in MClientSession from MDS

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #32709 into master

* refs/pull/32709/head:
mds/FSMap: fix adjust_standby_fscid

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32727 from liewegas/fix-rgw-os

qa/suite/rgw/verify: valgrind on centos again!

Merge PR #32668 into master

* refs/pull/32668/head:
qa: ignore slow ops for ffsb workunit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32723 from yuvalif/wip-yuval-notif-add-opaque-data

rgw/notification: add opaque data

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #32565 from rhcs-dashboard/43534-decouple-backend-unit-tests

mgr/dashboard: decouple backend unit tests from build.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32592 into master

* refs/pull/32592/head:
mgr/pg_autoscaler: calculate pool_pg_target using pool size

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Update npm package caniuse-lite (#32633)

mgr/dashboard: Update npm package caniuse-lite

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #32702 from rzarzynski/wip-bug-43646

common/bl: fix the dangling last_p issue.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32325 from Kriechi/fix-42982

monitoring: fix prometheus alert for full pools

Merge pull request #32692 from tchaikov/wip-remove-btree-dump

include/cpp-btree: drop btree::dump()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

qa/suite/rgw/verify: valgrind on centos again!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32724 into master

* refs/pull/32724/head:
qa/workunits/rados/test_crash.sh: suppress core files

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rados/test_crash.sh: suppress core files

The cores will make teuthology fail the job--and we don't want them for
this test, where we are deliberately causing crashes.

Fixes: https://tracker.ceph.com/issues/43653
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32684 from dillaman/wip-rbd-el8

tests: corrected issues with RBD tests under EL8 distros

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #32681 into master

* refs/pull/32681/head:
mgr/cephadm: fix section name for mon options in ceph.conf

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32006 into master

* refs/pull/32006/head:
mgr/cephadm: revert to OrchestratorClientMixin child
mgr/cephadm: initial upgrade logic
cephadm: return 'cephadm pull' result as JSON
mgr/cephadm: upgrade start/stop/pause/resume
mgr/cephadm: track ceph version as well as container in on 'upgrade check'
mgr/orchestrator_cli: upgrade {start,pause,resume,stop}
mgr/cephadm: fix upgrade check
mon/ConfigKeyService: remove config-key set warning

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32685 into master

* refs/pull/32685/head:
cephadm: remove -crash container before starting it
cephadm: stop the crash container the nice way
cephadm: no need for podman rm when passing --rm to run
ceph-crash: exit code 0 on SIGINT, SIGTERM

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32720 from tchaikov/wip-disable-crimson-socket-test

test/crimson: remove unittest_seastar_socket temporarily

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw/notification: add opaque data

opaque data may be set in topic configuration and later on sent inside
nottifications triggered by that topic.

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #32687 from dillaman/wip-42097

librbd: support compression allocation hints to the OSD

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32671 from dillaman/wip-43537

rbd-mirror: periodically poll remote mirror configuration

Reviewed-by: Mykola Golub <mgolub@suse.com>

test/crimson: remove unittest_seastar_socket temporarily

see also #32632. before we have the complete fix, we should disable this
test to avoid more noise from this test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32719 from liewegas/wip-o-upgrades

qa/suites/upgrade: fix mimic-x-singleton

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32673 into master

* refs/pull/32673/head:
crush/CrushWrapper: behave with empty weight vector
cmake: no need to link libglobal and libblkid for testing crush
test/crush: no need to use libglobal for testing crush

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32439 into master

* refs/pull/32439/head:
test/kv_store_bench: Fix double free error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32640 into master

* refs/pull/32640/head:
cephadm: fix help message for bootstrap --mgr-id

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32627 into master

* refs/pull/32627/head:
test_cephadm: use container shell for ceph cmds
test_cephadm: `-d` debug no longer exists
test_cephadm: clean-up whitespace (tabs -> spaces)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32628 into master

* refs/pull/32628/head:
test: Fix wait_for_state() to wait for a PG to get into a state

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32385 into master

* refs/pull/32385/head:
mgr/cephadm: check-host on 'host add'

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32712 into master

* refs/pull/32712/head:
cephadm: SO_REUSEADDR when doing bind check

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32711 into master

* refs/pull/32711/head:
qa/standalone/mon/osd-create-pool: fix utf-8 grep LANG

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32641 into master

* refs/pull/32641/head:
cephadm: fix logging defaults

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32700 into master

* refs/pull/32700/head:
rpm: Relax the selinux policy version for centos builds

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32560 into master

* refs/pull/32560/head:
cephadm: show contextual message when port is in use

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/xfstests_dev: change deps for xfstests-dev on ubuntu

Rename python to python2 if Ubuntu distro release is 19 or later.

Fixes: https://tracker.ceph.com/issues/43522
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge PR #32713 into master

* refs/pull/32713/head:
qa/tasks/cephadm: start watching initial daemons before bootstrap
qa/tasks/cephadm: create /etc/ceph if it doesn't exist
qa/tasks/cephadm: fix log whitelist when there is no whitelist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32701 into master

* refs/pull/32701/head:
qa/tasks/ceph_manager: fix ceph-objectstore-tool calls
qa/tasks/ceph_manager: fix admin_socket remote when using cephadm
qa/tasks/ceph_manager: --no-mon-config to ceph-objectstore-tool
qa/tasks/ceph_manager: fix filestore split command
qa/tasks/ceph_manager: fix import line

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/releases/octopus: disable autoscale warnings

Existing pools might have too many/few PGs and produce a warning that
prevents us from getting to HEALTH_OK.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: SO_REUSEADDR when doing bind check

This is what the ceph daemons use; our check should do it too.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: fix ceph-objectstore-tool calls

Pass the correct paths based on whether this is the importing or exporting
OSD.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: fix admin_socket remote when using cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: --no-mon-config to ceph-objectstore-tool

The config is currently fetched at osd.admin, so the keyring is not
found.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: fix filestore split command

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: fix import line

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: start watching initial daemons before bootstrap

This lets us see output while bootstrap is happening.

(Depends on the teuthology change to use journalctl, see
teuthology commit 4fa83040b05b604280789459f095d6f2ad1b0d01.)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: create /etc/ceph if it doesn't exist

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: fix log whitelist when there is no whitelist

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/mimix-x-singleton: upgrading to octopus, not n

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32698 into master

* refs/pull/32698/head:
common: drop the unused mutable_item_history container.
common: drop the unused prune() of {safe,mutable}_item_history.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/standalone/mon/osd-create-pool: fix utf-8 grep LANG

This needs en_US.UTF-8... en_US does not work.

Fixes: https://tracker.ceph.com/issues/43422
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32663 from dzafman/wip-41016

osd: Diagnostic logging for upmap cleaning

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: revert to OrchestratorClientMixin child

self.mgr does not exist--I can't figure out what this 'mgr' reference
is and where it comes from.  So reverting to being a child for the time
being; we can clean this up later.

Signed-off-by: Sage Weil <sage@redhat.com>

mds/FSMap: fix adjust_standby_fscid

The _fscid entry may not exist if this is a new starting daemon, or
if the daemon previously didn't have an fs set and does now.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: initial upgrade logic

- upgrade mgr, mon, osd, mds
- use ok-to-stop

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: return 'cephadm pull' result as JSON

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: upgrade start/stop/pause/resume

No actual upgrading yet.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: track ceph version as well as container in on 'upgrade check'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: upgrade {start,pause,resume,stop}

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix upgrade check

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigKeyService: remove config-key set warning

This has been there for both mimic and nautilus; we can drop it for
octopus.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32298 from dang/wip-dang-zipper-4

Zipper - RGWUser

common: drop the unused mutable_item_history container.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: drop the unused prune() of {safe,mutable}_item_history.

The presence of these never-used methods obscure bug investigation.
For instance, `safe_item_history` is used to record all messenger's
addresses. As we never lock reading from it in `_conn_prefix()`, we
expect they live as long their corresponding messenger instances.
When such an item becomes corrupted, presence of `prune()` imposes
verification the method is never called – this effort could have
been avoided if we hadn't have the dangerous method.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Zipper - RGWUser

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #32624 from alimaredia/wip-s3-tests-python-3

qa/suites: use s3-tests with python3 support

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rpm: Relax the selinux policy version for centos builds

We are currently running into issues when installing the test builds
produced on centos 8 as these might have older selinux-policy-base
version than is available to the rhel 8 nodes.

We still want the builds produced on RHEL 8 to use the proper version
condition so the %{?centos} macro should help us achieve that, here.

Signed-off-by: Boris Ranto <branto@redhat.com>

cephadm: fix logging defaults

- prefix stderr log lines with 'debug '
- send cluster log to stderr (only affects mons)
- default to sending cluser log to stderr only (not to file)

This aligns things with the container-y way, as we do
with rook.  It means that if you want legacy behavior (log files),
you have *2* settings to change:

 log_to_file = true
 mon_cluster_log_to_file = true

We could default this to leave cluster log to file by default
since they aren't that big (and we do have log rotation), but
it's not as clean as it could be...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32650 from votdev/fix_i18n

mgr/dashboard: Fix missing i18n.

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

test/cli-integration/rbd: fixed EL8 json and xml incompatibilities

Python 3 doesn't sort keys by default and xml_pp is not available under
EL8.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: use more recent qemu-iotests that support EL8

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks: use correct NFS service name for EL8 distros

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc/rbd: documented 'rbd compression hint' config option

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/bl: fix the dangling last_p issue.

Fixes: https://tracker.ceph.com/issues/43646
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rbd-mirror: ensure mirror status updater cannot attempt empty op

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/volumes: fail removing subvolume with snapshots

Fixes: http://tracker.ceph.com/issues/43645
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #32367 from capri1989/doc-gc-update

doc/radosgw/config-ref: paragraph to explain the gc settings

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #32049 from mgfritch/wip-cd-timeout

cephadm: add ability to specify a timeout

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32691 from rosinL/wip-txpkt

msg/async/dpdk: Fix infinite loop when sending packets

Reviewed-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: drop btree::dump()

btree::dump() is a helper for printing out all the leaf nodes in a
btree, it's used by "operator<<()". but this operator is not used.
let's remove it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async/dpdk: Fix infinite loop when sending packets

tx_pkt_burst loops until the next pointer is null, set next pointer of
the last segment nullptr to fix this issue.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>

cephadm: remove -crash container before starting it

This matches up with what the @.service unit does.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32114 into master

* refs/pull/32114/head:
doc: split index.rst of dev guide
doc: tweak few section titles in dev guide
doc: don't display contents twice in dev guide
doc: add guide for running tests with teuthology

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>