qa/tasks/ceph_manager: fix post-osd-kill pg peered check

This was asserting that all PGs are active or peered, but that assertion
could fail if the concurrent workload created a new pool.

Switch to a loop that checks several times for the condition to be true.

Fixes: https://tracker.ceph.com/issues/43656
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32681 into master

* refs/pull/32681/head:
mgr/cephadm: fix section name for mon options in ceph.conf

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32006 into master

* refs/pull/32006/head:
mgr/cephadm: revert to OrchestratorClientMixin child
mgr/cephadm: initial upgrade logic
cephadm: return 'cephadm pull' result as JSON
mgr/cephadm: upgrade start/stop/pause/resume
mgr/cephadm: track ceph version as well as container in on 'upgrade check'
mgr/orchestrator_cli: upgrade {start,pause,resume,stop}
mgr/cephadm: fix upgrade check
mon/ConfigKeyService: remove config-key set warning

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32685 into master

* refs/pull/32685/head:
cephadm: remove -crash container before starting it
cephadm: stop the crash container the nice way
cephadm: no need for podman rm when passing --rm to run
ceph-crash: exit code 0 on SIGINT, SIGTERM

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32720 from tchaikov/wip-disable-crimson-socket-test

test/crimson: remove unittest_seastar_socket temporarily

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #32687 from dillaman/wip-42097

librbd: support compression allocation hints to the OSD

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32671 from dillaman/wip-43537

rbd-mirror: periodically poll remote mirror configuration

Reviewed-by: Mykola Golub <mgolub@suse.com>

test/crimson: remove unittest_seastar_socket temporarily

see also #32632. before we have the complete fix, we should disable this
test to avoid more noise from this test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32719 from liewegas/wip-o-upgrades

qa/suites/upgrade: fix mimic-x-singleton

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32673 into master

* refs/pull/32673/head:
crush/CrushWrapper: behave with empty weight vector
cmake: no need to link libglobal and libblkid for testing crush
test/crush: no need to use libglobal for testing crush

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32439 into master

* refs/pull/32439/head:
test/kv_store_bench: Fix double free error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32640 into master

* refs/pull/32640/head:
cephadm: fix help message for bootstrap --mgr-id

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32627 into master

* refs/pull/32627/head:
test_cephadm: use container shell for ceph cmds
test_cephadm: `-d` debug no longer exists
test_cephadm: clean-up whitespace (tabs -> spaces)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32628 into master

* refs/pull/32628/head:
test: Fix wait_for_state() to wait for a PG to get into a state

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32385 into master

* refs/pull/32385/head:
mgr/cephadm: check-host on 'host add'

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32712 into master

* refs/pull/32712/head:
cephadm: SO_REUSEADDR when doing bind check

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32711 into master

* refs/pull/32711/head:
qa/standalone/mon/osd-create-pool: fix utf-8 grep LANG

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32641 into master

* refs/pull/32641/head:
cephadm: fix logging defaults

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32700 into master

* refs/pull/32700/head:
rpm: Relax the selinux policy version for centos builds

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32560 into master

* refs/pull/32560/head:
cephadm: show contextual message when port is in use

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32713 into master

* refs/pull/32713/head:
qa/tasks/cephadm: start watching initial daemons before bootstrap
qa/tasks/cephadm: create /etc/ceph if it doesn't exist
qa/tasks/cephadm: fix log whitelist when there is no whitelist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32701 into master

* refs/pull/32701/head:
qa/tasks/ceph_manager: fix ceph-objectstore-tool calls
qa/tasks/ceph_manager: fix admin_socket remote when using cephadm
qa/tasks/ceph_manager: --no-mon-config to ceph-objectstore-tool
qa/tasks/ceph_manager: fix filestore split command
qa/tasks/ceph_manager: fix import line

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/releases/octopus: disable autoscale warnings

Existing pools might have too many/few PGs and produce a warning that
prevents us from getting to HEALTH_OK.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: SO_REUSEADDR when doing bind check

This is what the ceph daemons use; our check should do it too.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: fix ceph-objectstore-tool calls

Pass the correct paths based on whether this is the importing or exporting
OSD.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: fix admin_socket remote when using cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: --no-mon-config to ceph-objectstore-tool

The config is currently fetched at osd.admin, so the keyring is not
found.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: fix filestore split command

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: fix import line

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: start watching initial daemons before bootstrap

This lets us see output while bootstrap is happening.

(Depends on the teuthology change to use journalctl, see
teuthology commit 4fa83040b05b604280789459f095d6f2ad1b0d01.)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: create /etc/ceph if it doesn't exist

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: fix log whitelist when there is no whitelist

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/mimix-x-singleton: upgrading to octopus, not n

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32698 into master

* refs/pull/32698/head:
common: drop the unused mutable_item_history container.
common: drop the unused prune() of {safe,mutable}_item_history.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/standalone/mon/osd-create-pool: fix utf-8 grep LANG

This needs en_US.UTF-8... en_US does not work.

Fixes: https://tracker.ceph.com/issues/43422
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32663 from dzafman/wip-41016

osd: Diagnostic logging for upmap cleaning

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: revert to OrchestratorClientMixin child

self.mgr does not exist--I can't figure out what this 'mgr' reference
is and where it comes from.  So reverting to being a child for the time
being; we can clean this up later.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: initial upgrade logic

- upgrade mgr, mon, osd, mds
- use ok-to-stop

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: return 'cephadm pull' result as JSON

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: upgrade start/stop/pause/resume

No actual upgrading yet.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: track ceph version as well as container in on 'upgrade check'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: upgrade {start,pause,resume,stop}

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix upgrade check

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigKeyService: remove config-key set warning

This has been there for both mimic and nautilus; we can drop it for
octopus.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32298 from dang/wip-dang-zipper-4

Zipper - RGWUser

common: drop the unused mutable_item_history container.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: drop the unused prune() of {safe,mutable}_item_history.

The presence of these never-used methods obscure bug investigation.
For instance, `safe_item_history` is used to record all messenger's
addresses. As we never lock reading from it in `_conn_prefix()`, we
expect they live as long their corresponding messenger instances.
When such an item becomes corrupted, presence of `prune()` imposes
verification the method is never called – this effort could have
been avoided if we hadn't have the dangerous method.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Zipper - RGWUser

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #32624 from alimaredia/wip-s3-tests-python-3

qa/suites: use s3-tests with python3 support

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rpm: Relax the selinux policy version for centos builds

We are currently running into issues when installing the test builds
produced on centos 8 as these might have older selinux-policy-base
version than is available to the rhel 8 nodes.

We still want the builds produced on RHEL 8 to use the proper version
condition so the %{?centos} macro should help us achieve that, here.

Signed-off-by: Boris Ranto <branto@redhat.com>

cephadm: fix logging defaults

- prefix stderr log lines with 'debug '
- send cluster log to stderr (only affects mons)
- default to sending cluser log to stderr only (not to file)

This aligns things with the container-y way, as we do
with rook.  It means that if you want legacy behavior (log files),
you have *2* settings to change:

 log_to_file = true
 mon_cluster_log_to_file = true

We could default this to leave cluster log to file by default
since they aren't that big (and we do have log rotation), but
it's not as clean as it could be...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32650 from votdev/fix_i18n

mgr/dashboard: Fix missing i18n.

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

doc/rbd: documented 'rbd compression hint' config option

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: ensure mirror status updater cannot attempt empty op

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32367 from capri1989/doc-gc-update

doc/radosgw/config-ref: paragraph to explain the gc settings

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #32049 from mgfritch/wip-cd-timeout

cephadm: add ability to specify a timeout

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32691 from rosinL/wip-txpkt

msg/async/dpdk: Fix infinite loop when sending packets

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async/dpdk: Fix infinite loop when sending packets

tx_pkt_burst loops until the next pointer is null, set next pointer of
the last segment nullptr to fix this issue.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>

cephadm: remove -crash container before starting it

This matches up with what the @.service unit does.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32114 into master

* refs/pull/32114/head:
doc: split index.rst of dev guide
doc: tweak few section titles in dev guide
doc: don't display contents twice in dev guide
doc: add guide for running tests with teuthology

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: support compression allocation hints to the OSD

A new "rbd_compression_hint" configuration option can be applied
globally, at the pool level, or to individual images to send
the associated compression allocation hint to the OSD on write
operations.

Fixes: https://tracker.ceph.com/issues/42097
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cephadm: stop the crash container the nice way

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: no need for podman rm when passing --rm to run

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-crash: exit code 0 on SIGINT, SIGTERM

This makes the systemd unit not go into a failure state when you kill the
cephadm unit's podman container.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32513 from IlsooByun/loop_rgw_none

rgw: move forward marker even in case of many rgw.none indexes

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #31878 from yuvalif/wip-yuval-tag-filter

rgw/pubsub: notification filtering by object tags

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32596 from linuxbox2/wip-rgwfile-pfront

rgw_file: avoid string::front() on empty path

Merge pull request #32453 from majianpeng/kv-rocksdbstore-fix-option-work

kv/RocksDBStore: make option: compaction_threads/disableWAL/flusher_t…

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #32443 from tchaikov/wip-btree

include/cpp-btree: cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/cephadm: fix section name for mon options in ceph.conf

If name is not explicitly specified, we were putting the options in
the [mon.] section instead of [mon.$name].

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: behave with empty weight vector

should not dereference an empty vector.

Fixes: https://tracker.ceph.com/issues/43404
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: no need to link libglobal and libblkid for testing crush

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crush: no need to use libglobal for testing crush

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #32670 into master

* refs/pull/32670/head:
doc: remove prod cluster examples from hardware recs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

rbd-mirror: pool watcher no longer needs to query mirror uuid

This is periodically polled now via the RemotePoolPoller and it detects
changes to the mirror uuid.

Fixes: https://tracker.ceph.com/issues/43537
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: removed peer ping from mirror status updater

This ping is now handled by the remote pool poller class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31923 from tspmelo/wip-vertical-navigation

mgr/dashboard: Add vertical menu

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

rbd-mirror: pass remote pool meta through to bootstrap

The snapshot-based replaying logic will need the remote's
mirror peer uuid to find its snapshots in the remote images.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: pool replayer should instantiate the remote pool poller

Let the poller pull the metadata from the remote before advancing to
the leader watcher initialization. If the remote metadata changes
during runtime, stop the pool replayer so that it can be
re-initialized.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: periodic remote pool metadata poller

The mirror uuid and mirror peer uuid should be periodically retrieved
from the remote peer. The peer ping logic can also be moved to this
helper class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm: check-host on 'host add'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32654 from liewegas/fix-43273

mgr/rook: provide full path for devices names in inventory

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32372 from sebastian-philipp/cephadm-progress

mgr/cephadm: Add progress to update_mgr()

Reviewed-by: Joshua Schmid <jschmid@suse.de>

test/kv_store_bench: Fix double free error

kvs->aio_xxx is asynchronous function, cb_args will be used and deleted
in aio_callbcak_timed, delete cb_agrs in test_teuthology_aio will cause
double free error.

Signed-off-by: Xuqiang Chen <chenxuqiang3@hisilicon.com>
Signed-off-by: luo rixin <luorixin@huawei.com>

doc: remove prod cluster examples from hardware recs

This PR removes the now-very-old (2012) production cluster
examples from the hardware recommendations page.

Sage requested this change in IRC on 14 Jan 2020. There is no issue link.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #32497 from zhengchengyao/snap_exists

rbd: add snap_exists method API

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: Diagnostic logging for upmap cleaning

Fixes: https://tracker.ceph.com/issues/41016
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #32541 into master

* refs/pull/32541/head:
mds: add comment in feature_bitset_t::empty

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rbd-mirror: do not run pool replayers against TX-only peers

Now that we track RX and TX states of peers, any peers that are
listed as TX only should be excluded from image replaying.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32619 into master

* refs/pull/32619/head:
kv/RocksDBStore: break out of compaction thread early on shutdown
kv/RocksDBStore: debug async compaction

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32569 from votdev/feature_toggle_output

mgr/dashboard: Cleanup feature toggle status output

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #32635 from Devp00l/wip-43594

mgr/dashbaord: Fix E2E pools page failure

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/rook: provide full path for devices names in inventory

This matches the cephadm behavior.

Fixes: https://tracker.ceph.com/issues/43273
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32411 into master

* refs/pull/32411/head:
qa/tasks/ceph_manager: enable ceph-objectstore-tool via cephadm
qa/tasks/cephadm: support crush_tunables config option

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/ceph_manager: enable ceph-objectstore-tool via cephadm

- drop support for keyvaluestore
- leave a few paths non-cephadm specific (filestore, upgrade workaround)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32643 into master

* refs/pull/32643/head:
doc/bootstrap: fixed default --keyring target

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32567 from trociny/wip-mirror-snap-promote

librbd: fix some edge cases for snapshot mirror mode promote

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Fix missing i18n.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashbaord: Fix E2E pools page failure

The problem was that pg autoscaling is set to "on" by default. Therefore
the pg_num field is hidden, which caused the error. Now the test sets
the pg autoscaling to "off" revealing the pg_num field to make the tests
pass.

Fixes: https://tracker.ceph.com/issues/43594
Fixes: https://tracker.ceph.com/issues/43102
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: PageHelper.selectOption

This helper method selects an option inside a select element.
It will also expect that the option was set.

Fixes: https://tracker.ceph.com/issues/43594
Signed-off-by: Stephan Müller <smueller@suse.com>