test/rgw: fixes for test-rgw-common.sh

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #33113 into master

* refs/pull/33113/head:
doc/mgr/telemetry: added device channel details

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33439 into master

* refs/pull/33439/head:
common/options.cc: make rocksdb_delete_range_threshold very high

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33487 into master

* refs/pull/33487/head:
mgr/cephadm: upgrade: fix handling when target container id changes
mgr/cephadm: upgrade: handle stopped daemons

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33496 into master

* refs/pull/33496/head:
mgr/cephadm: combine get_daemons_by_daemon -> get_daemons_by_service
mgr/cephadm: remove apply_mon support
mgr/cephadm: use generics for add_mon
mgr/cephadm: use _apply_service for mgrs
mgr/cephadm: refactor most daemon add methods
mgr/cephadm: refactor _update_service and all apply methods
mgr/cephadm: fix get_unique_name when name in use

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33465 into master

* refs/pull/33465/head:
mgr/cephadm: implement placement by label
mgr/cephadm: fix self.log call in get_unique_name
mgr/cephadm: fix NodePlacement host names
mgr/tests/test_orchestrator: test PlacementSpec parsing
mgr/orch: remove label: prefix when parsing placementspec
mgr/orch: PlacementSpec: add all_hosts property

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33514 into master

* refs/pull/33514/head:
mgr/cephadm: drop mixin parent

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33477 into master

* refs/pull/33477/head:
qa/tasks/mgr/test_orchestrator_cli: fix test_host_ls

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33474 into master

* refs/pull/33474/head:
mgr/orch: make arg hostname, not host

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33418 into master

* refs/pull/33418/head:
mon: stash newer map on bootstrap when addr doesn't match

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33285 into master

* refs/pull/33285/head:
src/common: fix help text for echo option of cephfs-shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33291 into master

* refs/pull/33291/head:
mds: fix use-after-free in Migrater

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/cephadm: combine get_daemons_by_daemon -> get_daemons_by_service

These were the same.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove apply_mon support

Our apply method doesn't support removing mons at this point.  And using
it for adding mons is just an awkward version of 'daemon add'.

Update docs and cephadm.py task accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use generics for add_mon

Adjust apply_mon a bit so that the arg order for _create_mon is still
correct, and so that the name inference is done in the caller.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use _apply_service for mgrs

Note that we are losing some of the special logic about removing standby
mgrs only.  This should be added back *after* we fix up the scheduler
to be more intelligent about choosing hosts that already host daemons,
and make removal pick hosts that aren't selected (by label, or by
scheduler, etc.).

A few bugs to track this:
 https://tracker.ceph.com/issues/44167
 https://tracker.ceph.com/issues/44252 (prefer standby mgrs *and* mdss)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refactor most daemon add methods

New _add_daemons() that shares what we can with _apply_service().

Unlike _apply_service(), the _add_daemon() path requires the user to
explicitly place daemons on hosts--no automatic placement is performed.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refactor _update_service and all apply methods

- Use a common _apply_service() helper
- Consolidate _add_new_daemon logic into _apply_service
- Do the NodeAssignment all in one place

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix get_unique_name when name in use

If we are not adding a uniqueifying suffix, we need to error out instead
of looping.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33500 from tchaikov/wip-cmake-cleanup

cmake: cleanups

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/cephadm: drop mixin parent

This was added a while back to let us block on completions, but that
code has since been removed.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: upgrade: fix handling when target container id changes

If the same target_name pulls a new hash partway through the upgrade
(e.g., because the tag was updated in the registry), we restart the
upgrade.

Fix the code so that we update target_id instead of (unused) image_id.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: upgrade: handle stopped daemons

A stopped daemon should have the correct target_name, and we should ensure
that the host has an up-to-date image, so that when it does start it
comes up with the new image.  If it has an old image name, we should
redeploy as per usual.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33484 into master

* refs/pull/33484/head:
mgr/cephadm: catch exceptions when scraping ceph-volume inventory

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: implement placement by label

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix self.log call in get_unique_name

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix NodePlacement host names

This broke back in 82e4f1e0f1332f7d4f27dd0f83d394a6f14c6bd2, I think.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/tests/test_orchestrator: test PlacementSpec parsing

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: remove label: prefix when parsing placementspec

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: PlacementSpec: add all_hosts property

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33481 from tchaikov/wip-crimson-cflags

cmake: move crimson-crush to crimson/

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33429 from votdev/add_always_on_column

mgr/dashboard: Add 'Always-on' column to mgr module list

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

cmake: reduce the indent level

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use loop for applying new policies

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orch: make arg hostname, not host

For optional args, it can't be --host because this (currently) conflicts
with a core ceph option.  We change all of them for consistency.

This fixes 'ceph orch ps --host foo'.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33485 into master

* refs/pull/33485/head:
qa/workunits/cephadm/test_adoption: run as root

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33467 into master

* refs/pull/33467/head:
doc/orchestrator: Substitute `host` for `node`
mgr/orchestrator: Substitute `node` with `host`, globally

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33328 into master

* refs/pull/33328/head:
osd/OSD: Log slow ops/types to cluster logs

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33380 into master

* refs/pull/33380/head:
common/admin_socket: Added printing of error message constucted in handlers.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/orchestrator: Substitute `host` for `node`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Substitute `node` with `host`, globally

Right now, there is a mix of `node` and `host`. Unify this to `host`

* mgr/rook is special, as Kubernetes nativaly uses "node"

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/workunits/cephadm/test_adoption: run as root

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33448 from dillaman/wip-librbd-group-snaps

librbd: fix broken group snapshot handling

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33335 from dillaman/wip-44043

rbd-mirror: apply image state during snapshot replay

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33451 from dillaman/wip-rbd-support-tasks

pybind/mgr/rbd_support: wait for latest OSD map prior to handling commands

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #31963 from yison/rwl-image-writeback-cache-seq2

rbd/cache: Replicated Write Log core codes part 2

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33461 into master

* refs/pull/33461/head:
cephadm: Simplify is_available
cephadm: Improve type signature of is_available
cephadm: Fix type errors found by mypy
cephadm: Enable and start chrony using any packager
cephadm: Add Packager for Zypper (openSUSE / SLES)
cephadm: Normalize mypy annotations

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33486 into master

* refs/pull/33486/head:
qa/suites/rados/cephadm/smoke: run a few orch commands
mgr/orch: fix 'host ls'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33475 into master

* refs/pull/33475/head:
qa/suites/rados/cephadm/upgrade: change start version

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33091 into master

* refs/pull/33091/head:
qa/suites/rados: disable device scraping
qa/standalone/ceph-helpers: disable device monitoring
qa/tasks/ceph.py: add pre-mgr-commands option for ceph task
mgr/devicehealth: set default monitoring to 'on'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33235 into master

* refs/pull/33235/head:
os/bluestore/BlueFS: fixed printing stats

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33362 into master

* refs/pull/33362/head:
common/config :remove duplicaed code.
common/config: remove useless code.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33404 into master

* refs/pull/33404/head:
selinux: Allow ceph to setsched

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/cephadm/smoke: run a few orch commands

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: fix 'host ls'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33472 into master

* refs/pull/33472/head:
cephadm: allow users to provide their dashboard cert during bootstrap

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm/upgrade: change start version

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33469 into master

* refs/pull/33469/head:
mgr/cephadm: fix upgrade ok-to-stop condition check

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: catch exceptions when scraping ceph-volume inventory

This matches what we do with the background 'cephadm ls'.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33473 from liewegas/remove-memstore-test

test/CMakeLists: disable memstore make check test

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: move crimson-crush to crimson/

so it's able to use crimson::cflags, this interface target could be
defined after crush/CMakeLists.txt is included by upper directory, so
just move it into crimson/CMakeLists.txt

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33106 from Yan-waller/wip-walle-fixbackfillundersized

osd/PG: restart peering for undersized PG on any down stray peer coming back

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Samuel Just <sjust@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: fix test_host_ls

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33401 from liewegas/wip-octopus-notes

doc/release/octopus: note about upgrade times

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33476 into master

* refs/pull/33476/head:
mgr/volumes: fix placement default value

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/volumes: fix placement default value

Fixes: https://tracker.ceph.com/issues/44244
Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: Simplify is_available

There is no need to return a closure to call
from is_available, it can just do the work
directly.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

cephadm: Improve type signature of is_available

Specify the function signature expected by `is_available`
using the mypy Callable type.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

cephadm: Fix type errors found by mypy

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

Merge pull request #33412 from idryomov/wip-krbd-fsx-build-deps

qa: rbd_workunit_suites_fsx: install build dependencies

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test/CMakeLists: disable memstore make check test

This is flaky, with frequent failures like

[ RUN      ] ObjectStore/StoreTest.CompressionTest/0
2020-02-21T19:06:13.940+0000 7f5cdErrors while running CTest
Build step 'Execute shell' marked build as failure

Tracked by https://tracker.ceph.com/issues/44243

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33468 into master

* refs/pull/33468/head:
debian: fix ceph-mgr-modules-core files

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33457 into master

* refs/pull/33457/head:
qa/workunits/cephadm: separate out test_adoption.sh; fix

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33464 into master

* refs/pull/33464/head:
mgr/BaseMgrModule: drop GIL for is_authorized check
mgr/ActivePyModules: drop GIL to register/unregister clients

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cephadm: allow users to provide their dashboard cert during bootstrap

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge PR #33466 into master

* refs/pull/33466/head:
cmake: exclude unittest_alloc_aging from "all"

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix upgrade ok-to-stop condition check

Broke this in 1072a7e3b5026c1e605dd45967fa555d2e6a0d14

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: only create new mirror image state records for primary mirror snapshots

The consolidation of primary and non-primary mirror snapshot namespaces
resulted in the creation of image state records for non-primary snapshots.
This was resulting in non-primary snapshots being prematurely marked as
complete.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: additional debugging messages for snapshot-based mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: additional debugging messages for snapshot-based mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33415 into master

* refs/pull/33415/head:
mgr/cephadm: progress for upgrade

Reviewed-by: Michael Fritch <mfritch@suse.com>

debian: fix ceph-mgr-modules-core files

Remove the files from ceph-mgr that are added to ceph-mgr-modules-core.

2020-02-21T17:59:56.031 INFO:teuthology.orchestra.run.smithi060.stdout:dpkg: error processing archive /tmp/apt-dpkg-install-YpwLHV/044-ceph-mgr_15.1.0-1037-ga6b324b-1bionic_amd64.deb (--unpack):
2020-02-21T17:59:56.031 INFO:teuthology.orchestra.run.smithi060.stdout: trying to overwrite '/usr/share/ceph/mgr/alerts/__init__.py', which is also in package ceph-mgr-modules-core 15.1.0-1037-ga6b324b-1bionic

Broken by 589626464d475b4ad96905fbddf11a5782a040f0

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: exclude unittest_alloc_aging from "all"

link it against GTest::Main, which was remove by
96a0439faf551dfae45676072eddd5f37a157289.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: Enable and start chrony using any packager

Add method to enable services to Packager, and call it
from `command_prepare_host`.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

Merge PR #33432 into master

* refs/pull/33432/head:
mgr/orchestrator: functools.partial doesn't work for methods

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33258 from sebastian-philipp/orchestrator-get_hosts-HostSpec

mgr/orchestrator: get_hosts return `HostSpec` instead of `InventoryDevice`

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32552 from Devp00l/wip-42617-new-tree

mgr/dashboard: Reload all CephFS directories

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #32680 from votdev/wip-force-pwd

mgr/dashboard: Enforce password change upon first login

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33193 from cbodley/wip-44068

rgw: data_sync_source_zones only contains 'exporting' zones

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

mgr/BaseMgrModule: drop GIL for is_authorized check

This is making a call all the way to the mon!  We need to drop the GIL.

Signed-off-by: Sage Weil <sage@redhat.com>

test/rbd-mirror: allow test to be run repeatedly

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: snapshot-based image replayer now applies image state

Additionally, the image state will be re-read from the local cluster
if the sync was interrupted.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: state machine to apply any image-state deltas

When replaying mirror snapshots from a remote image, the image state
tracks the current state of the image at the image the snapshot was
created. Therefore, the new state machine will apply any discovered
deltas between the remote image state and the local image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: filter implicit features and non-user snapshots from image state

The implicit features are not controlled by the user so they should not
be mirrored. Similarly, trash snapshots are implicitly created by need
and group snapshots are not supported for mirroring.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/ActivePyModules: drop GIL to register/unregister clients

We need to drop the GIL since we are taking ActivePyModules::lock.

Fixes: df507cde8d71063d5873a42f668156e4c32e86f9
Fixes: https://tracker.ceph.com/issues/44241
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33046 from ivancich/wip-listing-w-namespaces

rgw: fix bug with (un)ordered bucket listing and marker w/ namespace

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #33397 from aclamk/fix-disable-alloc-aging

cmake: Removed unittest_alloc_aging from make check

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33256 from tchaikov/wip-cmake-cares

cmake: should expose ${C-ARES_BINARY_DIR} from c-ares

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: the journal SnapLimitEvent was not properly handled

It was not being properly decoded since it was missing from the
factory and it would output as an "Unknown" event code.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix issues with snapshot-based mirroring image state

There is no need to record the snapshot id within the SnapState structure
since that will duplicate the id. Additionally, the protection status wasn't
being decoded. Also includes some improvements to the debug formatting.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>