rgw: increase log level for same or older period pull msg

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #33440 from dillaman/wip-44103

rbd-mirror: provide initial snapshot replay status

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #33497 into master

* refs/pull/33497/head:
qa/suites/rados/singleton/all/thrash-eio: whitelist 'slow request'
qa: Whitelist 'slow request' within a bunch of tests

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33138 into master

* refs/pull/33138/head:
common/TextTable: only pad between columns
mgr/status: align with ceph table style
mgr/osd_perf_query: make table match ceph style
mgr: adjust tables to have 2 space column separation
common/TextTable: default to 2 spaces separating columns

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33413 into master

* refs/pull/33413/head:
test: verify purge queue w/ large number of subvolumes
test: pass timeout argument to mount::wait_for_dir_empty()
mgr/volumes: access volume in lockless mode when fetching async job

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common/TextTable: only pad between columns

No need to pad on the right-hand side if there are no additional columns
that follow.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33113 into master

* refs/pull/33113/head:
doc/mgr/telemetry: added device channel details

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33439 into master

* refs/pull/33439/head:
common/options.cc: make rocksdb_delete_range_threshold very high

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd-mirror: implement basic status feedback for snapshot mirroring

The feedback includes the newest remote mirror snapshot timestamp,
the newest completely synced local mirror snapshot timestamp,
and optionally the in-progress sync snapshot timestamp and
percent complete.

Fixes: https://tracker.ceph.com/issues/44103
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved local to remote snapshot lookup to common function

This will be needed by the status formatter to lookup the remote
snapshot timestamp since the associated local snapshot timestamp
will be the time the snapshot was created on the local side.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: expand lock scope for snapshot replayer

The status callback will need access to the current state, so the
lock should be held while the snapshot scan is in-progress.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: periodically update sync position for snapshot-based images

This will allow the sync to restart at the last known successfully copied
object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33487 into master

* refs/pull/33487/head:
mgr/cephadm: upgrade: fix handling when target container id changes
mgr/cephadm: upgrade: handle stopped daemons

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33496 into master

* refs/pull/33496/head:
mgr/cephadm: combine get_daemons_by_daemon -> get_daemons_by_service
mgr/cephadm: remove apply_mon support
mgr/cephadm: use generics for add_mon
mgr/cephadm: use _apply_service for mgrs
mgr/cephadm: refactor most daemon add methods
mgr/cephadm: refactor _update_service and all apply methods
mgr/cephadm: fix get_unique_name when name in use

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33465 into master

* refs/pull/33465/head:
mgr/cephadm: implement placement by label
mgr/cephadm: fix self.log call in get_unique_name
mgr/cephadm: fix NodePlacement host names
mgr/tests/test_orchestrator: test PlacementSpec parsing
mgr/orch: remove label: prefix when parsing placementspec
mgr/orch: PlacementSpec: add all_hosts property

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33514 into master

* refs/pull/33514/head:
mgr/cephadm: drop mixin parent

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33477 into master

* refs/pull/33477/head:
qa/tasks/mgr/test_orchestrator_cli: fix test_host_ls

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33474 into master

* refs/pull/33474/head:
mgr/orch: make arg hostname, not host

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33418 into master

* refs/pull/33418/head:
mon: stash newer map on bootstrap when addr doesn't match

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33285 into master

* refs/pull/33285/head:
src/common: fix help text for echo option of cephfs-shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33291 into master

* refs/pull/33291/head:
mds: fix use-after-free in Migrater

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/cephadm: combine get_daemons_by_daemon -> get_daemons_by_service

These were the same.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove apply_mon support

Our apply method doesn't support removing mons at this point.  And using
it for adding mons is just an awkward version of 'daemon add'.

Update docs and cephadm.py task accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use generics for add_mon

Adjust apply_mon a bit so that the arg order for _create_mon is still
correct, and so that the name inference is done in the caller.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use _apply_service for mgrs

Note that we are losing some of the special logic about removing standby
mgrs only.  This should be added back *after* we fix up the scheduler
to be more intelligent about choosing hosts that already host daemons,
and make removal pick hosts that aren't selected (by label, or by
scheduler, etc.).

A few bugs to track this:
 https://tracker.ceph.com/issues/44167
 https://tracker.ceph.com/issues/44252 (prefer standby mgrs *and* mdss)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refactor most daemon add methods

New _add_daemons() that shares what we can with _apply_service().

Unlike _apply_service(), the _add_daemon() path requires the user to
explicitly place daemons on hosts--no automatic placement is performed.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refactor _update_service and all apply methods

- Use a common _apply_service() helper
- Consolidate _add_new_daemon logic into _apply_service
- Do the NodeAssignment all in one place

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix get_unique_name when name in use

If we are not adding a uniqueifying suffix, we need to error out instead
of looping.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33500 from tchaikov/wip-cmake-cleanup

cmake: cleanups

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/cephadm: drop mixin parent

This was added a while back to let us block on completions, but that
code has since been removed.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: upgrade: fix handling when target container id changes

If the same target_name pulls a new hash partway through the upgrade
(e.g., because the tag was updated in the registry), we restart the
upgrade.

Fix the code so that we update target_id instead of (unused) image_id.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: upgrade: handle stopped daemons

A stopped daemon should have the correct target_name, and we should ensure
that the host has an up-to-date image, so that when it does start it
comes up with the new image.  If it has an old image name, we should
redeploy as per usual.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton/all/thrash-eio: whitelist 'slow request'

Signed-off-by: Sage Weil <sage@redhat.com>

qa: Whitelist 'slow request' within a bunch of tests

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge PR #33484 into master

* refs/pull/33484/head:
mgr/cephadm: catch exceptions when scraping ceph-volume inventory

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: implement placement by label

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix self.log call in get_unique_name

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix NodePlacement host names

This broke back in 82e4f1e0f1332f7d4f27dd0f83d394a6f14c6bd2, I think.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/tests/test_orchestrator: test PlacementSpec parsing

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: remove label: prefix when parsing placementspec

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: PlacementSpec: add all_hosts property

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33481 from tchaikov/wip-crimson-cflags

cmake: move crimson-crush to crimson/

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test: verify purge queue w/ large number of subvolumes

Fixes: http://tracker.ceph.com/issues/44207
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: pass timeout argument to mount::wait_for_dir_empty()

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: access volume in lockless mode when fetching async job

Saw a deadlock when deleting lot of subvolumes -- purge threads were
stuck in accessing global lock for volume access. This can happen
when there is a concurrent remove (which renames and signals the
purge threads) and a purge thread is just about to scan the trash
directory for entries.

For the fix, purge threads fetches entries by accessing the volume
in lockless mode. This is safe from functionality point-of-view as
the rename and directory scan is correctly handled by the filesystem.
Worst case the purge thread would pick up the trash entry on next
scan, never leaving a stale trash entry.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #33429 from votdev/add_always_on_column

mgr/dashboard: Add 'Always-on' column to mgr module list

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

cmake: reduce the indent level

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use loop for applying new policies

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orch: make arg hostname, not host

For optional args, it can't be --host because this (currently) conflicts
with a core ceph option.  We change all of them for consistency.

This fixes 'ceph orch ps --host foo'.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/status: align with ceph table style

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/osd_perf_query: make table match ceph style

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: adjust tables to have 2 space column separation

Signed-off-by: Sage Weil <sage@redhat.com>

common/TextTable: default to 2 spaces separating columns

This is what other projects and libraries default to, and it is more
legible.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33485 into master

* refs/pull/33485/head:
qa/workunits/cephadm/test_adoption: run as root

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33467 into master

* refs/pull/33467/head:
doc/orchestrator: Substitute `host` for `node`
mgr/orchestrator: Substitute `node` with `host`, globally

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33328 into master

* refs/pull/33328/head:
osd/OSD: Log slow ops/types to cluster logs

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33380 into master

* refs/pull/33380/head:
common/admin_socket: Added printing of error message constucted in handlers.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/orchestrator: Substitute `host` for `node`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Substitute `node` with `host`, globally

Right now, there is a mix of `node` and `host`. Unify this to `host`

* mgr/rook is special, as Kubernetes nativaly uses "node"

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/workunits/cephadm/test_adoption: run as root

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33448 from dillaman/wip-librbd-group-snaps

librbd: fix broken group snapshot handling

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33335 from dillaman/wip-44043

rbd-mirror: apply image state during snapshot replay

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33451 from dillaman/wip-rbd-support-tasks

pybind/mgr/rbd_support: wait for latest OSD map prior to handling commands

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #31963 from yison/rwl-image-writeback-cache-seq2

rbd/cache: Replicated Write Log core codes part 2

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33461 into master

* refs/pull/33461/head:
cephadm: Simplify is_available
cephadm: Improve type signature of is_available
cephadm: Fix type errors found by mypy
cephadm: Enable and start chrony using any packager
cephadm: Add Packager for Zypper (openSUSE / SLES)
cephadm: Normalize mypy annotations

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33486 into master

* refs/pull/33486/head:
qa/suites/rados/cephadm/smoke: run a few orch commands
mgr/orch: fix 'host ls'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33475 into master

* refs/pull/33475/head:
qa/suites/rados/cephadm/upgrade: change start version

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33091 into master

* refs/pull/33091/head:
qa/suites/rados: disable device scraping
qa/standalone/ceph-helpers: disable device monitoring
qa/tasks/ceph.py: add pre-mgr-commands option for ceph task
mgr/devicehealth: set default monitoring to 'on'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33235 into master

* refs/pull/33235/head:
os/bluestore/BlueFS: fixed printing stats

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33362 into master

* refs/pull/33362/head:
common/config :remove duplicaed code.
common/config: remove useless code.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33404 into master

* refs/pull/33404/head:
selinux: Allow ceph to setsched

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/cephadm/smoke: run a few orch commands

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: fix 'host ls'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33472 into master

* refs/pull/33472/head:
cephadm: allow users to provide their dashboard cert during bootstrap

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm/upgrade: change start version

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33469 into master

* refs/pull/33469/head:
mgr/cephadm: fix upgrade ok-to-stop condition check

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: catch exceptions when scraping ceph-volume inventory

This matches what we do with the background 'cephadm ls'.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33473 from liewegas/remove-memstore-test

test/CMakeLists: disable memstore make check test

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: move crimson-crush to crimson/

so it's able to use crimson::cflags, this interface target could be
defined after crush/CMakeLists.txt is included by upper directory, so
just move it into crimson/CMakeLists.txt

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33106 from Yan-waller/wip-walle-fixbackfillundersized

osd/PG: restart peering for undersized PG on any down stray peer coming back

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Samuel Just <sjust@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: fix test_host_ls

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33401 from liewegas/wip-octopus-notes

doc/release/octopus: note about upgrade times

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33476 into master

* refs/pull/33476/head:
mgr/volumes: fix placement default value

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/volumes: fix placement default value

Fixes: https://tracker.ceph.com/issues/44244
Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: Simplify is_available

There is no need to return a closure to call
from is_available, it can just do the work
directly.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

cephadm: Improve type signature of is_available

Specify the function signature expected by `is_available`
using the mypy Callable type.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

cephadm: Fix type errors found by mypy

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

Merge pull request #33412 from idryomov/wip-krbd-fsx-build-deps

qa: rbd_workunit_suites_fsx: install build dependencies

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test/CMakeLists: disable memstore make check test

This is flaky, with frequent failures like

[ RUN      ] ObjectStore/StoreTest.CompressionTest/0
2020-02-21T19:06:13.940+0000 7f5cdErrors while running CTest
Build step 'Execute shell' marked build as failure

Tracked by https://tracker.ceph.com/issues/44243

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33468 into master

* refs/pull/33468/head:
debian: fix ceph-mgr-modules-core files

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33457 into master

* refs/pull/33457/head:
qa/workunits/cephadm: separate out test_adoption.sh; fix

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33464 into master

* refs/pull/33464/head:
mgr/BaseMgrModule: drop GIL for is_authorized check
mgr/ActivePyModules: drop GIL to register/unregister clients

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cephadm: allow users to provide their dashboard cert during bootstrap

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge PR #33466 into master

* refs/pull/33466/head:
cmake: exclude unittest_alloc_aging from "all"

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix upgrade ok-to-stop condition check

Broke this in 1072a7e3b5026c1e605dd45967fa555d2e6a0d14

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: only create new mirror image state records for primary mirror snapshots

The consolidation of primary and non-primary mirror snapshot namespaces
resulted in the creation of image state records for non-primary snapshots.
This was resulting in non-primary snapshots being prematurely marked as
complete.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: additional debugging messages for snapshot-based mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: additional debugging messages for snapshot-based mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33415 into master

* refs/pull/33415/head:
mgr/cephadm: progress for upgrade

Reviewed-by: Michael Fritch <mfritch@suse.com>

debian: fix ceph-mgr-modules-core files

Remove the files from ceph-mgr that are added to ceph-mgr-modules-core.

2020-02-21T17:59:56.031 INFO:teuthology.orchestra.run.smithi060.stdout:dpkg: error processing archive /tmp/apt-dpkg-install-YpwLHV/044-ceph-mgr_15.1.0-1037-ga6b324b-1bionic_amd64.deb (--unpack):
2020-02-21T17:59:56.031 INFO:teuthology.orchestra.run.smithi060.stdout: trying to overwrite '/usr/share/ceph/mgr/alerts/__init__.py', which is also in package ceph-mgr-modules-core 15.1.0-1037-ga6b324b-1bionic

Broken by 589626464d475b4ad96905fbddf11a5782a040f0

Signed-off-by: Sage Weil <sage@redhat.com>