test: Disable self-test of diskprediction_cloud since it isn't loaded

See qa/packages/packages.yaml

Signed-off-by: David Zafman <dzafman@redhat.com>

mon mgr osd: Add dump_osd_times interface for python

The dump_osd_times interface is here for future use of a manager module.
The osd_stats and pg_dump python interfaces don't return network ping
information.

User facing ping time information is formatted 3 decimal places
Use dump_float() (low overhead) for network ping times
   used by python dump_osd_times and dump commands like
   "ceph --formet=json pg dump osds" which for now yields output like
   >>>> "1min": 0.61599999999999999,

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr/progress: Add pg_ready interface for python to get pgmap_ready state

Fixes: https://tracker.ceph.com/issues/43557
Signed-off-by: David Zafman <dzafman@redhat.com>

mgr: Add pg_stats and pool_stats direct access

More efficient to get just what you need out of the PGMap

mgr/progress: Use pg_stats instead of pg_dump
mgr/diskprediction_cloud: Use pg_stats instead of pg_dump
mgr/balancer: Use pg_stats and pool_stats instead of pg_dump

Fixes: https://tracker.ceph.com/issues/43556
Signed-off-by: David Zafman <dzafman@redhat.com>

mgr mon: Expose network ping information except for python pg_dump

Caused by: ffb8a2a904426c8a3d6fc2fd70266418a98623c5

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #33418 into master

* refs/pull/33418/head:
mon: stash newer map on bootstrap when addr doesn't match

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33285 into master

* refs/pull/33285/head:
src/common: fix help text for echo option of cephfs-shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33291 into master

* refs/pull/33291/head:
mds: fix use-after-free in Migrater

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #33500 from tchaikov/wip-cmake-cleanup

cmake: cleanups

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge PR #33484 into master

* refs/pull/33484/head:
mgr/cephadm: catch exceptions when scraping ceph-volume inventory

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33481 from tchaikov/wip-crimson-cflags

cmake: move crimson-crush to crimson/

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33429 from votdev/add_always_on_column

mgr/dashboard: Add 'Always-on' column to mgr module list

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

cmake: reduce the indent level

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use loop for applying new policies

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33485 into master

* refs/pull/33485/head:
qa/workunits/cephadm/test_adoption: run as root

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33467 into master

* refs/pull/33467/head:
doc/orchestrator: Substitute `host` for `node`
mgr/orchestrator: Substitute `node` with `host`, globally

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33328 into master

* refs/pull/33328/head:
osd/OSD: Log slow ops/types to cluster logs

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33380 into master

* refs/pull/33380/head:
common/admin_socket: Added printing of error message constucted in handlers.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/orchestrator: Substitute `host` for `node`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Substitute `node` with `host`, globally

Right now, there is a mix of `node` and `host`. Unify this to `host`

* mgr/rook is special, as Kubernetes nativaly uses "node"

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/workunits/cephadm/test_adoption: run as root

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33448 from dillaman/wip-librbd-group-snaps

librbd: fix broken group snapshot handling

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33335 from dillaman/wip-44043

rbd-mirror: apply image state during snapshot replay

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33451 from dillaman/wip-rbd-support-tasks

pybind/mgr/rbd_support: wait for latest OSD map prior to handling commands

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #31963 from yison/rwl-image-writeback-cache-seq2

rbd/cache: Replicated Write Log core codes part 2

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33461 into master

* refs/pull/33461/head:
cephadm: Simplify is_available
cephadm: Improve type signature of is_available
cephadm: Fix type errors found by mypy
cephadm: Enable and start chrony using any packager
cephadm: Add Packager for Zypper (openSUSE / SLES)
cephadm: Normalize mypy annotations

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33486 into master

* refs/pull/33486/head:
qa/suites/rados/cephadm/smoke: run a few orch commands
mgr/orch: fix 'host ls'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33475 into master

* refs/pull/33475/head:
qa/suites/rados/cephadm/upgrade: change start version

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33091 into master

* refs/pull/33091/head:
qa/suites/rados: disable device scraping
qa/standalone/ceph-helpers: disable device monitoring
qa/tasks/ceph.py: add pre-mgr-commands option for ceph task
mgr/devicehealth: set default monitoring to 'on'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33235 into master

* refs/pull/33235/head:
os/bluestore/BlueFS: fixed printing stats

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33362 into master

* refs/pull/33362/head:
common/config :remove duplicaed code.
common/config: remove useless code.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33404 into master

* refs/pull/33404/head:
selinux: Allow ceph to setsched

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/cephadm/smoke: run a few orch commands

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: fix 'host ls'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33472 into master

* refs/pull/33472/head:
cephadm: allow users to provide their dashboard cert during bootstrap

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm/upgrade: change start version

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33469 into master

* refs/pull/33469/head:
mgr/cephadm: fix upgrade ok-to-stop condition check

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: catch exceptions when scraping ceph-volume inventory

This matches what we do with the background 'cephadm ls'.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33473 from liewegas/remove-memstore-test

test/CMakeLists: disable memstore make check test

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: move crimson-crush to crimson/

so it's able to use crimson::cflags, this interface target could be
defined after crush/CMakeLists.txt is included by upper directory, so
just move it into crimson/CMakeLists.txt

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33106 from Yan-waller/wip-walle-fixbackfillundersized

osd/PG: restart peering for undersized PG on any down stray peer coming back

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #33401 from liewegas/wip-octopus-notes

doc/release/octopus: note about upgrade times

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33476 into master

* refs/pull/33476/head:
mgr/volumes: fix placement default value

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/volumes: fix placement default value

Fixes: https://tracker.ceph.com/issues/44244
Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: Simplify is_available

There is no need to return a closure to call
from is_available, it can just do the work
directly.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

cephadm: Improve type signature of is_available

Specify the function signature expected by `is_available`
using the mypy Callable type.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

cephadm: Fix type errors found by mypy

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

Merge pull request #33412 from idryomov/wip-krbd-fsx-build-deps

qa: rbd_workunit_suites_fsx: install build dependencies

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test/CMakeLists: disable memstore make check test

This is flaky, with frequent failures like

[ RUN      ] ObjectStore/StoreTest.CompressionTest/0
2020-02-21T19:06:13.940+0000 7f5cdErrors while running CTest
Build step 'Execute shell' marked build as failure

Tracked by https://tracker.ceph.com/issues/44243

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33468 into master

* refs/pull/33468/head:
debian: fix ceph-mgr-modules-core files

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33457 into master

* refs/pull/33457/head:
qa/workunits/cephadm: separate out test_adoption.sh; fix

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33464 into master

* refs/pull/33464/head:
mgr/BaseMgrModule: drop GIL for is_authorized check
mgr/ActivePyModules: drop GIL to register/unregister clients

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cephadm: allow users to provide their dashboard cert during bootstrap

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge PR #33466 into master

* refs/pull/33466/head:
cmake: exclude unittest_alloc_aging from "all"

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix upgrade ok-to-stop condition check

Broke this in 1072a7e3b5026c1e605dd45967fa555d2e6a0d14

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: only create new mirror image state records for primary mirror snapshots

The consolidation of primary and non-primary mirror snapshot namespaces
resulted in the creation of image state records for non-primary snapshots.
This was resulting in non-primary snapshots being prematurely marked as
complete.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: additional debugging messages for snapshot-based mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: additional debugging messages for snapshot-based mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33415 into master

* refs/pull/33415/head:
mgr/cephadm: progress for upgrade

Reviewed-by: Michael Fritch <mfritch@suse.com>

debian: fix ceph-mgr-modules-core files

Remove the files from ceph-mgr that are added to ceph-mgr-modules-core.

2020-02-21T17:59:56.031 INFO:teuthology.orchestra.run.smithi060.stdout:dpkg: error processing archive /tmp/apt-dpkg-install-YpwLHV/044-ceph-mgr_15.1.0-1037-ga6b324b-1bionic_amd64.deb (--unpack):
2020-02-21T17:59:56.031 INFO:teuthology.orchestra.run.smithi060.stdout: trying to overwrite '/usr/share/ceph/mgr/alerts/__init__.py', which is also in package ceph-mgr-modules-core 15.1.0-1037-ga6b324b-1bionic

Broken by 589626464d475b4ad96905fbddf11a5782a040f0

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: exclude unittest_alloc_aging from "all"

link it against GTest::Main, which was remove by
96a0439faf551dfae45676072eddd5f37a157289.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: Enable and start chrony using any packager

Add method to enable services to Packager, and call it
from `command_prepare_host`.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

Merge PR #33432 into master

* refs/pull/33432/head:
mgr/orchestrator: functools.partial doesn't work for methods

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33258 from sebastian-philipp/orchestrator-get_hosts-HostSpec

mgr/orchestrator: get_hosts return `HostSpec` instead of `InventoryDevice`

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32552 from Devp00l/wip-42617-new-tree

mgr/dashboard: Reload all CephFS directories

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #32680 from votdev/wip-force-pwd

mgr/dashboard: Enforce password change upon first login

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33193 from cbodley/wip-44068

rgw: data_sync_source_zones only contains 'exporting' zones

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

mgr/BaseMgrModule: drop GIL for is_authorized check

This is making a call all the way to the mon!  We need to drop the GIL.

Signed-off-by: Sage Weil <sage@redhat.com>

test/rbd-mirror: allow test to be run repeatedly

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: snapshot-based image replayer now applies image state

Additionally, the image state will be re-read from the local cluster
if the sync was interrupted.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: state machine to apply any image-state deltas

When replaying mirror snapshots from a remote image, the image state
tracks the current state of the image at the image the snapshot was
created. Therefore, the new state machine will apply any discovered
deltas between the remote image state and the local image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: filter implicit features and non-user snapshots from image state

The implicit features are not controlled by the user so they should not
be mirrored. Similarly, trash snapshots are implicitly created by need
and group snapshots are not supported for mirroring.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/ActivePyModules: drop GIL to register/unregister clients

We need to drop the GIL since we are taking ActivePyModules::lock.

Fixes: df507cde8d71063d5873a42f668156e4c32e86f9
Fixes: https://tracker.ceph.com/issues/44241
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33046 from ivancich/wip-listing-w-namespaces

rgw: fix bug with (un)ordered bucket listing and marker w/ namespace

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #33397 from aclamk/fix-disable-alloc-aging

cmake: Removed unittest_alloc_aging from make check

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33256 from tchaikov/wip-cmake-cares

cmake: should expose ${C-ARES_BINARY_DIR} from c-ares

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: the journal SnapLimitEvent was not properly handled

It was not being properly decoded since it was missing from the
factory and it would output as an "Unknown" event code.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix issues with snapshot-based mirroring image state

There is no need to record the snapshot id within the SnapState structure
since that will duplicate the id. Additionally, the protection status wasn't
being decoded. Also includes some improvements to the debug formatting.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: switch to new metadata retrieval state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper image-meta list state machine

Several locations in librbd re-implement the same loop to collect
all image-meta from the OSD. Consolidate it all to a single state
machine.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cephadm: Add Packager for Zypper (openSUSE / SLES)

Adds basic support for using zypper as the packager.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

Merge pull request #31981 from petrutlucian94/windows

Windows support [part 1]

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: Normalize mypy annotations

Try to use `# type: ...` consistently.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

librbd: fix broken group snapshot handling

After commit ecf71d301f, only the snapshot namespace type and snapshot
name are considered for snapshot ordering. This breaks the expectations
in the group API when searching for a newly created snapshot.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/cephadm: separate out test_adoption.sh; fix

- fix adoption test to handle different legacy fsids
- separate it out into a different script

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33430 from rosinL/wip-testosdscrub

test/TestOSDScrub: fix mktime() error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33444 into master

* refs/pull/33444/head:
mgr/cephadm: no gibberish suffix for some services
mgr/cephadm: prometheus: scrape all mgrs
mgr/cephadm: make prometheus scrape all node-exporters
cephadm: reset-failed on reconfig
cephadm: fix reconfig for monitoring daemons

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #33447 into master

* refs/pull/33447/head:
mgr/cephadm: fix upgrade wait loop

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33435 from tchaikov/wip-viva-la-module

deb,rpm,doc: s/plugin/module/

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33422 from tchaikov/wip-ceph-mgr-essentials

rpm,deb: package always-enabled plugins in a separated package

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33088 from liewegas/wip-cephadm-doc

doc/bootstrap: add mds and rgw steps to bootstrap

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

osd/PeeringState: restart peering on any previous down acting member coming back

One of our customers wants to verify the data safety of Ceph during scaling
the cluster up, and the test case looks like:
- keep checking the status of a speficied pg, who's up is [1, 2, 3]
- add more osds: up [1, 2, 3] -> up [1, 4, 5], acting = [1, 2, 3], backfill_targets = [4, 5],
  pg is remapped
- stop osd.2: up [1, 4, 5], acting = [1, 3], backfill_targets = [4, 5], pg is undersized
- restart osd.2, acting will stay unchanged as 2 belongs to neither current up nor acting set,
  hence leaving the corresponding pg pinning undersized for a long time until all backfill
  targets completes

It does not pose any critical problem -- we'll end up getting that pg back into active + clean,
except that the long live DEGRADED warnings keep bothering our customer who cares about data
safety more than any thing else.

The right way to achieve the above goal is for:

boost::statechart::result PeeringState::Active::react(const MNotifyRec& notevt)

to check whether the newly booted node could be validly chosen for the acting set and
request a new temp mapping. The new temp mapping would then trigger a real interval change
that will get rid of the DEGRADED warning.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

qa/tasks/cephadm: `orch host ls` now returns hostnames

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: NodeAssignment.get_hosts_func now returns List[str]

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: get_hosts return `HostSpec` instead of `InventoryDevice`

This fixes a bad symmetry problem:

previsously:
```python
def add_host(host: HostSpec): ...
def get_hosts() -> List[InventoryDevice]: ...
```

now:
```python
def add_host(host: HostSpec): ...
def get_hosts() -> List[HostSpec]: ...
```

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: functools.partial doesn't work for methods

Use `partialmethod` instead:

```
def decorator_partial(f):
     return partial(f)

def decorator_lambda(f):
     return lambda *args, **kwargs: f(*args, **kwargs)

class C:
     @decorator_partial
     def f(self, arg): pass

     decorator_lambda
     def g(self, arg): pass

C().f(1)
TypeError: f() missing 1 required positional argument: 'arg'

C().g(1)
None
```

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33453 from tchaikov/wip-seastar

seastar: pickup change to add pthread linkage

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33452 from tchaikov/wip-docker-centos8

test: remove Dockerfile for centos7 and add Dockerfile for centos8

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

cmake: drop Threads::Threads linkage from libcrimson

since seastar is now linked against pthread by itself.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup change to add pthread linkage

Signed-off-by: Kefu Chai <kchai@redhat.com>