mgr/cephadm: catch exceptions when scraping ceph-volume inventory

This matches what we do with the background 'cephadm ls'.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33401 from liewegas/wip-octopus-notes

doc/release/octopus: note about upgrade times

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33476 into master

* refs/pull/33476/head:
mgr/volumes: fix placement default value

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/volumes: fix placement default value

Fixes: https://tracker.ceph.com/issues/44244
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33412 from idryomov/wip-krbd-fsx-build-deps

qa: rbd_workunit_suites_fsx: install build dependencies

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33468 into master

* refs/pull/33468/head:
debian: fix ceph-mgr-modules-core files

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33457 into master

* refs/pull/33457/head:
qa/workunits/cephadm: separate out test_adoption.sh; fix

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33464 into master

* refs/pull/33464/head:
mgr/BaseMgrModule: drop GIL for is_authorized check
mgr/ActivePyModules: drop GIL to register/unregister clients

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33466 into master

* refs/pull/33466/head:
cmake: exclude unittest_alloc_aging from "all"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33415 into master

* refs/pull/33415/head:
mgr/cephadm: progress for upgrade

Reviewed-by: Michael Fritch <mfritch@suse.com>

debian: fix ceph-mgr-modules-core files

Remove the files from ceph-mgr that are added to ceph-mgr-modules-core.

2020-02-21T17:59:56.031 INFO:teuthology.orchestra.run.smithi060.stdout:dpkg: error processing archive /tmp/apt-dpkg-install-YpwLHV/044-ceph-mgr_15.1.0-1037-ga6b324b-1bionic_amd64.deb (--unpack):
2020-02-21T17:59:56.031 INFO:teuthology.orchestra.run.smithi060.stdout: trying to overwrite '/usr/share/ceph/mgr/alerts/__init__.py', which is also in package ceph-mgr-modules-core 15.1.0-1037-ga6b324b-1bionic

Broken by 589626464d475b4ad96905fbddf11a5782a040f0

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: exclude unittest_alloc_aging from "all"

link it against GTest::Main, which was remove by
96a0439faf551dfae45676072eddd5f37a157289.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33432 into master

* refs/pull/33432/head:
mgr/orchestrator: functools.partial doesn't work for methods

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33258 from sebastian-philipp/orchestrator-get_hosts-HostSpec

mgr/orchestrator: get_hosts return `HostSpec` instead of `InventoryDevice`

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32552 from Devp00l/wip-42617-new-tree

mgr/dashboard: Reload all CephFS directories

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #32680 from votdev/wip-force-pwd

mgr/dashboard: Enforce password change upon first login

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33193 from cbodley/wip-44068

rgw: data_sync_source_zones only contains 'exporting' zones

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

mgr/BaseMgrModule: drop GIL for is_authorized check

This is making a call all the way to the mon!  We need to drop the GIL.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ActivePyModules: drop GIL to register/unregister clients

We need to drop the GIL since we are taking ActivePyModules::lock.

Fixes: df507cde8d71063d5873a42f668156e4c32e86f9
Fixes: https://tracker.ceph.com/issues/44241
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33046 from ivancich/wip-listing-w-namespaces

rgw: fix bug with (un)ordered bucket listing and marker w/ namespace

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #33397 from aclamk/fix-disable-alloc-aging

cmake: Removed unittest_alloc_aging from make check

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33256 from tchaikov/wip-cmake-cares

cmake: should expose ${C-ARES_BINARY_DIR} from c-ares

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #31981 from petrutlucian94/windows

Windows support [part 1]

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephadm: separate out test_adoption.sh; fix

- fix adoption test to handle different legacy fsids
- separate it out into a different script

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33430 from rosinL/wip-testosdscrub

test/TestOSDScrub: fix mktime() error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33444 into master

* refs/pull/33444/head:
mgr/cephadm: no gibberish suffix for some services
mgr/cephadm: prometheus: scrape all mgrs
mgr/cephadm: make prometheus scrape all node-exporters
cephadm: reset-failed on reconfig
cephadm: fix reconfig for monitoring daemons

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #33447 into master

* refs/pull/33447/head:
mgr/cephadm: fix upgrade wait loop

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33435 from tchaikov/wip-viva-la-module

deb,rpm,doc: s/plugin/module/

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33422 from tchaikov/wip-ceph-mgr-essentials

rpm,deb: package always-enabled plugins in a separated package

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33088 from liewegas/wip-cephadm-doc

doc/bootstrap: add mds and rgw steps to bootstrap

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/tasks/cephadm: `orch host ls` now returns hostnames

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: NodeAssignment.get_hosts_func now returns List[str]

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: get_hosts return `HostSpec` instead of `InventoryDevice`

This fixes a bad symmetry problem:

previsously:
```python
def add_host(host: HostSpec): ...
def get_hosts() -> List[InventoryDevice]: ...
```

now:
```python
def add_host(host: HostSpec): ...
def get_hosts() -> List[HostSpec]: ...
```

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: functools.partial doesn't work for methods

Use `partialmethod` instead:

```
def decorator_partial(f):
     return partial(f)

def decorator_lambda(f):
     return lambda *args, **kwargs: f(*args, **kwargs)

class C:
     @decorator_partial
     def f(self, arg): pass

     decorator_lambda
     def g(self, arg): pass

C().f(1)
TypeError: f() missing 1 required positional argument: 'arg'

C().g(1)
None
```

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33453 from tchaikov/wip-seastar

seastar: pickup change to add pthread linkage

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33452 from tchaikov/wip-docker-centos8

test: remove Dockerfile for centos7 and add Dockerfile for centos8

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

cmake: drop Threads::Threads linkage from libcrimson

since seastar is now linked against pthread by itself.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup change to add pthread linkage

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/centos-8: change to adapt to centos8

for more context, see

https://github.com/CentOS/sig-cloud-instance-images/issues/22
https://wiki.centos.org/AdditionalResources/Repositories/CR
https://fedoraproject.org/wiki/EPEL

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/centos-8: add centos8 Dockerfile

and drop centos-7 Dockerfile, as we've moved over centos8 for better
python3 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33441 into master

* refs/pull/33441/head:
mgr/orchestrator: add ability to parse placementspec from strings
mgr/volumes: add arg to fs volume create for mds daemons placement

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33443 into master

* refs/pull/33443/head:
cephadm: prepare-host: do not create Packager unless we need it

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33445 into master

* refs/pull/33445/head:
cephadm: fix adoption safety check

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33446 into master

* refs/pull/33446/head:
cephadm: pull: strip newline from version string

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32882 from trociny/wip-41559-2

rbd: add 'mirror snapshot schedule' commands

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33450 into master

* refs/pull/33450/head:
doc: add CephFS Octopus release notes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33368 from trociny/wip-44161

rbd-mirror: don't expect image map is always initialized

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33411 from trociny/wip-44159

rbd-mirror: improve detection of blacklisted state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33426 into master

* refs/pull/33426/head:
mgr/dashboard: fix tasks.mgr.dashboard.test_rgw suite

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add CephFS Octopus release notes

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/cephadm: fix upgrade wait loop

This was succeeding after 4 tries even though it wasn't ok-to-stop.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix adoption safety check

The 'type' file is only required for *osds*.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: pull: strip newline from version string

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33437 into master

* refs/pull/33437/head:
cephadm: allow multiple get_parm() calls

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: no gibberish suffix for some services

Lots of services only need to appear once per node; no need to create a
unique name.  (In fact, most of these *won't* run more than once per
node because they bind to a standard port.)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: prometheus: scrape all mgrs

Assume that all standby mgrs are running on the same port as the
active.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make prometheus scrape all node-exporters

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: reset-failed on reconfig

Sometimes we had a bad config that reconfig will fix, so we need to
reset-failed before restart.  If there was not failure this is harmless.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix reconfig for monitoring daemons

We shouldn't check for port usage here.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: add ability to parse placementspec from strings

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

mgr/volumes: add arg to fs volume create for mds daemons placement

add placement arg for mds daemons created by fs volume create

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

mgr/cephadm: progress for upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: prepare-host: do not create Packager unless we need it

Otherwise, bootstrap may fail on an unrecognized/unsupported distro even
though all the dependencies are present.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31338 from ivancich/wip-clean-up-ordered-list

rgw: clean up ordered list

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33338 from smithfarm/wip-qa-cephadm-smoke-opensuse

qa/cephadm/smoke: run on opensuse_15.1

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/cephadm: "ls -l" cephadm after downloading it

For better debugging when the download goes awry.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rbd: add 'mirror snapshot schedule' commands

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/rbd_support: rename 'dump' command to 'list'

and remove the old 'list' command -- we can live with one command.

Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/cephadm.py: better check for cephadm download success

When the URL passed to "curl --silent ..." does not exist, the
resulting file will be populated with the string:

    404: Not Found

If that (or something similar) happens, the file size will be
suspiciously low, like < 1000 bytes. Fail the test immediately in this
case.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/cephadm.py: lop off .git more robustly

The line

    rest.replace('.git/', '/')

was added to accommodate weird folks who run teuthology-suite
with an option like this:

    --suite-repo https://github.com/ceph/ceph.git/

but they might just as well give the option like this:

    --suite-repo https://github.com/ceph/ceph.git

Signed-off-by: Nathan Cutler <ncutler@suse.com>

cephadm: allow multiple get_parm() calls

This fixes deployment of monitoring components when the config-json is
passed via stdin.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31994 from pritha-srivastava/wip-rgw-assume-role-web-id-doc

rgw: adding documentation for AssumeRoleWithWebIdentity.

mgr/rbd_support: return 'rbd mirror snapshot schedule status' as json

Signed-off-by: Mykola Golub <mgolub@suse.com>

doc/mgr/dashboard.rst: s/plugin/module/

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm,deb: s/plugin/module/

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm,deb: package always-enabled plugins in a separated package

ceph-mgr depends on it in the sense that these plugins are
a hard dependency of ceph-mgr. like cmake depends on cmake-data,
even if the later cannot function on its own without the former.

the reason why we need a separated package is that mgr modules
are written in python, and should be arch independent. this change
helps the downstream to understand this and help to minimize the
divergence between upstream and downstream, because distros like
debian and its derivatives encourage splitting architecture independent
data into its own package.
see
https://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.html#architecture-independent-data

also, the python related runtime dependencies are also moved into
ceph-mgr-modules-core.

if a python dependency is shared by another ceph-mgr-* package and
ceph-mgr-modules-core. it's only added to ceph-mgr-modules-core for
simplicity and consistency. as the name of dependency might vary
on different distros, so it's tedious and error-prone to repeat
these conditons and checks.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33166 from dillaman/wip-43936

rbd-mirror: initial snapshot replay state machine

Reviewed-by: Mykola Golub <mgolub@suse.com>

ceph.spec.in: update to adapt to el7

make sure ceph-mgr installs on el7

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/TestOSDScrub: fix mktime() error

The var tm tm isn't initialized, when the tm.tm_isdst is a
positive value, mktime(&tm) return -1 result in test failed
in ubuntu 19.10 for aarch64 GLIBC2.30.

Signed-off-by: luo rixin <luorixin@huawei.com>

Merge pull request #33425 from tchaikov/wip-crimson-watcher

crimson/osd: capture watcher when calling its member function

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33423 from tchaikov/wip-doc-search

doc: fix errors in "search" page and use relative address for releases.json

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/cephadm/smoke: run on opensuse_15.1

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/distros: add opensuse_15.2.yaml

We don't have a FOG image for this, yet, and Shaman is not building for
it (yet), either, but we might as well get this file in place now, since
we already know it will be needed at some point.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/distros: drop obsolete opensuse distros

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #33371 from leseb/raw-list

ceph-volume: silence 'ceph-bluestore-tool' failures

Merge pull request #33407 from rhcs-dashboard/coverage-venv-python-version

mgr/dashboard: coverage venv python version same as mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: fix tasks.mgr.dashboard.test_rgw suite

Fixes: https://tracker.ceph.com/issues/44201
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

crimson/osd: capture watcher when calling its member function

otherwise the extracted node handler will be destroyed, and so does the
watcher enclosed in it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: do not use absolute address

use relative address instead, to ease the pain when testing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: do not add suffix for search result links

since sphinx 1.5, SOURCELINK_SUFFIX is always added to the source file,
and search page always tries to add it to the highlighted source result.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: do not try to patch #edit-on-github if not found

we add this element in _templates/page.html, which is only used for the
content pages, not in the "search" page. to avoid the js errors, just
disabled it if it's not found.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33394 into master

* refs/pull/33394/head:
mgr/cephadm: trigger device refresh after osd creation
mgr/orch,cephadm: implement 'device zap'
mgr/cephadm: remove inventory_cache
mgr/cephadm: reimplement 'device ls' in terms of cache
mgr/cephadm: fix invalidate helpers
mgr/cephadm: scrape and cache devices, too
mgr/cephadm: rename DaemonCache methods etc to generalize

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33400 into master

* refs/pull/33400/head:
mgr/progress: 'progress clear' command should clear events in 'ceph -s'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33402 into master

* refs/pull/33402/head:
mgr/pg_autoscaler: fix division by zero

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33352 into master

* refs/pull/33352/head:
ceph.in: catch KeyboardInterrupt from outer code
ceph.in: fix second exception on ^C

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33374 into master

* refs/pull/33374/head:
cephadm: remove distutils dependency
mgr/cephadm: add prepare-host command
cephadm: prepare-host during bootstrap
cephadm: add prepare-host command
cephadm: include docker.io in image name

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: trigger device refresh after osd creation

Presumably the new OSD(s) consumed one or more devices.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch,cephadm: implement 'device zap'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove inventory_cache

No longer used.

Clean up _get_hosts now that there is only one user.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: reimplement 'device ls' in terms of cache

Signed-off-by: Sage Weil <sage@redhat.com>