qa/workunits/cephadm/test_repos: don't try to use the refspec

This is usually a sha1, and we can't reliably find packages based on
that.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm/test_repos: apt update

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm/test_repos: fix subst

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm/.../test_repos: run without ulimit and coverage tools

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/workunit: accept coverage_and_limits: false

Allow workunits without teuthology tools (normally installed by ceph.py,
IIRC).

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm: move ubuntu_18.04_podman to shared location

Also set the registries.conf file so we can pull from docker.io.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm: fix conflicts, missing .qa link

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33070 into master

* refs/pull/33070/head:
mgr/telemetry: use raise_for_status()
mgr/telemetry: factor post into helper
mgr/telemetry: catch exception during requests.put

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge PR #33118 into master

* refs/pull/33118/head:
orchestrator cli: change 'rgw update' params order
mgr/cephadm: fix minor typo

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33062 into master

* refs/pull/33062/head:
qa/suites/rados/cephadm: collect all cephadm tests together here
qa/workunits/cephadm/test_repos: add test for the repo commands
cephadm: add '{add,rm}-repo', with initial centos/rhel/fedora/ubuntu support

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33119 into master

* refs/pull/33119/head:
mgr/upgrade: fix mgr self check
mgr/cephadm: fail upgrade if target image doesn't exist
mgr/cephadm: factor upgrade failure into helper
mgr/cephadm: refresh if we don't know a daemon's image_id
mgr/cephadm: refresh services in upgrade loop
mgr/cephadm: clean up upgrade messages a bit

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33086 from guits/guits-fix_cv_rerun

ceph-volume: skip osd creation when already done

qa/suites/rados/cephadm: collect all cephadm tests together here

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm/test_repos: add test for the repo commands

This isn't a great test, but it is something.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add '{add,rm}-repo', with initial centos/rhel/fedora/ubuntu support

Other distros to follow.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33111 into master

* refs/pull/33111/head:
qa/suites/rados/cephadm[-smoke]: test podman on ubuntu 18.04

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33128 from pponnuvel/fix_incorrect_link_documentation

coding-style: update a link and fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

coding-style: update a link and fix typos.

Signed-off-by: Ponnuvel Palaniyappan <pponnuvel@gmail.com>

ceph-volume: add unit test test_safe_prepare_osd_already_created

This commit adds a new unit test
`test_safe_prepare_osd_already_created()` in order to test when
`is_ceph_device()` returns `True` `RuntimeError` is well raised.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: skip osd creation when already done

When rerunning ceph-volume lvm create on a device already prepared and
activated, ceph-volume should skip the creation.

This is a regression introduced by bb4de1a3fc238eaf9f717dc59c6bdf338ef6d657

Fixes: https://tracker.ceph.com/issues/43981
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge PR #33042 into master

* refs/pull/33042/head:
cephadm: bootstrap: warn on fqdn hostname

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/suites/rados/cephadm[-smoke]: test podman on ubuntu 18.04

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/upgrade: fix mgr self check

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fail upgrade if target image doesn't exist

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: factor upgrade failure into helper

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refresh if we don't know a daemon's image_id

This notably happens right after we deploy a fresh daemon.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refresh services in upgrade loop

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: clean up upgrade messages a bit

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32634 from jan--f/c-v-inventory-fixes

ceph-volume: finer grained availability notion in inventory.

Merge pull request #31978 from jan--f/c-v-batch-no-db-dev-drop

ceph-volume/batch: fail on filtered devices when non-interactive

Merge PR #33075 into master

* refs/pull/33075/head:
examples/librados: fix bufferlist::copy() in hello_world.cc.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume: add available property in target specific flavors

This adds two properties available_[lvm,raw] to device (and thus inventory).
The goal is to have different notions of availability based on the
intended use case. For example finding LVM structures make a drive
unavailable for the raw mode, but might be available for the lvm mode.

Fixes: https://tracker.ceph.com/issues/43400
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #33112 from jan--f/c-v-lvm-list-regression-31700

ceph-volume: fix regression and improve output in lvm list

Merge pull request #32985 from sebastian-philipp/mgr-progress-mypy

mgr/progress: Add integration to pybind/mgr/tox.ini

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32432 into master

* refs/pull/32432/head:
mds: Reorganize structure members in snap header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

orchestrator cli: change 'rgw update' params order

Fixes: https://tracker.ceph.com/issues/44029
First realm, then zone -- to be consistent with the other commands

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/cephadm: fix minor typo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #33114 into master

* refs/pull/33114/head:
cephadm:Fix name argument parsing during image check for non-ceph components

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

cephadm:Fix name argument parsing during image check for non-ceph components

bug in parsing introduced in 97def7c
args.name may exist but will be none if flag is not used
check the value in addition to checking if it exists

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge PR #33109 into master

* refs/pull/33109/head:
cephadm: fix inspect-image

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33089 into master

* refs/pull/33089/head:
cephadm: re-introduce the `podman logs` command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33110 into master

* refs/pull/33110/head:
qa/distros: rhel and centos: whitelist cephadm logrotate selinux denial

Reviewed-by: Boris Ranto <branto@redhat.com>

ceph-volume: fix various lvm list issues

A single report on a non-lvm device now works.
Format was cleaned up, report lvm journal,wal, db only once.

Fixes: https://tracker.ceph.com/issues/44009
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: add get_device_lvs to easily retrieve all lvs per device

Also drop the sep argument from get_lvs and siblings, unused.
Introduce LV_CMD_OPTIONS to unify options to lvs.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

cephadm: re-introduce the `podman logs` command

Fixes: https://tracker.ceph.com/issues/43973
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #33093 into master

* refs/pull/33093/head:
build-integration-branch: don't fail on existing branch

Reviewed-by: Neha Ojha <nojha@redhat.com>

cephadm: fix inspect-image

This was broken by d8debba782cd4f40ed13db7f1af8ef43503ccec5
because the 'images' json output works with podman but not with
docker.  (Also, the inspect command is more explicit and cleaner.)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/distros: rhel and centos: whitelist cephadm logrotate selinux denial

This is fixed in RHEL 8.1.1 (and by extension centos/rhel 8.2+).

No fix for el 7 yet

Partially-fixes: https://tracker.ceph.com/issues/43703
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33071 into master

* refs/pull/33071/head:
mgr/cephadm: remove item from cache when removing

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33059 from tspmelo/wip-node-10-18-1

make-dist: Bump Node.js to v10.18.1

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32546 from votdev/issue_43089_passwd_cmplx_config

mgr/dashboard: Make password policy check configurable

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #32881 from tchaikov/wip-43657

mgr/orchestrator: use deepcopy for copying exceptions

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/tasks/cephadm: test "orchestrator host ls"

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks: drop test_cephadm_orchestrator.py

this test will end with a failure like

```
2020-01-30T18:15:15.870 INFO:tasks.ceph.mgr.x.smithi042.stderr:Warning: Permanently added 'smithi042.front.sepia.ceph.com,172.21.15.42' (ECDSA) to the list of known hosts.
2020-01-30T18:15:15.925 INFO:tasks.ceph.mgr.x.smithi042.stderr:Permission denied, please try again.
2020-01-30T18:15:15.932 INFO:tasks.ceph.mgr.x.smithi042.stderr:Permission denied, please try again.
2020-01-30T18:15:15.939 INFO:tasks.ceph.mgr.x.smithi042.stderr:root@smithi042.front.sepia.ceph.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
```

because mgr is not able to establish an ssh connection to that host with "root".
please note, the teuthology worker is acting using the "ubuntu" account on the
test node, and by default, "root" does not have its pubkey. and actually
`qa/tasks/cephadm.py` does push the pubkey to all the managed hosts before
testing cephadm.

since `qa/tasks/cephadm.py` is a better test for cephadm, let's just
drop this one.

as suites/rados/cephadm already covers cephadm

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: use deepcopy for copying exceptions

since rexec module has been removed in python3, we cannot use it
anymore.

Fixes: https://tracker.ceph.com/issues/43657
Signed-off-by: Kefu Chai <kchai@redhat.com>

build-integration-branch: don't fail on existing branch

This behavior is too annoying, and you can always get back to something
clobbered with 'git reflog'.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32946 into master

* refs/pull/32946/head:
qa/suites/rados: improve valgrind leak check
common/ceph_context: add an asok command to deliberately leak memory

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/telemetry: use raise_for_status()

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/telemetry: factor post into helper

And use it for devices too

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33055 into master

* refs/pull/33055/head:
qa/tasks/mgr/test_orchestrator_cli: support multiple DriveGroups

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33056 into master

* refs/pull/33056/head:
common: fix clang compile errors from cython_modules

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33058 into master

* refs/pull/33058/head:
mgr/cephadm: enforce that a host is a valid DNS name
mgr/cephadm: verify host's hostname matches our host name
cephadm: check-host: add optional --expect-hostname

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #33069 into master

* refs/pull/33069/head:
cephadm: use appropriate default image for non-ceph components

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33039 into master

* refs/pull/33039/head:
osd/OSD: prevent down osds from immediately rejoining the culster
osd/OSD: trim osd_markdown_log in tick() thread

Reviewed-by: yanjun <yan.jun8@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: enforce that a host is a valid DNS name

This combines the hostname restrictions

 * 1-63 chars
 * a-z, A-Z, 0-9, -

and the DNS name restrictions

 * .-delimited
 * no empty components (or leading or trailing .)
 * 250 chars total max

Note that this allows bare IPv4 addresses (which are indistinguishable from
a valid DNS name, AFAICS), but disallows bare IPv6 addresses.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove item from cache when removing

This makes the daemon disappear immediately from 'service ls', and also
avoids a temporary health warning about a stray service.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Fix fsevents and node-gyp error

Signed-off-by: Tiago Melo <tmelo@suse.com>

make-dist: Bump Node.js to v10.18.1

This will fix an error caused by the usage of the latest version of Angular CLI
and Node.js v10.16.0.

Fixes: https://tracker.ceph.com/issues/43961
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33085 from rzarzynski/wip-client-bl_iter_advance

client: fix FTBFS due to bl::iterator::advance().

Reviewed-by: Kefu Chai <kchai@redhat.com>

client: fix FTBFS due to bl::iterator::advance().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33026 from liewegas/wip-el81

qa/distros: add rhel/centos 8.1

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32996 from cbodley/wip-rgw-put-multipart-stripe

rgw: MultipartObjectProcessor supports stripe size > chunk size

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #32811 from multi-arch/master

test/rgw: fix test_rgw_reshard_wait with -DHAVE_BOOST_CONTEXT=OFF

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #33051 into master

* refs/pull/33051/head:
mgr/telemetry: check get_metadata return val

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33018 from mgfritch/cephadm-docker-disabled

qa/workunits/cephadm/test_cephadm.sh: skip docker when service is disabled

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32982 from krig/cephadm-fixes

cephadm: Read ceph version from io.ceph.version label if set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33025 from neha-ojha/wip-no-mgr

mon/MgrMonitor.cc: warn about missing mgr in a cluster with osds

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33077 from guits/guits-cv_fix_listing

ceph-volume: fix lvm list

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #33004 from matthewoliver/argparge_matchcnt_kwargs

ceph_argparse: increment matchcnt on kwargs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33074 from guits/guits-cv_quick_fix

ceph-volume: fix has_bluestore_label() function

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #33015 from rouming/double-unlock-p1-fix

msg/async: open() should be called with connection locked

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge pull request #33029 from yaarith/wip-telemetry-show-device

mgr/telemetry: anonymizing smartctl report itself

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33003 from tchaikov/wip-buffer-list-advance

include/buffer: add operator+=() for list::iterator

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33034 from rzarzynski/wip-include-ceph_atomic_in_mempool

include: switch mempool.h to ceph::atomic.

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: remove stderr in has_bluestore_label()

We don't want to generate this log when a call to
`has_bluestore_label()` fails.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: fix has_bluestore_label() function

When using vg/lv, this function throws an error like following:

```
 stderr: unable to read label for test_group/data-lv2: (2) No such file or directory
 stderr: 2020-02-04T21:03:32.153+0000 7fe091af4200 -1 bluestore(test_group/data-lv2) _read_bdev_label failed to open test_group/data-lv2: (2) No such file or directory
```

using `self.abspath` fixes this error.

Fixes: https://tracker.ceph.com/issues/43970
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: fix lvm list

17957d9beb42a04b8f180ccb7ba07d43179a41d3 introduced a regression in `lvm
list`.

When passing a vg/lv path for generating a single report, it fails
because the filter used in the `lvs` command isn't right. It uses the lv
name instead of the vg name because `os.path.basename(device)` is used
while it should be `os.path.dirname(device)`

Fixes: https://tracker.ceph.com/issues/43969
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #33076 from liu-chunmei/fix_crimson_pg_coll

crimson: fix crimson pg coll usage error

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33057 from wjwithagen/wjw-fix-buffer.copy

rbd-ggate: fix fallout from bufferlist.copy() change

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: fix crimson pg coll usage error

coll in pg isn't initialized, should use coll_ref->get_cid() insead.

Signed-off-by: Chunmei Liu <chunmei.liu@intel.com>

cephadm: use appropriate default image for non-ceph components

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33066 from gregsfortytwo/wip-specfile

rpm: fix up a specfile syntax error

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

examples/librados: fix bufferlist::copy() in hello_world.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/telemetry: catch exception during requests.put

Sometimes requests.put throws an exception.  When that happens, generate
an appropriate error but don't crash the module.

Fixes: https://tracker.ceph.com/issues/43963
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30033 from theanalyst/rgw/public-buckets

rgw: add PublicAccessBlock set of APIs on buckets

Reviewed-By: Casey Bodley <cbodley@redhat.com>

rpm: fix up a specfile syntax error

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #33049 from yuvalif/wip-yuval-fix-43768

qa/rgw/pubsub: fix tests to sync from master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/cephadm: verify host's hostname matches our host name

This ensures that we have a consistent view between the hosts that cephadm
is using and the hostname being reported as metadata by the ceph daemons.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: check-host: add optional --expect-hostname

Signed-off-by: Sage Weil <sage@redhat.com>

doc: add Pending Release Notes entry on public access config

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>