cephadm: use appropriate default image for non-ceph components

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33048 into master

* refs/pull/33048/head:
cephadm: fix error handling in `command_check_host()`
cephadm: increase default retry_max value

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32995 into master

* refs/pull/32995/head:
cephadm: add group 'disk' to privileged container
cephadm: adopt: disable ceph-volume unit on host
cephadm: refactor privileged arg handling
cephadm: chown block symlink targets
cephadm-adoption-corpus: add stud-mon.tgz
cephadm: adopt: rename leveldb *.ldb -> *.sst
cephadm: adopt: by default, pull first
cephadm: adopt: chown data content

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33020 into master

* refs/pull/33020/head:
osdc/Objecter: inline pool full check
osdc/Objecter: remove duplicated pause check code
osdc/Objecter: only pause if respects_full()
osdc/Objecter: move respects_full() to op_target_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32831 into master

* refs/pull/32831/head:
common, include: drop the copy{_in} from bufferlist entirely.
os/bluestore: switch copy_in() users to bufferlist::iterator.
osdc: switch users of bufferlist::copy{_in} to iterators.
osd: switch users of bufferlist::copy{_in} to iterators.
rgw: switch copy{_in} users to bufferlist::iterator.
ec: switch users of bufferlist::copy{_in} to iterators.
cls/queue: switch users of bufferlist::copy{_in} to iterators.
client: switch users of bufferlist::copy{_in} to iterators.
*: switch trivial users of bufferlist::copy{_in} to iterators.
test/bl: switch copy{_in} users to bufferlist::iterator.
common, include: kill the bl::last_p member.
common: encode for std::list<T> doesn't use bl::copy_in() anymore.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32531 from zdover23/wip-doc-landing-page-update

doc: Added the crisp getting started guide to index.rst

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cephadm: add group 'disk' to privileged container

This lets the osd read block devs that are group rw disk even after they
drop root privs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32977 into master

* refs/pull/32977/head:
qa/workunits/cephadm/test_cephadm.sh: add missing monitoring tests
cephadm: simplify Monitoring.components structure
cephadm: add proper tox type for monitoring components

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #33012 into master

* refs/pull/33012/head:
mgr/cephadm: prefix daemon ids with hostname
cephadm: bootstrap: name mgr with $hostname.$random

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: adopt: disable ceph-volume unit on host

This might be a simple or lvm unit.  Disable it so that the host doesn't
try to start this OSD after a reboot.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: refactor privileged arg handling

Pass a bool if we want a privileged container instead of explicitly
passing --privileged.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix error handling in `command_check_host()`

`find_program()` raises `ValueError` when the executable hasn't been
found. It means we need to catch `ValueError` exception in
`command_check_host()` and raise `Error` instead of `RuntimeError` since
only `Error` is caught at the end.

Typical failure:

```
INFO:cephadm:/usr/bin/ceph:stderr Error ENOENT: New host mon1 failed check: ['INFO:cephadm:podman|docker (/bin/podman) is present', 'INFO:cephadm:systemctl is present', 'Traceback (most recent call last):', '  File "<stdin>", line 2820, in <module>', '  File "<stdin>", line 2434, in command_check_host', '  File "<stdin>", line 796, in find_program', 'ValueError: lvcreate not found']
```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

cephadm: increase default retry_max value

When playing with cephadm, at multiple times, I've reached the max
number of attempt in `is_available()`

Increasing the `retry_max` helps to avoid failure like following:

```
INFO:cephadm:mgr not available, waiting (1/5)...
INFO:cephadm:mgr not available, waiting (2/5)...
INFO:cephadm:mgr not available, waiting (3/5)...
INFO:cephadm:mgr not available, waiting (4/5)...
INFO:cephadm:mgr not available, waiting (5/5)...
ERROR: mgr not available after 5 tries

```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #32155 from yuvalif/wip-yuval-multidelete-notifications

rgw/pubsub: send notifications from multi-delete op

mgr/cephadm: prefix daemon ids with hostname

This is friendlier to a human operator since they can immediately see
where an instance is located, as with the legacy scheme, while still
keeping the unique random suffix.  Use a . to separate so that we can
set per-host options.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32859 from liewegas/wip-cephadm-services

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33030 from smithfarm/wip-luminous-12.2.13-release-notes

doc: release notes for Luminous v12.2.13

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #33031 from smithfarm/wip-nautilus-14.2.7-release-notes

doc: release notes for Nautilus 14.2.7

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #33038 from tchaikov/wip-librados-reinterpret-cast

librados: prefer reinterpret_cast over c-style cast

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #32933 from DimStar77/obs-mini

ceph.spec.in: Use pkgconfig() style BuildRequires for udev/libudev-devel

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32553 from tspmelo/wip-ts-strict

mgr/dashboard: Enable compiler options used by Angular --strict flag

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

librados: prefer reinterpret_cast over c-style cast

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33016 from tchaikov/wip-do-cmake

do_cmake.sh: bail out if something goes wrong

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/workunits/cephadm/test_cephadm.sh: add missing monitoring tests

add tests for node-exporter, prometheus, and grafana

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: simplify Monitoring.components structure

by removing the nested image/metadata dict

Signed-off-by: Michael Fritch <mfritch@suse.com>

More commits to the landing page project.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

cephadm: bootstrap: name mgr with $hostname.$random

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32570 into master

* refs/pull/32570/head:
cephfs-shell: Add tests for setxattr, getxattr and listxattr
cephfs-shell: Add listxattr command
cephfs-shell: Add getxattr command
cephfs-shell: Add setxattr command
doc: Update about extended attributes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31633 into master

* refs/pull/31633/head:
cephfs-shell: Instead of assert use stat for tests in rmdir
cephfs-shell: Add function for common rmdir test code
cephfs-shell: Add rmdir test for non empty directory
cephfs-shell: Add rmdir -p test for non empty directory
cephfs-shell: Add rmdir -p test for non existing dir
cephfs-shell: Add rmdir -p test to delete all dirs in given path
cephfs-shell: Add rmdir -p test for root directory with empty directories
cephfs-shell: Add rmdir test for valid file
cephfs-shell: Add rmdir test for invalid directory
cephfs-shell: Add rmdir test for valid directory
cephfs-shell: Fix rmdir '-p' issues

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32986 into master

* refs/pull/32986/head:
qa/tasks/ceph_manager: fix movement of cot exports with cephadm

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: release notes for Nautilus 14.2.7

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: release notes for Luminous v12.2.13

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32030 from vshankar/wip-mgr-volumes-clone

mgr/volumes: clone from snapshot

Merge PR #33014 into master

* refs/pull/33014/head:
mgr/dashboard: pass a list of drive_group to create_osds

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32973 into master

* refs/pull/32973/head:
mgr/insights: make 'insights prune-health-history 0' zap current state

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32987 into master

* refs/pull/32987/head:
qa/tasks/ceph_manager: make fix_pgp_num behave when no pool is found

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32989 into master

* refs/pull/32989/head:
qa/tasks/ceph_manager: add --log-early to raw_cluster_cmd

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32830 into master

* refs/pull/32830/head:
global: ensure CEPH_ARGS is decoded before early arg processing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32806 into master

* refs/pull/32806/head:
common/bl: fix accessibility of bptr's _off and _len fields.
common/bl: drop get_raw() from the public buffer::ptr interface.
common: drop sharing of buffer::raw outside bufferlist.

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_manager: fix movement of cot exports with cephadm

I think this will finally work...

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: chown block symlink targets

Again, these need to match the container's uid/gid, not the host's.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm-adoption-corpus: add stud-mon.tgz

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: adopt: rename leveldb *.ldb -> *.sst

We may be adopting from ubuntu, which means we need to rename these files.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: adopt: by default, pull first

This just bit me: I had a months-old container revision locally and ended
up accidentally starting up ancient monitor code.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: adopt: chown data content

The ceph uid/gid vary between distros.  Chown everything to match the
target distro.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: inline pool full check

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: remove duplicated pause check code

_calc_target() already pauses and unpauses based on the
target_should_be_paused() helper, and we just called _calc_target at the
top of this method.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: only pause if respects_full()

If the FULL_TRY or _FORCE flag is set on an op, then we should not pause
it due to a full flag.

Fixes: https://tracker.ceph.com/issues/43813
Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: move respects_full() to op_target_t

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33017 from linuxbox2/wip-rgw-beastc

rgw: improve beast

Merge pull request #33010 from tchaikov/wip-cmd-getval

src/: drop cct from cmd_getval()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

do_cmake.sh: bail out if something goes wrong

so we can be aware if some updated submodule reference is missing in our
repo

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: pass a list of drive_group to create_osds

as orchestrator backends expect a list of drive_groups.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Unable to remove an iSCSI gateway that is alrea… (#32984)

mgr/dashboard: Unable to remove an iSCSI gateway that is already in use

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add Modified filter to Configuration table (#32923)

mgr/dashboard: Add Modified filter to Configuration table

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

rgw: improve beast

Avoid leaking connections that had partially-consumed
client data on unexpected disconnect.

Resolves CVE-2020-1700 (moderate impact flaw).

Fixes: https://tracker.ceph.com/issues/42531
Signed-off-by: Or Friedmann <ofriedma@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cephadm: add proper tox type for monitoring components

instead of `type: ignore`

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge branch 'octopus'

mgr/dashboard: Enable noImplicitAny TS compiler option

Fixes: https://tracker.ceph.com/issues/43525
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Enable noImplicitReturns TS compiler option

Fixes: https://tracker.ceph.com/issues/43525
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Enable noImplicitThis TS compiler option

Fixes: https://tracker.ceph.com/issues/43525
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Enable noFallthroughCasesInSwitch TS compiler option

Fixes: https://tracker.ceph.com/issues/43525
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Move TS compile rules into tsconfig.json

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #32999 from liewegas/fix-found

doc/foundation: fix amihan

mgr/cephadm: Don't duplicate cache entries in _create_daemon

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: test_cephadm: always set cluster fsid

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: less noisy pytest failue messages

Let's hide the `import pydevd` exception, as it doesn't provide any useful info

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

src/: drop cct from cmd_getval()

it's not used.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: add mgr/volumes clone commands

Fixes: http://tracker.ceph.com/issues/24880
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add subvolume clone tests

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: allow force removal of incomplete failed clones

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: asynchronous cloner module

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: purge thread uses new async interface

This also makes `_cancel_jobs()` thread safe, which was not the
case earlier (with `_cancel_purge_job()`) -- this also makes the
code simpler by sharing the lock betweent two condition variables.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: fetch oldest clone entry

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: add clone specific commands

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: interface for fetching cloned subvolume status

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: add protect/unprotect and snap clone interface

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: handle transient subvolume states

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: interface for creating a cloned subvolume

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: get/set property for subvolume mode/uid/gid

This will be required when creating a clone as the clone would
inherit source subvolumes creation mode and uid/gid.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: module to track pending clone operations

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: add operation state machine table

... and fetch creation state from state machine table.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #33001 from tchaikov/wip-drop-dpdk-submodule

dpdk: drop dpdk submodule

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #32814 from mty22/au_ceph_branch_correct_docs

docs: Update au.ceph.com maintainers, update README.md.

Reviewed-by: Kefu Chai <kchai@redhat.com>

dpdk: drop dpdk submodule

we are using spdk/dpdk for async msgr's dpdk backend since 01a9f178. so
no need to keep dpdk submodule anymore.

Fixes: http://tracker.ceph.com/issues/24032
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #32947 into master

* refs/pull/32947/head:
OSD: Allow 64-char hostname to be added as the "host" in CRUSH

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32974 into master

* refs/pull/32974/head:
mon: dump json from 'sessions' asok/tell command

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32976 into master

* refs/pull/32976/head:
cls_hello: fix typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32981 into master

* refs/pull/32981/head:
mon: elector: return after triggering a new election

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32988 into master

* refs/pull/32988/head:
qa/tasks/ceph: only re-request scrub on unscrubbed pgs

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/foundation: fix amihan

I misunderstood the status of their foundation membership.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32992 from ceph/wip-revert-dpdk

Revert "dpdk: drop dpdk submodule"

Revert "dpdk: drop dpdk submodule"

This reverts commit cb8087dfac31b8490fefdfca28d389b7b9901ef8.

The seastar submodule's .gitmodules links to `../dpdk` which is no longer present after removing dpdk from ceph.git's .gitmodules.

```
<dwfreed> the ceph/seastar repo uses awful URLs for the submodules
<dwfreed> and those awful URLs are the real reason it's failing
<dwfreed> dgalloway: ^^^
<dwfreed> seastar's .gitmodules references repos in the parent directory, so that when it's checked out as a submodule of ceph, you don't download the repos twice (and git will probably also use references instead of duplicating the local .git); however, ceph doesn't have a submodule for dpdk anymore
<dwfreed> so seastar's referencing a dpdk repo that doesn't exist
<dgalloway> i think i follow.  so you're suggesting revert https://github.com/ceph/ceph/commit/cb8087dfac31b8490fefdfca28d389b7b9901ef8 ?
<dwfreed> yep
<dwfreed> that'd be one way to fix it
...
<joshd> dgalloway: I'd suggest revert for now, and let the crimson folks figure out the longer term fix when they're back
```

Signed-off-by: David Galloway <dgallowa@redhat.com>

mgr/insights: make 'insights prune-health-history 0' zap current state

The self._health_slot value needs to be updated too if the current key
is removed.

Fixes: https://tracker.ceph.com/issues/43886
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix style of placeholder * services entry

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: prime cached service state with newly created services

If we create a service, add the expected state to our cache, so that it
immediately appears in the get_services result.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: by default, return current service state

In the normal case, return whatever state we currently have.

In the serve loop, refresh if the state is old.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32958 into master

* refs/pull/32958/head:
qa/suites/rados/singleton/all/lost-unfound*: whitelist SLOW_OPS

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32789 into master

* refs/pull/32789/head:
test: fix unused assert's variable in ceph_test_osd_stale_read.cc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32961 into master

* refs/pull/32961/head:
qa/standalone/osd/osd-bench: debug bluestore

Reviewed-by: Neha Ojha <nojha@redhat.com>