ceph-daemon: fix extract_uid_gid

This will get confused by the cephdaemon user!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31816 from jan--f/c-v-no-unittest-mock

ceph-volume: import mock.mock instead of unittest.mock (py2)

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #30863 from matthewoliver/improve_do_cmake

do_cmake.sh: Add CEPH_GIT_DIR

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31502 into master

* refs/pull/31502/head:
qa/tasks/ceph2: get ceph-daemon from same place as ceph
qa/tasks/ceph2: use safe_while
qa/tasks/ceph2: pull image using sha1
qa/tasks/ceph2: docker needs quay.io/ prefix for image name
qa/workunits/rados/test_python: make sure rbd pool exists
qa/suites/rados/ssh: new tests!
qa/tasks/ceph2: pull ceph-ci/ceph:$branch
qa/tasks/ceph2: register_daemons after pods start
qa/tasks/ceph2: fix conf
qa/tasks/ceph2: add restart
qa/tasks/ceph2: pass ceph-daemon path to DaemonState
qa/tasks/ceph2: tolerate no mdss or 1 mgr
qa/tasks/ceph: replace wait_for_osds_up with manager.wait_for_all_osds_up
qa/tasks/ceph: wait-until-healthy
qa/tasks/ceph2: set up managers
qa/tasks/ceph2: use seed ceph.conf
qa/tasks/ceph: healthy: use manager helpers (instead of teuthology/misc ones)
qa/tasks/ceph2: name mds daemons
qa/tasks/ceph2: fix osd ordering
qa/tasks/ceph2: start up mdss
qa/tasks/ceph2: set up daemon handles and use them to stop
qa/tasks/ceph2: make it multicluster-aware
qa/tasks/ceph2: can bring up mon, mgr, osds!
qa/tasks/ceph2: basic task to bring up cluster with ceph-daemon and ssh

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31657 from liewegas/fix-merge-vs-backoff

osd: release backoffs during merge

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/ceph2: get ceph-daemon from same place as ceph

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31698 into master

* refs/pull/31698/head:
rpm,deb: add cephdaemon user and sudoers file
ceph-daemon: use mgr/ssh command to create the ssh key
mgr/ssh: add 'ssh generate-key' and 'ssh clear-key' commands
mgr/ssh: add mode option
mgr/ssh: add 'ssh get-pub-key' and 'ssh get-user' commands
mgr/ssh: convert to command decorations
mgr/orchestrator: move command annotation helpers to orchestrator.py

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31790 into master

* refs/pull/31790/head:
Revert "Merge pull request #16715 from adamemerson/wip-I-Object!"

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

ceph-volume: import mock.mock instead of unittest.mock (py2)

Fixes: bb4de1a3fc238eaf9f717dc59c6bdf338ef6d657
Fixes: https://tracker.ceph.com/issues/42970
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31814 from jan--f/c-v-check-selinux-py2

ceph-volume: check if we run in an selinux environment, now also in py2

Merge PR #31788 into master

* refs/pull/31788/head:
ceph-daemon: append newline before public key string

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Revert "Merge pull request #16715 from adamemerson/wip-I-Object!"

This reverts commit 669453138d89e0f797a1bd37f38a2d68e6aac366, reversing
changes made to 36f5fcbb97eb2b1bceb526331eb3464f460fc701.

Signed-off-by: Sage Weil <sage@redhat.com>
- conflicts due to code rearrangement in 14b0db908f652032c358e419ffa90f5676698d0e

 ceph-volume: python2 raises OSError on Popen with missing binary.

For compatibility alias FileNotFoundError to OSError.

Fixes: 72b79f71fa84fe0d56b2fd1745f5bb130380cc38
Fixes: https://tracker.ceph.com/issues/42967
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #31798 into master

* refs/pull/31798/head:
ceph-daemon: ceph-volume works without an fsid
ceph-daemon: several commands that can infer fsids still require them
ceph-daemon: fix fsid inference

Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph-volume: py2 compatibility for selinux test

Fixes: af30c7f956380beb3bc4021d60b9229a82bda4e2
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31809 from jan--f/c-v-check-selinux

ceph-volume: check if we run in an selinux environment

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-daemon: ceph-volume works without an fsid

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: several commands that can infer fsids still require them

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: fix fsid inference

All return paths should return func.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31771 from idryomov/wip-krbd-read-only-test

qa: update krbd_blkroset.t and add krbd_get_features.t

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

ceph-volume: don't assume SELinux

Expect selinux related binaries to not be present.

Fixes: 33c8a64a54d9ea8962091caf8564cea3f603c5f5
Fixes: https://tracker.ceph.com/issues/42957
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #30621 from wjwithagen/wjw-fix-freebsd-linking

cmake: use GNU linker on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: fix test test_selinuxenabled_doesnt_exist

Popen does not open a shell by default, so A FileNotFoundError is
raised instead of returning a 127 return code.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31777 from smithfarm/wip-mimic-v13.2.7-release-notes

doc/releases: add release notes for mimic v13.2.7

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30387 from rzarzynski/wip-crimson-errorator

crimson: bring ceph::errorator with its first appliances

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31793 into master

* refs/pull/31793/head:
mds: release free heap pages after trim

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31799 from yuriw/wip-yuriw-crontab-master

qa/tests: trying to fix syntax error that prevented mimic-x to be add…

qa/tests: trying to fix syntax error that prevented mimic-x to be added to the schedule

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #31465 into master

* refs/pull/31465/head:
doc/install/containers: add summary of containers and branches

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge PR #31702 into master

* refs/pull/31702/head:
ceph-daemon: infer fsid for shell, enter, ceph-volume, unit, logs

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/tasks/ceph2: use safe_while

Signed-off-by: Sage Weil <sage@redhat.com>

rpm,deb: add cephdaemon user and sudoers file

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases: add release notes for mimic v13.2.7

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30960 from yuvalif/wip-yuval-add-kafka-notif-endpoint

rgw/pubsub: add kafka notification endpoint

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

ceph-daemon: use mgr/ssh command to create the ssh key

The mgr/ssh module has a command now to do this; use that instead of
generating the key ourselves and manually injecting it.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add 'ssh generate-key' and 'ssh clear-key' commands

Signed-off-by: Sage Weil <sage@redhat.com>

mds: release free heap pages after trim

MDS free heap space can grow to large for some workloads (like smallfile
and recursive deletes). This can cause the MDS mapped memory to grow
well beyond memory targets.

When we finally use the PriorityCache in the MDS, this will not be
necessary anymore as the PriorityCache already does this.

Fixes: https://tracker.ceph.com/issues/42938
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/ssh: add mode option

Set the mgr/ssh/mode option to ceph-daemon-package to switch to a mode
where we assume ceph-daemon is installed on the remote host and we can
ssh as user cephdaemon.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add 'ssh get-pub-key' and 'ssh get-user' commands

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: convert to command decorations

Also fix up set-ssh-config error checking a bit.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: move command annotation helpers to orchestrator.py

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31747 into master

* refs/pull/31747/head:
qa/suites/rados/singleton-nomsgr/all/balancer: whitelist PG_AVAILABILITY

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31734 from smithfarm/wip-cbs-fork-option

ceph-backport.sh: allow user to specify --fork explicitly

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

qa/tasks/ceph2: pull image using sha1

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: docker needs quay.io/ prefix for image name

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_python: make sure rbd pool exists

The ceph2 task doesn't create it for you.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/ssh: new tests!

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: pull ceph-ci/ceph:$branch

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: register_daemons after pods start

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: fix conf

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: add restart

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: pass ceph-daemon path to DaemonState

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: tolerate no mdss or 1 mgr

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: replace wait_for_osds_up with manager.wait_for_all_osds_up

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: wait-until-healthy

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: set up managers

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: use seed ceph.conf

Based on ceph.conf.template, but edited down a bit.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: healthy: use manager helpers (instead of teuthology/misc ones)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: name mds daemons

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: fix osd ordering

Sort by the numeric key

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: start up mdss

TODO: name them properly

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: set up daemon handles and use them to stop

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: make it multicluster-aware

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: can bring up mon, mgr, osds!

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: basic task to bring up cluster with ceph-daemon and ssh

This is pretty minimal at this point, but it works so far (or will, once a few open PRs merge).

Main problematic area right now is that the multi-cluster stuff isn't right.  We should fix that before continuing, since ceph-daemon for once can do multiple clusters well.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31120 into master

* refs/pull/31120/head:
mds: Reorganize class members in MDSRank header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-daemon: append newline before public key string

Sometimes, `authorized_keys` file does not end with a newline and
therefore the public key generated by ceph-daemon gets appended to
the same line of the previous key. Therefore we need to append a
newline before our key to guarantee that the above case does not
happen.

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Cross sign button not working for some modals (#31783)

mgr/dashboard: Cross sign button not working for some modals

Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: sort monitors by open sessions correctly. (#31752)

mgr/dashboard: sort monitors by open sessions correctly.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31738 from ricardoasmarques/wip-41776

mgr/dashboard: Unable to set boolean values to false when default is true

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #31773 from idryomov/wip-krbd-snapshot-removal-test

qa: kernel.sh: update for read-only changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Don't use any xlf file when building the defaul… (#31727)

mgr/dashboard: Don't use any xlf file when building the default language

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #31748 into master

* refs/pull/31748/head:
qa/standalone/test_ceph_daemon.sh: remove old vg before creating
qa/standalone/test_ceph_daemon.sh: sudo for untar
qa/standalone/test_ceph_daemon.sh: sudo for losetup etc
qa/standalone/test_ceph_daemon.sh: fix overwrites of temp files

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: Update dev documentation URL (#31724)

mgr/dashboard: Update dev documentation URL

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Use new ImageSpec class (#31622)

mgr/dashboard: Use new ImageSpec class

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Dashboard can't handle self-signed cert on Graf… (#31129)

mgr/dashboard: Dashboard can't handle self-signed cert on Grafana API

Reviewed-by: Stephan Müller <smueller@suse.com>

qa: add krbd_get_features.t test

Run it together with krbd_blkroset.t.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: open files with UTF-8 encoding in Grafana check… (#31750)

mgr/dashboard: open files with UTF-8 encoding in Grafana checking script

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Cross sign button not working for some modals

Fixes: https://tracker.ceph.com/issues/42179
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mds: Reorganize class members in MDSRank header

Fixes: https://tracker.ceph.com/issues/42465
Signed-off-by: Varsha Rao <varao@redhat.com>

qa/standalone/test_ceph_daemon.sh: remove old vg before creating

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/test_ceph_daemon.sh: sudo for untar

The deepsea.tgz tar contains actual device nodes for the OSD block devices
(not symlinks or files).  Must be root to untar.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/test_ceph_daemon.sh: sudo for losetup etc

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/test_ceph_daemon.sh: fix overwrites of temp files

mktemp creates these files, so we have to pass --allow-overwrite (or
delete them after we get the unique name but before we write to them--this
is easier).

Broken by c7fe27a72a61d1345a66b8830fd17e7b922abd44

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31418 into master

* refs/pull/31418/head:
test: use distinct subvolume/group/snapshot names

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #30553 into master

* refs/pull/30553/head:
qa/tasks: check if fs mounted in umount_wait

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31534 into master

* refs/pull/31534/head:
mds: fix invalid access of mdr->dn[0].back()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31548 into master

* refs/pull/31548/head:
mds: properly evaluate unstable locks when evicting client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31531 into master

* refs/pull/31531/head:
cmake: link ceph-fuse against librt

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: kernel.sh: update for read-only changes

Snapshot existence validation code was removed from krbd.  It was racy
and relied on having watch established for snapshots.

Fixes: https://tracker.ceph.com/issues/42916
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

crimson: avoid seastar::do_with() due to performance reasons.

`seastar::do_with(T&& rvalue, F&& f) takes object for lifetime
extension by rvalue reference. This imposes materialization of
a temporary to move from even when `do_with()` is being called
like:

  `do_with(OpsExecuter{...}, [] { /* ... */)`.

The reason behind that is following language rule:

  "Temporary objects are created when a prvalue is materialized
  so that it can be used as a glvalue, which occurs (since C++17)
  in the following situations:

   * binding a reference to a prvalue"
  (from: "Temporary object lifetime", cppreference.com)

As OpsExecuter is pretty heavy-weight, it is reasonable to avoid
`do_with()` and perform the lifetime extension with smart pointer.
Additional benefit is squeezing plain-to-errorated conversion in
`seastar::internal::do_with_state::get_future()`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: OpsExecuter::submit_changes() operates on errorated future.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: introduce make_exception_future to errorator.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: drop extra copy in errorator's appliance.

Save few unneeded instructions on the read path. I expect
negligible impact on performance. The patch is rather for
easier reading of assembler.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: dissect error handling path from ::safe_then().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: drop ceph::do_with(). Specialize sestar::do_with() instead.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: specialize seastar::futurize to avoid copying.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: _load_{os,ss} return errorated future to avoid copying.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: drop as_plain_future() from errorated futures.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: replace make_plain_exception_future() with errorize{}.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>