mgr/ssh: fix remove_mds exception

Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/orchestrator_cli: remove extra characters after checkmarks

Not sure what those were.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/orchestrator_cli: no {stop,start,restart} for all the random svcs

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add 'redeploy' operation

This will stop, reprovision container, and restart. For most daemons, that
means we can

 - change the container image
 - refresh the config
 - refresh the keyring

For OSDs, it means we also re-run ceph-volume activate.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: stop + reset-failed existing unit on daemon deploy

Normally it won't exist, but it will on a re-deploy.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: make second call arg optional

If not specified, pull it from command[0].

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: make 'deploy' on mon only to mkfs step is mon doesn't yet exist

This allows us to (re)deploy a mon container like the other daemon types,
which means rewriting the config, keyring, and systemd unit(s).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh,orchestrator_cli: add 'restart' service action

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: service[-instance] [start|stop]

Note that 'reload' is meaningless for ssh orch...

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: simplify get_services call for 'mds update'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: make _get_services take service *id* or service *name*

Name is the group, id is the instance.  A bit confusing, but this is what
orchestrator.py uses.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: fix 'service ls' on stopped services

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: move some code around

Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/orchestrator_cli: 'service ls' works for ssh

See 7fc78b63425abd192a82f812958349f2bf079dea merge commit.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/orchestrator_cli: we already completed mds ops for rook+ssh

See f5bc2bd74a05bb486fbbca2de4dbe9a1cacd605c merge.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31270 into master

* refs/pull/31270/head:
ceph-daemon: no extra newline needed in authorized_keys file
ceph-daemon: use client.admin keyring during bootstrap

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #31265 from idryomov/wip-krbd-unmap-msgr1

qa/suites/krbd: run unmap subsuite with msgr1 only

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Remove convertion to seconds (#31262)

mgr/dashboard: Remove convertion to seconds

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ni-Feng Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: edit/clone/copy rbd image after its data is rec… (#31245)

mgr/dashboard: edit/clone/copy rbd image after its data is received

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31247 from smithfarm/wip-bri-regression

backport-resolve-issue: "recognize that Target version is populated" and "prune duplicate URLs"

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #31231 into master

* refs/pull/31231/head:
qa: add tests for CephFS admin commands
mds/MDSRank: report state to mgr as mds id, not rank

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-daemon: no extra newline needed in authorized_keys file

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: use client.admin keyring during bootstrap

It's usually okay to use the mon. key for CLI commands, except we had a
mgr but that prevented you from issuing mgr commands correctly.  We have
the new client.admin key available, so use that instead.

Update tests to not --skip-ssh (now that it doesn't hang).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31264 into master

* refs/pull/31264/head:
ceph-daemon: deal with lack of TemporaryDirectory on py2
ceph-daemon: make StringIO usage py2 compatible
ceph-daemon: make ConfigParser py2 compatible

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa/suites/krbd: run unmap subsuite with msgr1 only

pre-single-major.yaml kernel doesn't have any of the monitor client
fixes that came in 4.6.  If the connection is closed, it closes the
session and retries only after 10 seconds.  On top of that, there is
nothing to prevent it from picking the same monitor when reconnecting.
This means that when given both v1 and v2 ports (which look like two
different monitors), it is susceptible to mount_timeout (60 seconds):

  $ sudo rbd map img
  rbd: sysfs write failed
  In some cases useful info is found in syslog - try "dmesg | tail".
  rbd: map failed: (5) Input/output error

  [  822.242313] libceph: mon0 172.21.15.132:3300 socket closed (con state CONNECTING)
  [  832.265494] libceph: mon0 172.21.15.132:3300 socket closed (con state CONNECTING)
  [  842.296175] libceph: mon0 172.21.15.132:3300 socket closed (con state CONNECTING)
  [  852.326924] libceph: mon0 172.21.15.132:3300 socket closed (con state CONNECTING)
  [  862.357611] libceph: mon0 172.21.15.132:3300 socket closed (con state CONNECTING)
  [  872.388373] libceph: mon0 172.21.15.132:3300 socket closed (con state CONNECTING)
  [  882.676136] libceph: mon0 172.21.15.132:3300 socket closed (con state CONNECTING)

Unlike newer kernels that return ETIMEDOUT, it returns EIO.

Newer kernels are much more aggressive about retries and will pick
a different monitor when reconnecting, hence they are always able to
establish the session in time.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: add tests for CephFS admin commands

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-daemon: deal with lack of TemporaryDirectory on py2

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: make StringIO usage py2 compatible

Otherwise,

Traceback (most recent call last):
  File "../src/ceph-daemon", line 1698, in <module>
    r = args.func()
  File "../src/ceph-daemon", line 767, in command_bootstrap
    with StringIO() as f:
AttributeError: StringIO instance has no attribute '__exit__'

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: make ConfigParser py2 compatible

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Remove convertion to seconds

This is no longer need since the values are already in seconds.

Fixes: https://tracker.ceph.com/issues/42567
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #31195 into master

* refs/pull/31195/head:
vstart: Fix /dev/tty No such device or address

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31161 from trociny/wip-42488

rbd-mirror: mirrored clone should be same format

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31149 from wonderpow/master1

cls/rbd: add snapshot_add raise -ESTALE test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31140 from Gangbiao/add-pybind-test-case

test/pybind: add create_snap rasie ImageExists test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30994 from JackSlateur/rbd_merged_du

rbd: add --merge to disk-usage

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: remove space in tox.ini deps (#31246)

mgr/dashboard: remove space in tox.ini deps

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

vstart: Fix /dev/tty No such device or address

Running on Jenkins slave or as a daemon there will be no writable console.

Fixes: https://tracker.ceph.com/issues/42487
Signed-off-by: Volker Theile <vtheile@suse.com>

backport-resolve-issue: prune duplicate URLs

I have seen PR descriptions with a single backport tracker URL mentioned twice.
When backport-resolve-issue hit one of these, it would say:

Found backport tracker: https://tracker.ceph.com/issues/39272
Found backport tracker: https://tracker.ceph.com/issues/39272
-----------------------------------------------------------------
INFO:root:Tracker https://tracker.ceph.com/issues/39272 links to PR https://github.com/ceph/ceph/pull/29153
INFO:root:Backport Tracker 39272 target version already populated with correct value v12.2.13
INFO:root:Backport Tracker 39272 status is already set to Resolved
-----------------------------------------------------------------
INFO:root:Tracker https://tracker.ceph.com/issues/39272 links to PR https://github.com/ceph/ceph/pull/29153
INFO:root:Backport Tracker 39272 target version already populated with correct value v12.2.13
INFO:root:Backport Tracker 39272 status is already set to Resolved
=================================================================

This commit fixes the issue.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Allow removal of single notification (#31048)

mgr/dashboard: Allow removal of single notification

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Use checkbox for multi selection (#30495)

mgr/dashboard: Use checkbox for multi selection

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: edit/clone/copy rbd image after its data is received

Fixes: https://tracker.ceph.com/issues/42561
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #31193 from rhcs-dashboard/wip-42507-master

mgr/dashboard: update dashboard CODEOWNERShip

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

backport-resolve-issue: recognize that Target version is populated

Due to a regression, the script stopped recognizing that a Backport tracker
issue's "Target version" field is populated.

Fixes: af43b3cc3b05e70ec248ad5c23da6fbd4c2e78a1
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30988 from smithfarm/wip-bri-multi

backport-resolve-issue: resolve multiple backport issues

mgr/dashboard: remove space in tox.ini deps

Fixes: https://tracker.ceph.com/issues/42469
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31163 from callithea/wip-issue-42489

doc: Small update of SubmittingPatches-backports

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30879 from smithfarm/wip-cbs-check-missing-milestone

ceph-backport.sh: implement --milestones feature and more-careful vetting

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #29930 from IlsooByun/rocksdb_ver

cmake: require RocksDB 5.14 or higher

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/MDSRank: report state to mgr as mds id, not rank

Introduced by 625dffe65c0f8001b3b6ca6d0b12732a1a103849, which added
periodic scrub stats reporting.

Fixes: https://tracker.ceph.com/issues/42494
Fixes: https://tracker.ceph.com/issues/41525
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #30369 into master

* refs/pull/30369/head:
doc/cephfs: add doc for cephfs io path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31201 into master

* refs/pull/31201/head:
doc: document rank option for journal reset

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #31181 into master

* refs/pull/31181/head:
ceph-daemon: only pass podman -it if need an interactive shell

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31159 from alfredodeza/wip-rm42484

ceph-volume tests remove xenial from functional testing

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

test/pybind: add create_snap rasie ImageExists test case

Signed-off-by: Gangbiao Liu <liugangbiao@cmss.chinamobile.com>

Merge pull request #31222 from sebastian-philipp/python-common-avoid-setup_requires

python-common: avoid using setup_requires in setup.py

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Allow removal of single notification

Signed-off-by: Tiago Melo <tmelo@suse.com>

ceph-daemon: only pass podman -it if need an interactive shell

Otherwise, we get errors like

2019-10-26T17:29:42.004 INFO:tasks.workunit.client.0.mira109.stderr:+ sudo /usr/sbin/ceph-daemon shell -- ceph -v
2019-10-26T17:29:42.004 INFO:tasks.workunit.client.0.mira109.stderr:+ grep 'ceph version'
2019-10-26T17:29:42.149 INFO:tasks.workunit.client.0.mira109.stderr:the input device is not a TTY

Fixes: https://tracker.ceph.com/issues/42499
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31059 into master

* refs/pull/31059/head:
mgr/rook: implement update_mds
mgr/ssh: implement 'mds update'
mgr/orchestrator_cli: add 'mds update' command
mgr/ssh: unique names for mds instances
mgr/ssh: name mgrs with random string id; add removal support
mgr/ssh: provide get_unique_name helper
mgr/orchestrator_cli: rename 'mds rm' arg
mgr/orchestrator_cli: rename 'mds add' arg
mgr/ssh: add 'mds rm'
mgr/ssh: (broken) mds rm
mgr/ssh: make mds add work
mgr/orchestrator: add count, hosts args to 'mds add'
mgr/ssh: be consistently verbose
mgr/ActivePyModules: expose 'mds_metadata' to modules

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31197 from jan--f/c-v-fix-lvm-mokeypatch

ceph-volume: mokeypatch calls to lvm related binaries

python-common: avoid using setup_requires in setup.py

Fixes: https://tracker.ceph.com/issues/42528
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31023 from idryomov/wip-krbd-udev-enumerate-retry

krbd: retry on transient errors from udev_enumerate_scan_devices()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

cmake: require RocksDB 5.14 or higher

error: ‘class rocksdb::Env’ has no member named ‘SetAllowNonOwnerAccess’
       opt.env->SetAllowNonOwnerAccess(false);
                    ^~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Ilsoo Byun <ilsoobyun@linecorp.com>

Merge pull request #31072 from sebastian-philipp/ceph-volume-inventory-device-id

ceph-volume: add Ceph's device id to inventory

Merge PR #31063 into master

* refs/pull/31063/head:
qa: disable too few PG warning during Mimic deploy

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31178 from tchaikov/wip/rpm/pyyaml

ceph.spec.in: add missing python-yaml dependency for mgr-k8sevents

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge PR #31168 into master

* refs/pull/31168/head:
ceph-daemon: try py2 import before py3
qa/suites/rados/singleton-nomsgr/ceph-daemon: make sure python3 is installed
qa/standalone/test_ceph_damon.sh: test with python2 and python3
mgr/ssh: python, not python3
ceph-daemon: python, not python3
ceph-daemon: os.makedirs
ceph-daemon: configparser is ConfigParser on py2
ceph-daemon: avoid py3-isms

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr/rook: implement update_mds

Note that this still doesn't provide control over active vs standby MDS
counts.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: implement 'mds update'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: add 'mds update' command

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: unique names for mds instances

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: name mgrs with random string id; add removal support

- stop naming mgrs after the host, since that doesn't scale well
- add mgr removal support.  prefer mgrs that aren't recognized by the
  cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: provide get_unique_name helper

Signed-off-by: Sage Weil <sage@redhat.com>

doc: document rank option for journal reset

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-daemon: try py2 import before py3

See https://github.com/ceph/ceph-medic/commit/f499a89b15b72a5bb842e1ebbc7cafc4daf58c1c

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton-nomsgr/ceph-daemon: make sure python3 is installed

Centos7 doesn't have it by default.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/test_ceph_damon.sh: test with python2 and python3

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: python, not python3

python 2.x is okay too.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: python, not python3

python2 is okay too.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: os.makedirs

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: configparser is ConfigParser on py2

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: avoid py3-isms

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: rename 'mds rm' arg

'name' can be either a group prefix or an instance name, so use a general
variable name.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: rename 'mds add' arg

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add 'mds rm'

This can take *either* an mds instance name *or* a group prefix.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: (broken) mds rm

This wrong because it takes a daemon name, not a group name, and because
it relies on mds_metadata--connected daemons, not deployed daemons.  Must
fix once we have better inventory.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: make mds add work

This is clearly primitive and not where we want to end up, but an initial
step.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: add count, hosts args to 'mds add'

Rook already uses the count; the hosts hint is not relevant.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: be consistently verbose

Print args and stdin in _run_ceph_daemon; drop various debug lines for
printing keyrings and configs along the way.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ActivePyModules: expose 'mds_metadata' to modules

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: add --merge-snapshots to disk-usage

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

Merge pull request #31180 from liewegas/revert-bluestore-pinned-list

os/bluestore: revert cache pinned list

Reviewed-by: Mark Nelson <mnelson@redhat.com>

ceph-volume: add Ceph's device id to inventory

This will benefit the orchestrator and dashboard to show a unified view of devices with SMART data

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph-volume: mokeypatch calls to lvm related binaries

Otherwise these tests fail occasionally when run on a system that does
not have lvm installed.

Fixes: https://tracker.ceph.com/issues/42045
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31071 from sebastian-philipp/python-common-mypy

python-common: Add mypy testing

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #30995 from wjwithagen/wjw-wip-volume-zfs-inventory

ceph-volume-zfs: add the inventory command

Merge pull request #31014 from tchaikov/wip-38378

mgr/dashboard: accept expected exception when SSL handshaking

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31092 from liewegas/wip-revert-42026-test

Revert "test: librados startup/shutdown racer test"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #31153 from ceph/wip-restful-node-items

restful: Query nodes_by_id for items

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31176 from tchaikov/wip/mgr/diskprediction_cloud/cleanup

mgr/diskprediction_cloud: refactor timeout() decorator

Reviewed-by: Rick Chen <rick.chen@prophetstor.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31175 from tchaikov/wip/mon/OSDMonitor/cleanup

mon/OSDMonitor: use initializer_list<> for {si,iec}_options

Reviewed-by: Sage Weil <sage@redhat.com>