ceph-daemon: Use `shutil` to move/copy files

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: Automatically use correct chromedriver version (#31352)

mgr/dashboard: Automatically use correct chromedriver version

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #31292 into master

* refs/pull/31292/head:
mgr/ssh: fix remove_mds exception
doc/mgr/orchestrator_cli: remove extra characters after checkmarks
doc/mgr/orchestrator_cli: no {stop,start,restart} for all the random svcs
mgr/ssh: add 'redeploy' operation
ceph-daemon: stop + reset-failed existing unit on daemon deploy
ceph-daemon: make second call arg optional
ceph-daemon: make 'deploy' on mon only to mkfs step is mon doesn't yet exist
mgr/ssh,orchestrator_cli: add 'restart' service action
mgr/ssh: service[-instance] [start|stop]
mgr/ssh: simplify get_services call for 'mds update'
mgr/ssh: make _get_services take service *id* or service *name*
mgr/ssh: fix 'service ls' on stopped services
mgr/ssh: move some code around
doc/mgr/orchestrator_cli: 'service ls' works for ssh
doc/mgr/orchestrator_cli: we already completed mds ops for rook+ssh

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/ssh: fix remove_mds exception

Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/orchestrator_cli: remove extra characters after checkmarks

Not sure what those were.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/orchestrator_cli: no {stop,start,restart} for all the random svcs

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add 'redeploy' operation

This will stop, reprovision container, and restart. For most daemons, that
means we can

 - change the container image
 - refresh the config
 - refresh the keyring

For OSDs, it means we also re-run ceph-volume activate.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: stop + reset-failed existing unit on daemon deploy

Normally it won't exist, but it will on a re-deploy.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: make second call arg optional

If not specified, pull it from command[0].

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: make 'deploy' on mon only to mkfs step is mon doesn't yet exist

This allows us to (re)deploy a mon container like the other daemon types,
which means rewriting the config, keyring, and systemd unit(s).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh,orchestrator_cli: add 'restart' service action

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: service[-instance] [start|stop]

Note that 'reload' is meaningless for ssh orch...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31266 from rhcs-dashboard/wip-42565-master

mgr/dashboard: fix LazyUUID4 not serializable

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: CephFS directory component (#30047)

mgr/dashboard: CephFS directory component

Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Automatically use correct chromedriver version

Every Chrome version requires a corresponding chromedriver version for the e2e
test to work.
Since we were using a fixed chromedriver version, each time chrome was updated
the e2e scripts stopped working.
With this new script we will check which version of Chrome is installed and
update chromedriver to the correct version.

Fixes: https://tracker.ceph.com/issues/42607
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #31191 from sebastian-philipp/ceph-volume-pr-template

github: Add ceph-volume to list of jenkins commands

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31326 from cbodley/wip-cmake-mgr-cap

cmake: define mgr_cap_obj library when WITH_MGR=OFF

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31015 into master

* refs/pull/31015/head:
mgr/orchestrator: Fix raise_if_exception for Python 3

Reviewed-by: Tim Serong <tserong@suse.com>

Merge PR #31337 into master

* refs/pull/31337/head:
Fixing typo in ceph-daemon script for the output_pub_ssh_key argument

Reviewed-by: Sage Weil <sage@redhat.com>

Fixing typo in ceph-daemon script for the output_pub_ssh_key argument

Signed-off-by: John McGowan <john@steakfest.com>

Merge pull request #31335 from wjwithagen/wjw-fix-objecstore-bluestore

test: move bluestore dependent code under WITH_BLUESTORE

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31312 from changchengx/qa

qa: correct zap disk with ceph-deploy tool

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30791 from kshtsk/wip-python3-no-iteritems-ceph-deploy

tasks/ceph_deploy: get rid of iteritems for python3

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Vasu Kulkarni <vasu@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31209 from wjwithagen/wjw-fix-snprintf

common: fix missing <stdio.h> include

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31184 from pdvian/wip-fix-sign-compare-warning

crush/CrushWrapper.cc: Fix sign compare compiler warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31306 from wjwithagen/wjw-fix-src_test_objectstore_CMakeLists.txt

test: only compile ceph_test_bmap_alloc_replay WITH_BLUESTORE

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31313 from tchaikov/wip/debian/rules/ceph-daemon

debian/rules: run dh_python2 with ceph-daemon

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31238 from tchaikov/wip/install-deps/el8

install-deps.sh: use sepia/lab-extra/8

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge PR #30751 into master

* refs/pull/30751/head:
doc: Document Export Process during Subtree Migrations

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30749 into master

* refs/pull/30749/head:
doc: document mds journal event types

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Douglas Fuller <dfuller@redhat.com>

test: move bluestore dependant code under WITH_BLUESTORE

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #31152 from LongDuncan/feature/mgr_restful_rquest_api_support_multicommand

mgr/restful: requests api adds support multiple commands

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge PR #30938 into master

* refs/pull/30938/head:
mds: Reorganize class members in MDSCacheObject header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28077 into master

* refs/pull/28077/head:
client: optimize rename operation under different quota root

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30365 into master

* refs/pull/30365/head:
cephfs-shell: get rid of "return" at the end of the method
pybind/cephfs.pyx: add NotDirectory exception
cephfs-shell: don't expect both Error and OSEror together
cephfs-shell: print built-in error messages for exceptions
cephfs-shell: use poutput defined with the module
cephfs-shell: use perror() defined within the module
cephfs-shell: don't pass default values in calls to perror
cephfs-shell: align perror's signature with cmd2.Cmd.perror
pybind/cephfs: make actual error message part of strerror of OSError

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge PR #30054 into master

* refs/pull/30054/head:
doc: fs subvolume resize command
qa/tasks: tests for resize subvolume
mgr/volumes: fs subvolume resize command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30402 into master

* refs/pull/30402/head:
client: add warning when cap != in->auth_cap.

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #30990 into master

* refs/pull/30990/head:
mds: Reorganize class members in MDSDaemon header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31207 into master

* refs/pull/31207/head:
qa/tasks: Better handling of thrasher names and __init__ calls

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31137 into master

* refs/pull/31137/head:
msg/DispatchQueue: Set throttle stamp for local_delivery

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31319 into master

* refs/pull/31319/head:
ceph-daemon: absolute path for /bin/install
ceph-daemon: ExecStart=/bin/bash script

Reviewed-by: Alfredo Deza <adeza@redhat.com>

cmake: define mgr_cap_obj library when WITH_MGR=OFF

resolves the cmake error when WITH_MGR=OFF:

  Error evaluating generator expression:

    $<TARGET_OBJECTS:mgr_cap_obj>

  Objects of target "mgr_cap_obj" referenced but no such target exists.
Call Stack (most recent call first):
  src/mon/CMakeLists.txt:34 (add_library)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #31318 into master

* refs/pull/31318/head:
ceph.spec.in: run pathfix.py
ceph-daemon: use /usr/bin/python, not /usr/bin/env python

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31226 into master

* refs/pull/31226/head:
mgr/telemetry: add stats about crush map
mgr/telemetry: add rgw metadata
mgr/telemetry: include fs size (files, bytes, snaps)
mds: report r{files,bytes,snaps} via perfcounters
mgr/telemetry: mds cache stats
mgr/telemetry: add some rbd metadata
mgr/telemetry: note whether osd cluster_network is in use
mgr/telemetry: add host counts
mgr/telemetry: add more pool metadata
mgr/telemetry: remove crush rule name
mgr/telemetry: include min_mon_release and msgr v1 vs v2 addr count
mgr/telemetry: add CephFS metadata

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #25374 into master

* refs/pull/25374/head:
osd: add flag to prevent truncate_seq copy in copy-from operation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31256 into master

* refs/pull/31256/head:
mgr/MgrClient: fix open condition

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/dashboard: AttributeError: 'NoneType' object has no attrib… (#31078)

mgr/dashboard: Fixed AttributeError: 'NoneType' in smart data integration

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph.spec.in: run pathfix.py

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: absolute path for /bin/install

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31320 into master

* refs/pull/31320/head:
ceph-daemon: fix os.mkdir call

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-daemon: fix os.mkdir call

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31314 into master

* refs/pull/31314/head:
mgr/orchestrator_cli: pass default value to "req=False" params

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31315 into master

* refs/pull/31315/head:
ceph-daemon: create ~/.ssh if not exist

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-daemon: ExecStart=/bin/bash script

Not sure why this matters, but it doesn't work on centos7 without the
/bin/bash part.

https://stackoverflow.com/questions/45776003/fixing-a-systemd-service-203-exec-failure-no-such-file-or-directory

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: create ~/.ssh if not exist

there is chance that /root/.ssh does not exist yet, when we are trying
to add key to authorized_keys.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/telemetry: add stats about crush map

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: use /usr/bin/python, not /usr/bin/env python

This works around noise from the rpm build:

*** ERROR: ambiguous python shebang in /usr/sbin/ceph-daemon: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31045 from tchaikov/wip/librados/no-more-ack-commit

librados,test,rgw: cleanups to deprecate safe_cb related functions

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31311 from tchaikov/wip/rbd/cleanup

librbd: always initialize local variables

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30912 from dillaman/wip-42265

mgr: add 'rbd' profiles to support 'rbd_support' module commands

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31244 from changchengx/deploy_doc

doc: update with osd addition

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31196 from liewegas/wip-osdmon-iec

mon/OSDMonitor: expand iec_options for 'osd pool set'

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: use new aio_create_completion()

Signed-off-by: Kefu Chai <kchai@redhat.com>

journal: use new aio_create_completion()

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librados_test_stub: implement aio_create_completion2() and friends

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: use new aio_create_completion()

to silence warnings like:

src/librbd/Utils.h:139:48: warning: 'static librados::v14_2_0::AioCompletion* librados::v14_2_0::Rados::aio_create_completion(void*, librados::callback_t, librados::callback_t)' is deprecated
[-Wdeprecated-declarations]
   return librados::Rados::aio_create_completion(
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
     obj, &detail::rados_callback<T, MF>, nullptr);
     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/libradosstriper: use new aio_create_completion()

to silence -Wdeprecated-declarations warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls/journal: s/create_completion/create_completion2/

src/cls/journal/cls_journal_client.cc:154:72: warning: 'static librados::v14_2_0::AioCompletion* librados::v14_2_0::Rados::aio_create_completion(void*, librados::callback_t, librados::callback_t)' is
deprecated [-Wdeprecated-declarations]
       librados::Rados::aio_create_completion(this, rados_callback, NULL);
                                                                        ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: s/create_completion/create_completion2/

to silence the warnings like

s_user_client.cc:151:77: warning: 'static librados::v14_2_0::AioCompletion* librados::v14_2_0::Rados::aio_create_completion(void*, librados::callback_t, librados::callback_t)' is deprecated
[-Wdeprecated-declarations]
   AioCompletion *c = librados::Rados::aio_create_completion(NULL, NULL, NULL);
                                                                             ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: s/aio_create_completion/aio_create_completion2/

the former is deprecated

Signed-off-by: Kefu Chai <kchai@redhat.com>

libradosstriper: s/create_completion/create_completion2/

replace `librados::Rados::aio_create_completion(arg, complete_cb,
safe_cb)` with `librados::Rados::aio_create_completion(arg,
complete_cb)`

the former is deprecated

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools: s/rados_aio_create_completion/rados_aio_create_completion2/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: s/rados_aio_create_completion/rados_aio_create_completion2/

the former is deprecated and replaced with the latter

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: s/wait_for_safe/wait_for_complete/

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: s/wait_for_safe/wait_for_complete/

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: s/wait_for_safe/wait_for_complete/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/ceph_kvstorebench: s/wait_for_safe/wait_for_complete/

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rados: s/wait_for_safe/wait_for_complete/

wait_for_safe() and friend are deprecated, so use wait_for_complete()
instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator_cli: pass default value to "req=False" params

in mds_add() command, both "num" and "hosts" are optional, so they
should have default values of "None". this change was introduced in
c35db383.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian/rules: run dh_python2 with ceph-daemon

to set substitute variable of `${python:Depends}` for this package

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: correct zap disk with ceph-deploy tool

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: update with osd addition

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

crush/CrushWrapper.cc: Fix sign compare compiler warning

introduced by 712a39e5c9d

Signed-off-by: Prashant D <pdhange@redhat.com>

librbd: always initialize local variables

silence warnings like:

src/librbd/io/ImageRequestWQ.cc:576:7: warning: 'length' may be used uninitialized in this function [-Wmaybe-uninitialized]
       if (block_overlapping_io(&m_in_flight_extents, offset, length)) {
       ^~
...
src/librbd/io/ImageRequestWQ.cc:935:12: note: 'length' was declared here
   uint64_t length;
            ^~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #31225 into master

* refs/pull/31225/head:
mgr/telemetry: send device telemetry via per-host POST to device endpoint

Reviewed-by: Dan Mick <dmick@redhat.com>

pybind/rados: do not use wait_for_safe() anymore

in librados, wait_for_safe() is an alias of wait_for_complete(). so in
this change, wait_for_safe() calls wait_for_complete(), and the callers
of wait_for_safe() will just call wait_for_complete(), instead of
calling both of them.

also, rados_aio_create_completion() is replaced with
rados_aio_create_completion2().

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: add Rados::aio_create_completion(arg, cb)

which does not take safe_cb, and will take the place of
Rados::aio_create_completion(arg, cb_complete, cb_safe)

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: add rados_aio_create_completion2()

rados_aio_create_completion2() is almost identical to
rados_aio_create_completion(). the only difference is that the former
does not accept the `safe` callback. as it's an alias of `complete`
callback since af01db76f63a22178509b402776da54164af1eb0

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: mark wait_for_safe() functions deprecated

safe is an alias of completed since
af01db76f63a22178509b402776da54164af1eb0. there is no need to have
two callbacks for tracking the state of an async operation anymore.

in this change, wait_for_safe() and friends are marked "deprecated"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #31046 into master

* refs/pull/31046/head:
doc: hide page contents for Ceph Internals

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

test: only compile ceph_test_bmap_alloc_replay WITH_BLUESTORE

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge PR #31206 into master

* refs/pull/31206/head:
qa: test fs:upgrade when running upgrade suite

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/ssh: simplify get_services call for 'mds update'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: make _get_services take service *id* or service *name*

Name is the group, id is the instance.  A bit confusing, but this is what
orchestrator.py uses.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: fix 'service ls' on stopped services

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: move some code around

Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/orchestrator_cli: 'service ls' works for ssh

See 7fc78b63425abd192a82f812958349f2bf079dea merge commit.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/orchestrator_cli: we already completed mds ops for rook+ssh

See f5bc2bd74a05bb486fbbca2de4dbe9a1cacd605c merge.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31270 into master

* refs/pull/31270/head:
ceph-daemon: no extra newline needed in authorized_keys file
ceph-daemon: use client.admin keyring during bootstrap

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #31265 from idryomov/wip-krbd-unmap-msgr1

qa/suites/krbd: run unmap subsuite with msgr1 only

Reviewed-by: Jason Dillaman <dillaman@redhat.com>