cls/rbd: add snapshot limit UINT64_MAX test case

Signed-off-by: Chen Pan <chenpan@cmss.chinamobile.com>

Merge pull request #31326 from cbodley/wip-cmake-mgr-cap

cmake: define mgr_cap_obj library when WITH_MGR=OFF

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31015 into master

* refs/pull/31015/head:
mgr/orchestrator: Fix raise_if_exception for Python 3

Reviewed-by: Tim Serong <tserong@suse.com>

Merge PR #31337 into master

* refs/pull/31337/head:
Fixing typo in ceph-daemon script for the output_pub_ssh_key argument

Reviewed-by: Sage Weil <sage@redhat.com>

Fixing typo in ceph-daemon script for the output_pub_ssh_key argument

Signed-off-by: John McGowan <john@steakfest.com>

Merge pull request #31335 from wjwithagen/wjw-fix-objecstore-bluestore

test: move bluestore dependent code under WITH_BLUESTORE

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31312 from changchengx/qa

qa: correct zap disk with ceph-deploy tool

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30791 from kshtsk/wip-python3-no-iteritems-ceph-deploy

tasks/ceph_deploy: get rid of iteritems for python3

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Vasu Kulkarni <vasu@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31209 from wjwithagen/wjw-fix-snprintf

common: fix missing <stdio.h> include

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31184 from pdvian/wip-fix-sign-compare-warning

crush/CrushWrapper.cc: Fix sign compare compiler warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31306 from wjwithagen/wjw-fix-src_test_objectstore_CMakeLists.txt

test: only compile ceph_test_bmap_alloc_replay WITH_BLUESTORE

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31313 from tchaikov/wip/debian/rules/ceph-daemon

debian/rules: run dh_python2 with ceph-daemon

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31238 from tchaikov/wip/install-deps/el8

install-deps.sh: use sepia/lab-extra/8

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge PR #30751 into master

* refs/pull/30751/head:
doc: Document Export Process during Subtree Migrations

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30749 into master

* refs/pull/30749/head:
doc: document mds journal event types

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Douglas Fuller <dfuller@redhat.com>

test: move bluestore dependant code under WITH_BLUESTORE

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #31152 from LongDuncan/feature/mgr_restful_rquest_api_support_multicommand

mgr/restful: requests api adds support multiple commands

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge PR #30938 into master

* refs/pull/30938/head:
mds: Reorganize class members in MDSCacheObject header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28077 into master

* refs/pull/28077/head:
client: optimize rename operation under different quota root

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30365 into master

* refs/pull/30365/head:
cephfs-shell: get rid of "return" at the end of the method
pybind/cephfs.pyx: add NotDirectory exception
cephfs-shell: don't expect both Error and OSEror together
cephfs-shell: print built-in error messages for exceptions
cephfs-shell: use poutput defined with the module
cephfs-shell: use perror() defined within the module
cephfs-shell: don't pass default values in calls to perror
cephfs-shell: align perror's signature with cmd2.Cmd.perror
pybind/cephfs: make actual error message part of strerror of OSError

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge PR #30054 into master

* refs/pull/30054/head:
doc: fs subvolume resize command
qa/tasks: tests for resize subvolume
mgr/volumes: fs subvolume resize command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30402 into master

* refs/pull/30402/head:
client: add warning when cap != in->auth_cap.

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #30990 into master

* refs/pull/30990/head:
mds: Reorganize class members in MDSDaemon header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31207 into master

* refs/pull/31207/head:
qa/tasks: Better handling of thrasher names and __init__ calls

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31137 into master

* refs/pull/31137/head:
msg/DispatchQueue: Set throttle stamp for local_delivery

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31319 into master

* refs/pull/31319/head:
ceph-daemon: absolute path for /bin/install
ceph-daemon: ExecStart=/bin/bash script

Reviewed-by: Alfredo Deza <adeza@redhat.com>

cmake: define mgr_cap_obj library when WITH_MGR=OFF

resolves the cmake error when WITH_MGR=OFF:

  Error evaluating generator expression:

    $<TARGET_OBJECTS:mgr_cap_obj>

  Objects of target "mgr_cap_obj" referenced but no such target exists.
Call Stack (most recent call first):
  src/mon/CMakeLists.txt:34 (add_library)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #31318 into master

* refs/pull/31318/head:
ceph.spec.in: run pathfix.py
ceph-daemon: use /usr/bin/python, not /usr/bin/env python

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31226 into master

* refs/pull/31226/head:
mgr/telemetry: add stats about crush map
mgr/telemetry: add rgw metadata
mgr/telemetry: include fs size (files, bytes, snaps)
mds: report r{files,bytes,snaps} via perfcounters
mgr/telemetry: mds cache stats
mgr/telemetry: add some rbd metadata
mgr/telemetry: note whether osd cluster_network is in use
mgr/telemetry: add host counts
mgr/telemetry: add more pool metadata
mgr/telemetry: remove crush rule name
mgr/telemetry: include min_mon_release and msgr v1 vs v2 addr count
mgr/telemetry: add CephFS metadata

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #25374 into master

* refs/pull/25374/head:
osd: add flag to prevent truncate_seq copy in copy-from operation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31256 into master

* refs/pull/31256/head:
mgr/MgrClient: fix open condition

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/dashboard: AttributeError: 'NoneType' object has no attrib… (#31078)

mgr/dashboard: Fixed AttributeError: 'NoneType' in smart data integration

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph.spec.in: run pathfix.py

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: absolute path for /bin/install

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31320 into master

* refs/pull/31320/head:
ceph-daemon: fix os.mkdir call

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-daemon: fix os.mkdir call

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31314 into master

* refs/pull/31314/head:
mgr/orchestrator_cli: pass default value to "req=False" params

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31315 into master

* refs/pull/31315/head:
ceph-daemon: create ~/.ssh if not exist

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-daemon: ExecStart=/bin/bash script

Not sure why this matters, but it doesn't work on centos7 without the
/bin/bash part.

https://stackoverflow.com/questions/45776003/fixing-a-systemd-service-203-exec-failure-no-such-file-or-directory

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: create ~/.ssh if not exist

there is chance that /root/.ssh does not exist yet, when we are trying
to add key to authorized_keys.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/telemetry: add stats about crush map

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: use /usr/bin/python, not /usr/bin/env python

This works around noise from the rpm build:

*** ERROR: ambiguous python shebang in /usr/sbin/ceph-daemon: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31045 from tchaikov/wip/librados/no-more-ack-commit

librados,test,rgw: cleanups to deprecate safe_cb related functions

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31311 from tchaikov/wip/rbd/cleanup

librbd: always initialize local variables

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30912 from dillaman/wip-42265

mgr: add 'rbd' profiles to support 'rbd_support' module commands

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31244 from changchengx/deploy_doc

doc: update with osd addition

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31196 from liewegas/wip-osdmon-iec

mon/OSDMonitor: expand iec_options for 'osd pool set'

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: use new aio_create_completion()

Signed-off-by: Kefu Chai <kchai@redhat.com>

journal: use new aio_create_completion()

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librados_test_stub: implement aio_create_completion2() and friends

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: use new aio_create_completion()

to silence warnings like:

src/librbd/Utils.h:139:48: warning: 'static librados::v14_2_0::AioCompletion* librados::v14_2_0::Rados::aio_create_completion(void*, librados::callback_t, librados::callback_t)' is deprecated
[-Wdeprecated-declarations]
   return librados::Rados::aio_create_completion(
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
     obj, &detail::rados_callback<T, MF>, nullptr);
     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/libradosstriper: use new aio_create_completion()

to silence -Wdeprecated-declarations warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls/journal: s/create_completion/create_completion2/

src/cls/journal/cls_journal_client.cc:154:72: warning: 'static librados::v14_2_0::AioCompletion* librados::v14_2_0::Rados::aio_create_completion(void*, librados::callback_t, librados::callback_t)' is
deprecated [-Wdeprecated-declarations]
       librados::Rados::aio_create_completion(this, rados_callback, NULL);
                                                                        ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: s/create_completion/create_completion2/

to silence the warnings like

s_user_client.cc:151:77: warning: 'static librados::v14_2_0::AioCompletion* librados::v14_2_0::Rados::aio_create_completion(void*, librados::callback_t, librados::callback_t)' is deprecated
[-Wdeprecated-declarations]
   AioCompletion *c = librados::Rados::aio_create_completion(NULL, NULL, NULL);
                                                                             ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: s/aio_create_completion/aio_create_completion2/

the former is deprecated

Signed-off-by: Kefu Chai <kchai@redhat.com>

libradosstriper: s/create_completion/create_completion2/

replace `librados::Rados::aio_create_completion(arg, complete_cb,
safe_cb)` with `librados::Rados::aio_create_completion(arg,
complete_cb)`

the former is deprecated

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools: s/rados_aio_create_completion/rados_aio_create_completion2/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: s/rados_aio_create_completion/rados_aio_create_completion2/

the former is deprecated and replaced with the latter

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: s/wait_for_safe/wait_for_complete/

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: s/wait_for_safe/wait_for_complete/

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: s/wait_for_safe/wait_for_complete/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/ceph_kvstorebench: s/wait_for_safe/wait_for_complete/

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rados: s/wait_for_safe/wait_for_complete/

wait_for_safe() and friend are deprecated, so use wait_for_complete()
instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator_cli: pass default value to "req=False" params

in mds_add() command, both "num" and "hosts" are optional, so they
should have default values of "None". this change was introduced in
c35db383.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian/rules: run dh_python2 with ceph-daemon

to set substitute variable of `${python:Depends}` for this package

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: correct zap disk with ceph-deploy tool

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: update with osd addition

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

crush/CrushWrapper.cc: Fix sign compare compiler warning

introduced by 712a39e5c9d

Signed-off-by: Prashant D <pdhange@redhat.com>

librbd: always initialize local variables

silence warnings like:

src/librbd/io/ImageRequestWQ.cc:576:7: warning: 'length' may be used uninitialized in this function [-Wmaybe-uninitialized]
       if (block_overlapping_io(&m_in_flight_extents, offset, length)) {
       ^~
...
src/librbd/io/ImageRequestWQ.cc:935:12: note: 'length' was declared here
   uint64_t length;
            ^~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #31225 into master

* refs/pull/31225/head:
mgr/telemetry: send device telemetry via per-host POST to device endpoint

Reviewed-by: Dan Mick <dmick@redhat.com>

pybind/rados: do not use wait_for_safe() anymore

in librados, wait_for_safe() is an alias of wait_for_complete(). so in
this change, wait_for_safe() calls wait_for_complete(), and the callers
of wait_for_safe() will just call wait_for_complete(), instead of
calling both of them.

also, rados_aio_create_completion() is replaced with
rados_aio_create_completion2().

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: add Rados::aio_create_completion(arg, cb)

which does not take safe_cb, and will take the place of
Rados::aio_create_completion(arg, cb_complete, cb_safe)

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: add rados_aio_create_completion2()

rados_aio_create_completion2() is almost identical to
rados_aio_create_completion(). the only difference is that the former
does not accept the `safe` callback. as it's an alias of `complete`
callback since af01db76f63a22178509b402776da54164af1eb0

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: mark wait_for_safe() functions deprecated

safe is an alias of completed since
af01db76f63a22178509b402776da54164af1eb0. there is no need to have
two callbacks for tracking the state of an async operation anymore.

in this change, wait_for_safe() and friends are marked "deprecated"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #31046 into master

* refs/pull/31046/head:
doc: hide page contents for Ceph Internals

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

test: only compile ceph_test_bmap_alloc_replay WITH_BLUESTORE

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge PR #31206 into master

* refs/pull/31206/head:
qa: test fs:upgrade when running upgrade suite

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #31270 into master

* refs/pull/31270/head:
ceph-daemon: no extra newline needed in authorized_keys file
ceph-daemon: use client.admin keyring during bootstrap

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #31265 from idryomov/wip-krbd-unmap-msgr1

qa/suites/krbd: run unmap subsuite with msgr1 only

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: AttributeError: 'NoneType' object has no attribute 'items' in smart data integration

Fixes: https://tracker.ceph.com/issues/42443
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: Remove convertion to seconds (#31262)

mgr/dashboard: Remove convertion to seconds

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ni-Feng Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: edit/clone/copy rbd image after its data is rec… (#31245)

mgr/dashboard: edit/clone/copy rbd image after its data is received

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31247 from smithfarm/wip-bri-regression

backport-resolve-issue: "recognize that Target version is populated" and "prune duplicate URLs"

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/MgrClient: fix open condition

Do not open a mgr daemon session if we are a client using a non-client
key to authenticate.  This is a bit of a corner case, but it can
happen when using the mon. key do bootstrap functions.

This aligns the MgrClient condition with the mgr/DaemonServer.cc
handle_report condition, which rejects non-daemon client connections
based on the connection entity type, not the auth identity.

Fixes: https://tracker.ceph.com/issues/42566
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/telemetry: add rgw metadata

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31231 into master

* refs/pull/31231/head:
qa: add tests for CephFS admin commands
mds/MDSRank: report state to mgr as mds id, not rank

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-daemon: no extra newline needed in authorized_keys file

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: use client.admin keyring during bootstrap

It's usually okay to use the mon. key for CLI commands, except we had a
mgr but that prevented you from issuing mgr commands correctly.  We have
the new client.admin key available, so use that instead.

Update tests to not --skip-ssh (now that it doesn't hang).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31264 into master

* refs/pull/31264/head:
ceph-daemon: deal with lack of TemporaryDirectory on py2
ceph-daemon: make StringIO usage py2 compatible
ceph-daemon: make ConfigParser py2 compatible

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa/suites/krbd: run unmap subsuite with msgr1 only

pre-single-major.yaml kernel doesn't have any of the monitor client
fixes that came in 4.6.  If the connection is closed, it closes the
session and retries only after 10 seconds.  On top of that, there is
nothing to prevent it from picking the same monitor when reconnecting.
This means that when given both v1 and v2 ports (which look like two
different monitors), it is susceptible to mount_timeout (60 seconds):

  $ sudo rbd map img
  rbd: sysfs write failed
  In some cases useful info is found in syslog - try "dmesg | tail".
  rbd: map failed: (5) Input/output error

  [  822.242313] libceph: mon0 172.21.15.132:3300 socket closed (con state CONNECTING)
  [  832.265494] libceph: mon0 172.21.15.132:3300 socket closed (con state CONNECTING)
  [  842.296175] libceph: mon0 172.21.15.132:3300 socket closed (con state CONNECTING)
  [  852.326924] libceph: mon0 172.21.15.132:3300 socket closed (con state CONNECTING)
  [  862.357611] libceph: mon0 172.21.15.132:3300 socket closed (con state CONNECTING)
  [  872.388373] libceph: mon0 172.21.15.132:3300 socket closed (con state CONNECTING)
  [  882.676136] libceph: mon0 172.21.15.132:3300 socket closed (con state CONNECTING)

Unlike newer kernels that return ETIMEDOUT, it returns EIO.

Newer kernels are much more aggressive about retries and will pick
a different monitor when reconnecting, hence they are always able to
establish the session in time.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: add tests for CephFS admin commands

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-daemon: deal with lack of TemporaryDirectory on py2

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: make StringIO usage py2 compatible

Otherwise,

Traceback (most recent call last):
  File "../src/ceph-daemon", line 1698, in <module>
    r = args.func()
  File "../src/ceph-daemon", line 767, in command_bootstrap
    with StringIO() as f:
AttributeError: StringIO instance has no attribute '__exit__'

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: make ConfigParser py2 compatible

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Remove convertion to seconds

This is no longer need since the values are already in seconds.

Fixes: https://tracker.ceph.com/issues/42567
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #31195 into master

* refs/pull/31195/head:
vstart: Fix /dev/tty No such device or address

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31161 from trociny/wip-42488

rbd-mirror: mirrored clone should be same format

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31149 from wonderpow/master1

cls/rbd: add snapshot_add raise -ESTALE test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31140 from Gangbiao/add-pybind-test-case

test/pybind: add create_snap rasie ImageExists test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30994 from JackSlateur/rbd_merged_du

rbd: add --merge to disk-usage

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>