rbd: normalize group command aliases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: dump op features in image info

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/rbd: normalize image group API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: normalize image group C/C++ API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: shorten the generated group image snapshot name

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: properly handle group snap add failure case

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplified retrieving snapshots from image header

Added a new cls 'snapshot_get' method that retrieves all snapshot
specific details to avoid the need to add a new cls method to
retrieve each individual snapshot data field.

The cls 'get_snapshot_namespace' method has been dropped since Luminous
clients will treat that as a user-generated snapshot (and that's all
that is supported by Luminous clients). The cls 'get_snapshot_timestamp'
method has been deprecated and can be removed once <=Luminous clients
are no longer supported.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: automatically manage group op feature on images

When an image is added to a group, set the group op feature
bit and when removed, clear the bit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: renamed image group add/remove/get operations

The methods are used by the luminous release of librbd but are essentially
no-ops since group support was removed from the API. However, we need to
prevent odd interactions once a new client creates a group and uses an
older client to manipulate the image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: add new op feature bits for groups and snap trash

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: removed unused snapshot feature bitmask

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: cleanup helper method visibility

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: group image add/remove ops should notify update

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: remove 'consistency' verbiage from group feature

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: treat empty features environment variable as old format

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20064 from tchaikov/wip-cmake-py3

cmake: enabled py3 only build

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr: be compatible with py3

so ceph-mgr can be compiled using py3

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build ceph-mgr with specified version of python

* add an option named "MGR_PYTHON_VERSION", so we can build ceph-mgr
  which use py3 for running plugins
* also drop the line to specify the "Python_ADDITIONAL_VERSIONS", because
  2.7 is listed by all the the FindPythonInterp and FindPythonLibs in
  cmake 2.8.12 and up.
* use ${MGR_PYTHON_EXECUTABLE} for holding the path to the python
  interpreter used by mgr. because this variable might be overwritten by
  "find_package(PythonInterp 2 REQUIRED)" when checking for building env
  of pybinding for python2.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add WITH_PYTHON2 option

* add WITH_PYTHON2 option, so we can build python3 bindings only.
* change the default value of WITH_PYTHON3 option to "OFF", as the
option() command in cmake only allow the initial value to be "ON" or
"OFF". we could use a cached string for this option, but i think it
would be more explicit to continue using the option() command.
* fix the installation dir of "ceph_rest_api.py". please note, we still
have a *default* python version, which is specified by the last element
of ${py_vers}. for instance, ${PYTHON_VERSION} will be 3 if ${py_vers}
is 2;3. in this change, 2 is still the default python version, if both
WITH_PYTHON2 and WITH_PYTHON3 are enabled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19922 from Rubab-Syed/performance_counter_browser

mgr/dashboard: performance counter browsers

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20009 from Liuchang0812/fix-22727

mon: do not use per_pool_sum_delta to show recovery summary

Reviewed-by: Varada Kari <varada.kari@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20048 from jcsp/wip-22096

mon: fix mgr using auth_client_required policy

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19994 from runsisi/wip-objecter-cleanup

objecter: minor cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19651 from yanghonggang/master

mon/OSDMonitor.cc: fix expected_num_objects interpret error

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20124 from dillaman/wip-devstack

qa/workunits/rbd: switch devstack to pike release

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20105 from dillaman/wip-22791

librbd: force removal of a snapshot cannot ignore dependent children

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #19320 from vshankar/rbd-mirror-image-map-policy-test

test/rbd-mirror: image map policy test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19974 from shun-s/wip-accelerate-rbd-snap-create-rm-speed

librbd: cache last index position to accelerate snap create/rm

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19996 from Songweibin/wip-rbd-deep-cp

rbd: add deep cp CLI method

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: performance counter browsers

Signed-off-by: Rubab-Syed <rubab.syed21@gmail.com>

qa/workunits/rbd: switch devstack to pike release

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #20060 into master

* refs/pull/20060/head:
ceph-debug-docker.sh: allow specifying branch sha1

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

librbd: force removal of a snapshot cannot ignore dependent children

Fixes: http://tracker.ceph.com/issues/22791
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20047 from jcsp/wip-prometheus-qa

qa: add new prometheus test to rados/mgr suite

Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19596 from qrGitHub/wip-rgw-lc-scheduleNextStartTime

rgw: optimize next start time for lifecycle

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #20077 from smithfarm/wip-spdk-fast-forward

spdk: update submodule to more recent upstream

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: build Boost.python with specified version of python

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19235 from jcsp/wip-mgr-can-run

mgr: improved module loading for error reporting etc

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: fix snap create/rm may taking long time

  fix snap create/rm may taking long time
  http://tracker.ceph.com/issues/22716

Signed-off-by: Song Shun <song.shun3@zte.com.cn>

Merge pull request #19122 from smithfarm/wip-lowmem-builder

build/ops: rpm: set build parallelism based on available memory

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

build/ops: cmake: do not guard certain tools with WITH_TESTS

As a follow-up to d7b493a7108a68302bc0f48337bf3c253a720266 we need to stop guarding

    ceph-osdomap-tool
    ceph-monstore-tool

with WITH_TESTS because they have been moved out of the ceph-test package.

(N.B. ceph-kvstore-tool was also moved out of ceph-test, but apparently never
had the guard.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: let ctest benefit from memory calculation

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: override %_smp_mflags if not enough memory

Sometimes the build machine has lots of processor cores and not enough
memory to successfully build Ceph on all of them at once. Calculate
how many parallel build processes we can sustain with the memory we
have and set a lower build parallelism if necessary. Never exceed
the value set by %_smp_mflags even if memory is aplenty.

Credits to Tomáš Chvátal for the original idea and implementation.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: fix Group for rados-objclass-devel subpackage

Fixes RPMLINT warning "non-standard-group Development/Libraries"

Also, the Group: line is only needed for SUSE so put it in an appropriate
distro conditional.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19384 from ivancich/wip-rgw-lazy-cache-put-on-excl-create

rgw: do not update all gateway caches upon creation of system obj w/ exclusive flag

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa: always dump corrupt messages in rados/upgrade

Previously only a service with debug_ms>=1 would dump
corrupt messages: in an upgrade test we're *alway*
interested in a corrupt message.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: refactor get_modules/list_modules

list_modules is really about searching for them
on disk, so it's now probe_modules and private.

Both methods now return values instead of populating
an argument, since when called they were always writing into
a newly constructed container.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: note new mgr module error codes

Signed-off-by: John Spray <john.spray@redhat.com>

mon: expose can_run flag in `module ls` + mgr dump

Enable people to see can_run failures and the explanatory
messages (telling them about a missing dependency)
before trying to enable a module.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: add health checks for failed modules

Signed-off-by: John Spray <john.spray@redhat.com>

qa/mgr: add test for command execution errors

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: load command definitions earlier

...and for all modules, not just the active ones.

This enables us to give better feedback to the user
when they try and use a command from a disabled module,
and also fixes the race between enabling a module and
trying to use its commands.

Fixes: http://tracker.ceph.com/issues/21683
Signed-off-by: John Spray <john.spray@redhat.com>

mgr: evaluate `can_run` method on modules

...and transmit the result to the monitor in
our beacon.

Fixes: http://tracker.ceph.com/issues/21502
Signed-off-by: John Spray <john.spray@redhat.com>

mgr: load all modules (not just active ones)

This is to enable us to learn more about the module
before it is enabled, such as whether its can_run method
return true.

We can also use this to enable loading a module's
commands before it is enabled, to give the user
a better response when they try to use a command
belong to a module that is not loaded.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #20054 from ceph/wip-rm22619

ceph-volume: dmcrypt support for lvm

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

build/ops: rpm: introduce _remote_tarball_prefix macro

With this macro, we can use a single Source0 line for all supported distros.
RH/CentOS/Fedora needs the prefix, while SUSE builds in the OBS use a local
tarball.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: use %_fillupdir macro in openSUSE/SLES

Replace references to /var/adm/fillup-templates with new %_fillupdir macro

Fixes: https://bugzilla.opensuse.org/show_bug.cgi?id=1069468
Signed-off-by: Richard Brown <rbrown@suse.com>

build/ops: rpm: reconstruct SUSE bconds

Disable java build completely. Enable lttng build on SLES only, and only for
certain architectures.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20083 from liewegas/wip-msgr2-goals

doc/dev/msgr2.rst: a few notes on protocol goals

Reviewed-by: Ricardo Dias <rdias@suse.com>

rbd: add deep cp CLI method

Signed-off-by: songweibin <song.weibin@zte.com.cn>

doc/dev/msgr2.rst: a few notes on protocol goals

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20010 from ceph/wip-rm22653

ceph-volume: adds a --destroy flag to ceph-volume lvm zap

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume add lvm/encryption to the index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add dmcrypt support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add docs for zap sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add docs for list sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume update internal LVM metadata information

Includes dmcrypt information, and updated key names to reflect current
code

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add xenial filestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20020 from mikechristie/wip-doc-mnc-update-iscsi-client

doc: update ceph iscsi kernel and package info

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18323 from thmour/thmour_rbdtrash

rbd: replace trash delay option, add rbd trash purge command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20014 from liewegas/wip-rbd-validate

common/options,librbd/Utils: refactor RBD feature validation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr: apply auth_service_required to client conns

Previously was using auth_cluster_required for all
connections, which meant that if someone had
disabled client cephx, they'd get BADAUTHORIZER
from their CLI when it tried to load mgr
command descriptions.

Disabling cephx on the admin CLI is odd, but the mon
tolerates it so the mgr should too.

Fixes: https://tracker.ceph.com/issues/22096
Signed-off-by: John Spray <john.spray@redhat.com>

osdc: build MOSDOp message in Objecter::_send_op

it's more intuitive and consistent

Signed-off-by: runsisi <runsisi@zte.com.cn>

spdk: update submodule to more recent upstream

This commit fast-forwards the SPDK submodule to upstream commit
93bd60e0de6b161567182d965239735bf467e85c to enable the py3-only build.
(Without said commit, SPDK configure script fails to find the python
interpreter in py3-only environment.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rbd trash: replace cli delay option, add rbd trash purge command

Replaced the delay argument for the trash move
command with a string acceptable by /bin/date, e.g.:

$ rbd trash move --pool foo --image bar --expires-in "2 weeks"

Added a "rbd trash purge" command that deletes any expired
image from the trash, has also a command to alter the current
expiration date with the "--older-than" argument which accepts
again a valid argument for /bin/date, e.g.:

$rbd trash purge mypool --older-than "2017-08-20"

There is also the "threshold" argument which tries to remove the
oldest trashed images (by deferment end time) until the pool space
is freed up to a percentage point, e.g.:

$ rbd trash purge mypool --threshold 0.9

If mypool uses 1GB it will try to remove trashed images until the
pool usage becomes equal to or lower than 900MB.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

osdc: remove Objecter::kick_requests which is unused

Signed-off-by: runsisi <runsisi@zte.com.cn>

qa: assign prometheus ports during selftest

This was throwing IOError("Port 9283 not free on '::'",)
when trying to serve, since merging https://github.com/ceph/ceph/pull/19744

It's because the standbys (on the same node as the active) are
now trying to listen too.

Fixes: https://tracker.ceph.com/issues/22755
Signed-off-by: John Spray <john.spray@redhat.com>

qa: add new prometheus test to rados/mgr suite

Added in https://github.com/ceph/ceph/pull/19744

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #20005 from jcsp/wip-mgr-minor

mgr: remove a few junk lines

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: remove a few junk lines

Signed-off-by: John Spray <john.spray@redhat.com>

mon: use pool_sum as param name in PGMapDigest::recovery_summary

the param name of delta_sum of PGMapDigest::recovery_summary() is misleading too.
delta_sum fits the recovery_rate_summary() well, but not recovery_summary().

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #19988 from liewegas/wip-22673

osd: only exit if *latest* map(s) say we are destroyed

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: image map policy tests

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #20059 from ceph/wip-rm22427

doc/ceph-volume OSD use the fsid file, not the osd_fsid

Merge pull request #19979 from k0ste/prometheus_metadata

mgr: prometheus: set metadata metrics value to '1' (#22717).

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #19980 from k0ste/prometheus_latency

mgr: prometheus: added osd commit/apply latency metrics (#22718).

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19986 from tchaikov/wip-denc-size-t-featured

denc: should check element's type not 'size_t'

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20028 from mogeb/steady-clock-rados

rados: use WaitInterval()'s return value instead of manual timing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20044 from tchaikov/wip-22419

osd: update store with options after pg is created

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20032 from liewegas/wip-name-shadowing

mon/OSDMonitor: rename outer name declaration to avoid shadowing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20015 from cbodley/wip-crypto--

crypto: remove cryptopp library

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20029 from tchaikov/wip-osx-build

osdc,os,osd: fix build on osx

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20046 from tchaikov/wip-22711

test: reduce the chance to have degraded PGs

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fix ceph-iscsi-config version number

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc/ceph-volume OSD use the fsid file, not the osd_fsid

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-debug-docker.sh: allow specifying branch sha1

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests.functional add xenial bluestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add dmcrypt support in tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add centos7 filestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add centos7 bluestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>