Merge pull request #19828 from xxhdx1985126/wip_xxh_trackedop_tracking_start

common: mark events of TrackedOp outside its constructor

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20022 from badone/wip-OSDMonitor-error-message-outputs

mon/OSDMonitor: Fix OSDMonitor error message outputs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20115 from liewegas/wip-features

mon: fix structure of 'features' command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20096 from myoungwon/wip-manifest-two-pool-test

src/test/osd: add two pool test for manifest objects

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20018 from smithfarm/wip-python3-only

build/ops: rpm: conditionalize Python 2 availability to enable Ceph build on Python 3-only system

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #20134 from tchaikov/wip-spdk-aarch64

spdk: update SPDK to fix the build failure on aarch64

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

src/test/osd/TestRados: add a comment

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

Merge pull request #20064 from tchaikov/wip-cmake-py3

cmake: enabled py3 only build

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

spdk: enable SPDK on AArch64 by default

Enable SPDK in Ceph on AArch64 by default.

Change-Id: I81ad8e0b860968d62fd2740ae04923f477dab75a
Signed-off-by: Tone Zhang <tone.zhang@arm.com>

spdk: update SPDK to fix the build failure on aarch64

to include 29be88fab6bb026df27c93ee89c61965b3c43133 which addresses a
build failure on aarch64.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: be compatible with py3

so ceph-mgr can be compiled using py3

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build ceph-mgr with specified version of python

* add an option named "MGR_PYTHON_VERSION", so we can build ceph-mgr
  which use py3 for running plugins
* also drop the line to specify the "Python_ADDITIONAL_VERSIONS", because
  2.7 is listed by all the the FindPythonInterp and FindPythonLibs in
  cmake 2.8.12 and up.
* use ${MGR_PYTHON_EXECUTABLE} for holding the path to the python
  interpreter used by mgr. because this variable might be overwritten by
  "find_package(PythonInterp 2 REQUIRED)" when checking for building env
  of pybinding for python2.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add WITH_PYTHON2 option

* add WITH_PYTHON2 option, so we can build python3 bindings only.
* change the default value of WITH_PYTHON3 option to "OFF", as the
option() command in cmake only allow the initial value to be "ON" or
"OFF". we could use a cached string for this option, but i think it
would be more explicit to continue using the option() command.
* fix the installation dir of "ceph_rest_api.py". please note, we still
have a *default* python version, which is specified by the last element
of ${py_vers}. for instance, ${PYTHON_VERSION} will be 3 if ${py_vers}
is 2;3. in this change, 2 is still the default python version, if both
WITH_PYTHON2 and WITH_PYTHON3 are enabled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19922 from Rubab-Syed/performance_counter_browser

mgr/dashboard: performance counter browsers

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20009 from Liuchang0812/fix-22727

mon: do not use per_pool_sum_delta to show recovery summary

Reviewed-by: Varada Kari <varada.kari@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20048 from jcsp/wip-22096

mon: fix mgr using auth_client_required policy

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/thrash/workloads: add paramters to support two pools

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

build/ops: rpm: correct Group for python libraries

Signed-off-by: Nathan Cutler <ncutler@suse.com>

make-dist: add OBS-specific release suffix on SUSE

This is needed to prevent OBS from overriding the nice release number generated
by make-dist.

For more information, see
https://en.opensuse.org/openSUSE:Build_Service_Tips_and_Tricks#How_to_control_a_Release_number_of_resulted_packages

Signed-off-by: Nathan Cutler <ncutler@suse.com>

cmake: empty INSTALL_RPATH for libceph_zstd.so.2.0.0

See 235448879e5 for explanation.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: introduce _python_buildid macro

This eliminates several ugly conditional blocks. This commit also does some
cleanup like dropping the ceph-base runtime dependency on "python", which is
implicit in "python-requests".

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: move distro-conditional make check deps

These were in the wrong section.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: PendingReleaseNotes: python-ceph-compat deprecate/drop

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: python3-ceph-argparse only if Python 2 available

The python3-ceph-argparse package is only necessary in environments where
Python 3 might not be available. For distros where Python 3 has become
standard with Python 2 an optional extension (instead of vice versa as
in the current status quo), the files in this package can (and should)
be included in ceph-common directly.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: package byte-compiled ceph-rest-api

This is only necessary on Python 3-only systems.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: conditionalize Python 2 availability

Not all build targets have Python 2 available. Allow the build
to succeed on pure Python 3 systems.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19994 from runsisi/wip-objecter-cleanup

objecter: minor cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19651 from yanghonggang/master

mon/OSDMonitor.cc: fix expected_num_objects interpret error

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20124 from dillaman/wip-devstack

qa/workunits/rbd: switch devstack to pike release

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20105 from dillaman/wip-22791

librbd: force removal of a snapshot cannot ignore dependent children

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #19320 from vshankar/rbd-mirror-image-map-policy-test

test/rbd-mirror: image map policy test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19974 from shun-s/wip-accelerate-rbd-snap-create-rm-speed

librbd: cache last index position to accelerate snap create/rm

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19996 from Songweibin/wip-rbd-deep-cp

rbd: add deep cp CLI method

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: performance counter browsers

Signed-off-by: Rubab-Syed <rubab.syed21@gmail.com>

qa/workunits/rbd: switch devstack to pike release

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #20060 into master

* refs/pull/20060/head:
ceph-debug-docker.sh: allow specifying branch sha1

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

librbd: force removal of a snapshot cannot ignore dependent children

Fixes: http://tracker.ceph.com/issues/22791
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20047 from jcsp/wip-prometheus-qa

qa: add new prometheus test to rados/mgr suite

Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19596 from qrGitHub/wip-rgw-lc-scheduleNextStartTime

rgw: optimize next start time for lifecycle

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #20077 from smithfarm/wip-spdk-fast-forward

spdk: update submodule to more recent upstream

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: build Boost.python with specified version of python

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19235 from jcsp/wip-mgr-can-run

mgr: improved module loading for error reporting etc

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: fix snap create/rm may taking long time

  fix snap create/rm may taking long time
  http://tracker.ceph.com/issues/22716

Signed-off-by: Song Shun <song.shun3@zte.com.cn>

Merge pull request #19122 from smithfarm/wip-lowmem-builder

build/ops: rpm: set build parallelism based on available memory

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

mon: fix structure of 'features' command

We are dumping an array of objects all called "group"--we need an array
here, not an object (dict) section.

Signed-off-by: Sage Weil <sage@redhat.com>

build/ops: cmake: do not guard certain tools with WITH_TESTS

As a follow-up to d7b493a7108a68302bc0f48337bf3c253a720266 we need to stop guarding

    ceph-osdomap-tool
    ceph-monstore-tool

with WITH_TESTS because they have been moved out of the ceph-test package.

(N.B. ceph-kvstore-tool was also moved out of ceph-test, but apparently never
had the guard.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: let ctest benefit from memory calculation

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: override %_smp_mflags if not enough memory

Sometimes the build machine has lots of processor cores and not enough
memory to successfully build Ceph on all of them at once. Calculate
how many parallel build processes we can sustain with the memory we
have and set a lower build parallelism if necessary. Never exceed
the value set by %_smp_mflags even if memory is aplenty.

Credits to Tomáš Chvátal for the original idea and implementation.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: fix Group for rados-objclass-devel subpackage

Fixes RPMLINT warning "non-standard-group Development/Libraries"

Also, the Group: line is only needed for SUSE so put it in an appropriate
distro conditional.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19384 from ivancich/wip-rgw-lazy-cache-put-on-excl-create

rgw: do not update all gateway caches upon creation of system obj w/ exclusive flag

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa: always dump corrupt messages in rados/upgrade

Previously only a service with debug_ms>=1 would dump
corrupt messages: in an upgrade test we're *alway*
interested in a corrupt message.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: refactor get_modules/list_modules

list_modules is really about searching for them
on disk, so it's now probe_modules and private.

Both methods now return values instead of populating
an argument, since when called they were always writing into
a newly constructed container.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: note new mgr module error codes

Signed-off-by: John Spray <john.spray@redhat.com>

mon: expose can_run flag in `module ls` + mgr dump

Enable people to see can_run failures and the explanatory
messages (telling them about a missing dependency)
before trying to enable a module.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: add health checks for failed modules

Signed-off-by: John Spray <john.spray@redhat.com>

qa/mgr: add test for command execution errors

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: load command definitions earlier

...and for all modules, not just the active ones.

This enables us to give better feedback to the user
when they try and use a command from a disabled module,
and also fixes the race between enabling a module and
trying to use its commands.

Fixes: http://tracker.ceph.com/issues/21683
Signed-off-by: John Spray <john.spray@redhat.com>

mgr: evaluate `can_run` method on modules

...and transmit the result to the monitor in
our beacon.

Fixes: http://tracker.ceph.com/issues/21502
Signed-off-by: John Spray <john.spray@redhat.com>

mgr: load all modules (not just active ones)

This is to enable us to learn more about the module
before it is enabled, such as whether its can_run method
return true.

We can also use this to enable loading a module's
commands before it is enabled, to give the user
a better response when they try to use a command
belong to a module that is not loaded.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #20054 from ceph/wip-rm22619

ceph-volume: dmcrypt support for lvm

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

build/ops: rpm: introduce _remote_tarball_prefix macro

With this macro, we can use a single Source0 line for all supported distros.
RH/CentOS/Fedora needs the prefix, while SUSE builds in the OBS use a local
tarball.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: use %_fillupdir macro in openSUSE/SLES

Replace references to /var/adm/fillup-templates with new %_fillupdir macro

Fixes: https://bugzilla.opensuse.org/show_bug.cgi?id=1069468
Signed-off-by: Richard Brown <rbrown@suse.com>

build/ops: rpm: reconstruct SUSE bconds

Disable java build completely. Enable lttng build on SLES only, and only for
certain architectures.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

src/test/osd/RadosModel: modify SetCunk to support two pool

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

src/test/osd/RadosModel: modify SetRedirect to support two pool

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

src/test/osd/RadosModel: add two pool test

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

Merge pull request #20083 from liewegas/wip-msgr2-goals

doc/dev/msgr2.rst: a few notes on protocol goals

Reviewed-by: Ricardo Dias <rdias@suse.com>

rbd: add deep cp CLI method

Signed-off-by: songweibin <song.weibin@zte.com.cn>

doc/dev/msgr2.rst: a few notes on protocol goals

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20010 from ceph/wip-rm22653

ceph-volume: adds a --destroy flag to ceph-volume lvm zap

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume add lvm/encryption to the index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add dmcrypt support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add docs for zap sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add docs for list sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume update internal LVM metadata information

Includes dmcrypt information, and updated key names to reflect current
code

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add xenial filestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20020 from mikechristie/wip-doc-mnc-update-iscsi-client

doc: update ceph iscsi kernel and package info

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18323 from thmour/thmour_rbdtrash

rbd: replace trash delay option, add rbd trash purge command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20014 from liewegas/wip-rbd-validate

common/options,librbd/Utils: refactor RBD feature validation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr: apply auth_service_required to client conns

Previously was using auth_cluster_required for all
connections, which meant that if someone had
disabled client cephx, they'd get BADAUTHORIZER
from their CLI when it tried to load mgr
command descriptions.

Disabling cephx on the admin CLI is odd, but the mon
tolerates it so the mgr should too.

Fixes: https://tracker.ceph.com/issues/22096
Signed-off-by: John Spray <john.spray@redhat.com>

osdc: build MOSDOp message in Objecter::_send_op

it's more intuitive and consistent

Signed-off-by: runsisi <runsisi@zte.com.cn>

spdk: update submodule to more recent upstream

This commit fast-forwards the SPDK submodule to upstream commit
93bd60e0de6b161567182d965239735bf467e85c to enable the py3-only build.
(Without said commit, SPDK configure script fails to find the python
interpreter in py3-only environment.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rbd trash: replace cli delay option, add rbd trash purge command

Replaced the delay argument for the trash move
command with a string acceptable by /bin/date, e.g.:

$ rbd trash move --pool foo --image bar --expires-in "2 weeks"

Added a "rbd trash purge" command that deletes any expired
image from the trash, has also a command to alter the current
expiration date with the "--older-than" argument which accepts
again a valid argument for /bin/date, e.g.:

$rbd trash purge mypool --older-than "2017-08-20"

There is also the "threshold" argument which tries to remove the
oldest trashed images (by deferment end time) until the pool space
is freed up to a percentage point, e.g.:

$ rbd trash purge mypool --threshold 0.9

If mypool uses 1GB it will try to remove trashed images until the
pool usage becomes equal to or lower than 900MB.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

osdc: remove Objecter::kick_requests which is unused

Signed-off-by: runsisi <runsisi@zte.com.cn>

qa: assign prometheus ports during selftest

This was throwing IOError("Port 9283 not free on '::'",)
when trying to serve, since merging https://github.com/ceph/ceph/pull/19744

It's because the standbys (on the same node as the active) are
now trying to listen too.

Fixes: https://tracker.ceph.com/issues/22755
Signed-off-by: John Spray <john.spray@redhat.com>

qa: add new prometheus test to rados/mgr suite

Added in https://github.com/ceph/ceph/pull/19744

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #20005 from jcsp/wip-mgr-minor

mgr: remove a few junk lines

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: remove a few junk lines

Signed-off-by: John Spray <john.spray@redhat.com>

mon: use pool_sum as param name in PGMapDigest::recovery_summary

the param name of delta_sum of PGMapDigest::recovery_summary() is misleading too.
delta_sum fits the recovery_rate_summary() well, but not recovery_summary().

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #19988 from liewegas/wip-22673

osd: only exit if *latest* map(s) say we are destroyed

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: image map policy tests

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #20059 from ceph/wip-rm22427

doc/ceph-volume OSD use the fsid file, not the osd_fsid

Merge pull request #19979 from k0ste/prometheus_metadata

mgr: prometheus: set metadata metrics value to '1' (#22717).

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #19980 from k0ste/prometheus_latency

mgr: prometheus: added osd commit/apply latency metrics (#22718).

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19986 from tchaikov/wip-denc-size-t-featured

denc: should check element's type not 'size_t'

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20028 from mogeb/steady-clock-rados

rados: use WaitInterval()'s return value instead of manual timing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20044 from tchaikov/wip-22419

osd: update store with options after pg is created

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20032 from liewegas/wip-name-shadowing

mon/OSDMonitor: rename outer name declaration to avoid shadowing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20015 from cbodley/wip-crypto--

crypto: remove cryptopp library

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20029 from tchaikov/wip-osx-build

osdc,os,osd: fix build on osx

Reviewed-by: Adam Emerson <aemerson@redhat.com>