mgr: be compatible with py3

so ceph-mgr can be compiled using py3

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build ceph-mgr with specified version of python

* add an option named "MGR_PYTHON_VERSION", so we can build ceph-mgr
  which use py3 for running plugins
* also drop the line to specify the "Python_ADDITIONAL_VERSIONS", because
  2.7 is listed by all the the FindPythonInterp and FindPythonLibs in
  cmake 2.8.12 and up.
* use ${MGR_PYTHON_EXECUTABLE} for holding the path to the python
  interpreter used by mgr. because this variable might be overwritten by
  "find_package(PythonInterp 2 REQUIRED)" when checking for building env
  of pybinding for python2.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add WITH_PYTHON2 option

* add WITH_PYTHON2 option, so we can build python3 bindings only.
* change the default value of WITH_PYTHON3 option to "OFF", as the
option() command in cmake only allow the initial value to be "ON" or
"OFF". we could use a cached string for this option, but i think it
would be more explicit to continue using the option() command.
* fix the installation dir of "ceph_rest_api.py". please note, we still
have a *default* python version, which is specified by the last element
of ${py_vers}. for instance, ${PYTHON_VERSION} will be 3 if ${py_vers}
is 2;3. in this change, 2 is still the default python version, if both
WITH_PYTHON2 and WITH_PYTHON3 are enabled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build Boost.python with specified version of python

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19235 from jcsp/wip-mgr-can-run

mgr: improved module loading for error reporting etc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19122 from smithfarm/wip-lowmem-builder

build/ops: rpm: set build parallelism based on available memory

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

build/ops: cmake: do not guard certain tools with WITH_TESTS

As a follow-up to d7b493a7108a68302bc0f48337bf3c253a720266 we need to stop guarding

    ceph-osdomap-tool
    ceph-monstore-tool

with WITH_TESTS because they have been moved out of the ceph-test package.

(N.B. ceph-kvstore-tool was also moved out of ceph-test, but apparently never
had the guard.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: let ctest benefit from memory calculation

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: override %_smp_mflags if not enough memory

Sometimes the build machine has lots of processor cores and not enough
memory to successfully build Ceph on all of them at once. Calculate
how many parallel build processes we can sustain with the memory we
have and set a lower build parallelism if necessary. Never exceed
the value set by %_smp_mflags even if memory is aplenty.

Credits to Tomáš Chvátal for the original idea and implementation.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: fix Group for rados-objclass-devel subpackage

Fixes RPMLINT warning "non-standard-group Development/Libraries"

Also, the Group: line is only needed for SUSE so put it in an appropriate
distro conditional.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19384 from ivancich/wip-rgw-lazy-cache-put-on-excl-create

rgw: do not update all gateway caches upon creation of system obj w/ exclusive flag

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa: always dump corrupt messages in rados/upgrade

Previously only a service with debug_ms>=1 would dump
corrupt messages: in an upgrade test we're *alway*
interested in a corrupt message.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: refactor get_modules/list_modules

list_modules is really about searching for them
on disk, so it's now probe_modules and private.

Both methods now return values instead of populating
an argument, since when called they were always writing into
a newly constructed container.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: note new mgr module error codes

Signed-off-by: John Spray <john.spray@redhat.com>

mon: expose can_run flag in `module ls` + mgr dump

Enable people to see can_run failures and the explanatory
messages (telling them about a missing dependency)
before trying to enable a module.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: add health checks for failed modules

Signed-off-by: John Spray <john.spray@redhat.com>

qa/mgr: add test for command execution errors

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: load command definitions earlier

...and for all modules, not just the active ones.

This enables us to give better feedback to the user
when they try and use a command from a disabled module,
and also fixes the race between enabling a module and
trying to use its commands.

Fixes: http://tracker.ceph.com/issues/21683
Signed-off-by: John Spray <john.spray@redhat.com>

mgr: evaluate `can_run` method on modules

...and transmit the result to the monitor in
our beacon.

Fixes: http://tracker.ceph.com/issues/21502
Signed-off-by: John Spray <john.spray@redhat.com>

mgr: load all modules (not just active ones)

This is to enable us to learn more about the module
before it is enabled, such as whether its can_run method
return true.

We can also use this to enable loading a module's
commands before it is enabled, to give the user
a better response when they try to use a command
belong to a module that is not loaded.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #20054 from ceph/wip-rm22619

ceph-volume: dmcrypt support for lvm

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

build/ops: rpm: introduce _remote_tarball_prefix macro

With this macro, we can use a single Source0 line for all supported distros.
RH/CentOS/Fedora needs the prefix, while SUSE builds in the OBS use a local
tarball.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: use %_fillupdir macro in openSUSE/SLES

Replace references to /var/adm/fillup-templates with new %_fillupdir macro

Fixes: https://bugzilla.opensuse.org/show_bug.cgi?id=1069468
Signed-off-by: Richard Brown <rbrown@suse.com>

build/ops: rpm: reconstruct SUSE bconds

Disable java build completely. Enable lttng build on SLES only, and only for
certain architectures.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20083 from liewegas/wip-msgr2-goals

doc/dev/msgr2.rst: a few notes on protocol goals

Reviewed-by: Ricardo Dias <rdias@suse.com>

doc/dev/msgr2.rst: a few notes on protocol goals

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20010 from ceph/wip-rm22653

ceph-volume: adds a --destroy flag to ceph-volume lvm zap

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume add lvm/encryption to the index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add dmcrypt support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add docs for zap sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add docs for list sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume update internal LVM metadata information

Includes dmcrypt information, and updated key names to reflect current
code

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add xenial filestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20020 from mikechristie/wip-doc-mnc-update-iscsi-client

doc: update ceph iscsi kernel and package info

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18323 from thmour/thmour_rbdtrash

rbd: replace trash delay option, add rbd trash purge command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20014 from liewegas/wip-rbd-validate

common/options,librbd/Utils: refactor RBD feature validation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd trash: replace cli delay option, add rbd trash purge command

Replaced the delay argument for the trash move
command with a string acceptable by /bin/date, e.g.:

$ rbd trash move --pool foo --image bar --expires-in "2 weeks"

Added a "rbd trash purge" command that deletes any expired
image from the trash, has also a command to alter the current
expiration date with the "--older-than" argument which accepts
again a valid argument for /bin/date, e.g.:

$rbd trash purge mypool --older-than "2017-08-20"

There is also the "threshold" argument which tries to remove the
oldest trashed images (by deferment end time) until the pool space
is freed up to a percentage point, e.g.:

$ rbd trash purge mypool --threshold 0.9

If mypool uses 1GB it will try to remove trashed images until the
pool usage becomes equal to or lower than 900MB.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

Merge pull request #20005 from jcsp/wip-mgr-minor

mgr: remove a few junk lines

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: remove a few junk lines

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #19988 from liewegas/wip-22673

osd: only exit if *latest* map(s) say we are destroyed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20059 from ceph/wip-rm22427

doc/ceph-volume OSD use the fsid file, not the osd_fsid

Merge pull request #19979 from k0ste/prometheus_metadata

mgr: prometheus: set metadata metrics value to '1' (#22717).

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #19980 from k0ste/prometheus_latency

mgr: prometheus: added osd commit/apply latency metrics (#22718).

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19986 from tchaikov/wip-denc-size-t-featured

denc: should check element's type not 'size_t'

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20028 from mogeb/steady-clock-rados

rados: use WaitInterval()'s return value instead of manual timing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20044 from tchaikov/wip-22419

osd: update store with options after pg is created

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20032 from liewegas/wip-name-shadowing

mon/OSDMonitor: rename outer name declaration to avoid shadowing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20015 from cbodley/wip-crypto--

crypto: remove cryptopp library

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20029 from tchaikov/wip-osx-build

osdc,os,osd: fix build on osx

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20046 from tchaikov/wip-22711

test: reduce the chance to have degraded PGs

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fix ceph-iscsi-config version number

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc/ceph-volume OSD use the fsid file, not the osd_fsid

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add xenial bluestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add dmcrypt support in tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add centos7 filestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add centos7 bluestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add util.prepare unit tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add fixture helpers for call and run

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util allow prepare module to work with encryption keys

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.activate allow encryption setups for luks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare allow encryption setups for luks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process allow silencing output on file logging for sensitive data

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util create a helper module for encryption

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common add dmcrypt flag

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create an encryption doc explaining dmcrypt workflows

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19960 from gohighsec/default_bluestore

doc: update man page to explain ceph-volume support bluestore

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20034 from linuxbox2/wip-rgw-altlatchfix

rgw_file: alternate fix deadlock on lru eviction

Merge pull request #20030 from tchaikov/wip-doc-cxx11

doc/dev/cxx: add C++11 ABI related doc

Merge pull request #19708 from qrGitHub/wip-rgw-unused-variable4

rgw: drop unused variable bucket_instance_ids

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18717 from shinobu-x/sk_mgr_daemonserver_using_get_val

mgr/DaemonServer.cc: [Cleanup] Change to using  get_val template function

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19362 from myoungwon/wip-manifest-rados-op-promote

osd, librados: add a rados op (TIER_PROMOTE)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20045 from shangzhong/doc-cephfs-multimds

doc: redundant "cephfs" when set the "allow_multimds"

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19744 from jan--f/mgr-prometheus-standby-mondown

pybind/mgr/prometheus: add StandbyModule and handle failed MON cluster

Reviewed-by: John Spray <john.spray@redhat.com>

test: reduce the chance to have degraded PGs

in test_mon_osd_misc(), there is good chance that the cluster chooses
to use an unbalanced weight because of the data distribution at that moment.
but this setting could prevent the CRUSH from choosing enough number of
OSDs for test_mon_cephdf_commands(), where 32 PGs are to be created. so
it's more likely that the CRUSH fails to pick enough OSDs for all PGs.
that's why we have a curr_object_copies_rate = 0.5.

so, in this change, pg=pgp=1 is specified for the new pool.

Fixes: http://tracker.ceph.com/issues/22711
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr: add test_prometheus; smoke tests for prometheus module

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/prometheus: tidy up cherrypy engine start and stop

Calling cherrypy.engine.block() in the stanby module results in a failing
mgr failover.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #19965 from ovh/bp-reorder-mimic-fields

osd_types.cc: reorder fields in serialized pg_stat_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19963 from runsisi/wip-cmake

cmake: do not find bzip2/lz4 for rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19944 from badone/wip-better-prepare_command_pool_set-E2BIG-error-message

mon/OSDMonitor: Better prepare_command_pool_set E2BIG error message

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19934 from david-z/wip-rocksdb-dump-mem

kv/RocksDB: get index and filter blocks memory usage

Reviewed-by: Sage Weil <sage@redhat.com>

doc: redundant "cephfs" when set the "allow_multimds"

Signed-off-by: Shangzhong Zhu <zhu.shangzhong@zte.com.cn>

Merge pull request #19948 from tchaikov/wip-standby-module-get-config

pybind/mgr/mgr_module: add default param for MgrStandbyModule.get_con…

Reviewed-by: John Spray <john.spray@redhat.com>

pybind/mgr/mgr_module: add default param for MgrStandbyModule.get_config()

this matches its counterpart of MgrModule.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19863 from renhwztetecs/renhw-wip-common-time

common/time: add time.h for Alpine build

Reviewed-by: John Coyle <dx9err@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19899 from smithfarm/wip-rip-rcceph

build/ops: rpm: rip out rcceph script

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: update store with options after pg is created

Fixes: http://tracker.ceph.com/issues/22419
Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "osd/PG: pass pool options to ObjectStore on pg create"

pg is not created by the underlying store, hence the
osd->store->set_collection_opts() call always returns -ENOENT.

This reverts commit 37d7fd3b21cab055721504e1546c6b1657a400af.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: print a space after __func__ in log message

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19998 from liewegas/wip-22715

fix broken use of streamstream::rdbuf()

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: add ceph iscsi test kernel info

Signed-off-by: Mike Christie <mchristi@redhat.com>

rgw_file: alternate fix deadlock on lru eviction

This change is an alternate fix for two problems found and fixed
by Yao Zongyou <yaozongyou@vip.qq.com>.

The deadlock can be avoided just by not taking it in the recycle
case, which invariantly holds the lock.

The invalidation of the insert iterator by the recyle-path unlink
we'd like to handle as a condition in order to preserve the cached
insertion point optimization we get in the common case.  (The
original behavior was, indeed, incorrect.)

Based on feedback from Yao, removed the RGWFileHandle dtor version
of the unlink check, which I think happened twice.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #19597 from joscollin/wip-lspools-output

mon: align lspools output

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #19936 from linuxbox2/wip-rgw-22121

rgw: URL-decode S3 and Swift object-copy URLs

Merge pull request #20019 from zhouyuan/fix_librbdpy_example

doc: fix librbdpy example

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

doc/dev/cxx: add C++11 ABI related doc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19577 from vshankar/wip-policy-state-machine-rollback

rbd-mirror: rollback state transitions in image policy

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19821 from trociny/wip-nbd-global-options

rbd: properly pass ceph global command line args to subprocess

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19912 from adamemerson/wip-minimax-rbd

librbd: prefer templates to macros

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19937 from trociny/wip-rbd-showmapped

rbd: adjusted "showmapped" JSON and XML formatted output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common: remove typedef for byte

Signed-off-by: Casey Bodley <cbodley@redhat.com>