doc: note new mgr module error codes

Signed-off-by: John Spray <john.spray@redhat.com>

mon: expose can_run flag in `module ls` + mgr dump

Enable people to see can_run failures and the explanatory
messages (telling them about a missing dependency)
before trying to enable a module.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: add health checks for failed modules

Signed-off-by: John Spray <john.spray@redhat.com>

qa/mgr: add test for command execution errors

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: load command definitions earlier

...and for all modules, not just the active ones.

This enables us to give better feedback to the user
when they try and use a command from a disabled module,
and also fixes the race between enabling a module and
trying to use its commands.

Fixes: http://tracker.ceph.com/issues/21683
Signed-off-by: John Spray <john.spray@redhat.com>

mgr: evaluate `can_run` method on modules

...and transmit the result to the monitor in
our beacon.

Fixes: http://tracker.ceph.com/issues/21502
Signed-off-by: John Spray <john.spray@redhat.com>

mgr: load all modules (not just active ones)

This is to enable us to learn more about the module
before it is enabled, such as whether its can_run method
return true.

We can also use this to enable loading a module's
commands before it is enabled, to give the user
a better response when they try to use a command
belong to a module that is not loaded.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #20054 from ceph/wip-rm22619

ceph-volume: dmcrypt support for lvm

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20083 from liewegas/wip-msgr2-goals

doc/dev/msgr2.rst: a few notes on protocol goals

Reviewed-by: Ricardo Dias <rdias@suse.com>

doc/dev/msgr2.rst: a few notes on protocol goals

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20010 from ceph/wip-rm22653

ceph-volume: adds a --destroy flag to ceph-volume lvm zap

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume add lvm/encryption to the index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add dmcrypt support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add docs for zap sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add docs for list sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume update internal LVM metadata information

Includes dmcrypt information, and updated key names to reflect current
code

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add xenial filestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20020 from mikechristie/wip-doc-mnc-update-iscsi-client

doc: update ceph iscsi kernel and package info

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18323 from thmour/thmour_rbdtrash

rbd: replace trash delay option, add rbd trash purge command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20014 from liewegas/wip-rbd-validate

common/options,librbd/Utils: refactor RBD feature validation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd trash: replace cli delay option, add rbd trash purge command

Replaced the delay argument for the trash move
command with a string acceptable by /bin/date, e.g.:

$ rbd trash move --pool foo --image bar --expires-in "2 weeks"

Added a "rbd trash purge" command that deletes any expired
image from the trash, has also a command to alter the current
expiration date with the "--older-than" argument which accepts
again a valid argument for /bin/date, e.g.:

$rbd trash purge mypool --older-than "2017-08-20"

There is also the "threshold" argument which tries to remove the
oldest trashed images (by deferment end time) until the pool space
is freed up to a percentage point, e.g.:

$ rbd trash purge mypool --threshold 0.9

If mypool uses 1GB it will try to remove trashed images until the
pool usage becomes equal to or lower than 900MB.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

Merge pull request #20005 from jcsp/wip-mgr-minor

mgr: remove a few junk lines

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: remove a few junk lines

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #19988 from liewegas/wip-22673

osd: only exit if *latest* map(s) say we are destroyed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20059 from ceph/wip-rm22427

doc/ceph-volume OSD use the fsid file, not the osd_fsid

Merge pull request #19979 from k0ste/prometheus_metadata

mgr: prometheus: set metadata metrics value to '1' (#22717).

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #19980 from k0ste/prometheus_latency

mgr: prometheus: added osd commit/apply latency metrics (#22718).

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19986 from tchaikov/wip-denc-size-t-featured

denc: should check element's type not 'size_t'

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20028 from mogeb/steady-clock-rados

rados: use WaitInterval()'s return value instead of manual timing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20044 from tchaikov/wip-22419

osd: update store with options after pg is created

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20032 from liewegas/wip-name-shadowing

mon/OSDMonitor: rename outer name declaration to avoid shadowing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20015 from cbodley/wip-crypto--

crypto: remove cryptopp library

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20029 from tchaikov/wip-osx-build

osdc,os,osd: fix build on osx

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20046 from tchaikov/wip-22711

test: reduce the chance to have degraded PGs

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fix ceph-iscsi-config version number

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc/ceph-volume OSD use the fsid file, not the osd_fsid

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add xenial bluestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add dmcrypt support in tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add centos7 filestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add centos7 bluestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add util.prepare unit tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add fixture helpers for call and run

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util allow prepare module to work with encryption keys

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.activate allow encryption setups for luks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare allow encryption setups for luks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process allow silencing output on file logging for sensitive data

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util create a helper module for encryption

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common add dmcrypt flag

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create an encryption doc explaining dmcrypt workflows

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19960 from gohighsec/default_bluestore

doc: update man page to explain ceph-volume support bluestore

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20034 from linuxbox2/wip-rgw-altlatchfix

rgw_file: alternate fix deadlock on lru eviction

Merge pull request #20030 from tchaikov/wip-doc-cxx11

doc/dev/cxx: add C++11 ABI related doc

Merge pull request #19708 from qrGitHub/wip-rgw-unused-variable4

rgw: drop unused variable bucket_instance_ids

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18717 from shinobu-x/sk_mgr_daemonserver_using_get_val

mgr/DaemonServer.cc: [Cleanup] Change to using  get_val template function

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19362 from myoungwon/wip-manifest-rados-op-promote

osd, librados: add a rados op (TIER_PROMOTE)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20045 from shangzhong/doc-cephfs-multimds

doc: redundant "cephfs" when set the "allow_multimds"

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19744 from jan--f/mgr-prometheus-standby-mondown

pybind/mgr/prometheus: add StandbyModule and handle failed MON cluster

Reviewed-by: John Spray <john.spray@redhat.com>

test: reduce the chance to have degraded PGs

in test_mon_osd_misc(), there is good chance that the cluster chooses
to use an unbalanced weight because of the data distribution at that moment.
but this setting could prevent the CRUSH from choosing enough number of
OSDs for test_mon_cephdf_commands(), where 32 PGs are to be created. so
it's more likely that the CRUSH fails to pick enough OSDs for all PGs.
that's why we have a curr_object_copies_rate = 0.5.

so, in this change, pg=pgp=1 is specified for the new pool.

Fixes: http://tracker.ceph.com/issues/22711
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr: add test_prometheus; smoke tests for prometheus module

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/prometheus: tidy up cherrypy engine start and stop

Calling cherrypy.engine.block() in the stanby module results in a failing
mgr failover.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #19965 from ovh/bp-reorder-mimic-fields

osd_types.cc: reorder fields in serialized pg_stat_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19963 from runsisi/wip-cmake

cmake: do not find bzip2/lz4 for rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19944 from badone/wip-better-prepare_command_pool_set-E2BIG-error-message

mon/OSDMonitor: Better prepare_command_pool_set E2BIG error message

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19934 from david-z/wip-rocksdb-dump-mem

kv/RocksDB: get index and filter blocks memory usage

Reviewed-by: Sage Weil <sage@redhat.com>

doc: redundant "cephfs" when set the "allow_multimds"

Signed-off-by: Shangzhong Zhu <zhu.shangzhong@zte.com.cn>

Merge pull request #19948 from tchaikov/wip-standby-module-get-config

pybind/mgr/mgr_module: add default param for MgrStandbyModule.get_con…

Reviewed-by: John Spray <john.spray@redhat.com>

pybind/mgr/mgr_module: add default param for MgrStandbyModule.get_config()

this matches its counterpart of MgrModule.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19863 from renhwztetecs/renhw-wip-common-time

common/time: add time.h for Alpine build

Reviewed-by: John Coyle <dx9err@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19899 from smithfarm/wip-rip-rcceph

build/ops: rpm: rip out rcceph script

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: update store with options after pg is created

Fixes: http://tracker.ceph.com/issues/22419
Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "osd/PG: pass pool options to ObjectStore on pg create"

pg is not created by the underlying store, hence the
osd->store->set_collection_opts() call always returns -ENOENT.

This reverts commit 37d7fd3b21cab055721504e1546c6b1657a400af.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: print a space after __func__ in log message

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19998 from liewegas/wip-22715

fix broken use of streamstream::rdbuf()

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: add ceph iscsi test kernel info

Signed-off-by: Mike Christie <mchristi@redhat.com>

rgw_file: alternate fix deadlock on lru eviction

This change is an alternate fix for two problems found and fixed
by Yao Zongyou <yaozongyou@vip.qq.com>.

The deadlock can be avoided just by not taking it in the recycle
case, which invariantly holds the lock.

The invalidation of the insert iterator by the recyle-path unlink
we'd like to handle as a condition in order to preserve the cached
insertion point optimization we get in the common case.  (The
original behavior was, indeed, incorrect.)

Based on feedback from Yao, removed the RGWFileHandle dtor version
of the unlink check, which I think happened twice.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #19597 from joscollin/wip-lspools-output

mon: align lspools output

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #19936 from linuxbox2/wip-rgw-22121

rgw: URL-decode S3 and Swift object-copy URLs

Merge pull request #20019 from zhouyuan/fix_librbdpy_example

doc: fix librbdpy example

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

doc/dev/cxx: add C++11 ABI related doc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19577 from vshankar/wip-policy-state-machine-rollback

rbd-mirror: rollback state transitions in image policy

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19821 from trociny/wip-nbd-global-options

rbd: properly pass ceph global command line args to subprocess

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19912 from adamemerson/wip-minimax-rbd

librbd: prefer templates to macros

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19937 from trociny/wip-rbd-showmapped

rbd: adjusted "showmapped" JSON and XML formatted output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common: remove typedef for byte

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: remove cryptopp digest impls

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove cryptopp BlockCrypt impls

Signed-off-by: Casey Bodley <cbodley@redhat.com>

auth: remove cryptopp CryptoKeyHandler impl

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: remove cryptopp option

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/options,librbd/Utils: refactor RBD feature validation

Move conversion of RBD features to/from string/uint64_t to helpers, and
use those from the option validator.

Keep the resulting features in integer form to avoid breaking Cinder.
Meh.

Change behavior from prior version:

- fail to parse if any of the specified features is invalid (previously
  we would parse the ones we understood and ignore the others).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19972 from liewegas/wip-mgr-rados

pybind/mgr/mgr_module: make rados handle available to all modules

Reviewed-by: John Spray <john.spray@redhat.com>

mon/OSDMonitor: rename outer name declaration to avoid shadowing

Signed-off-by: Sage Weil <sage@redhat.com>

osdc,os,osd: fix build on osx

on clang on MacOS, size_t is not identical to uint64_t, so we need to do
specify the template parameter if the two parameters' types are
different.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fix librbdpy example

need to connect cluster before opening ioctx

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

rgw: URL-decode S3 and Swift object-copy URLs

Fixes: http://tracker.ceph.com/issues/22121
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

ceph-volume: clean up docstrings for lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: print a message to terminal if --destroy is skipped

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: no need to set PVolume.vg_name explicitly

This will be set automatically in __init__ by looping
through kw.items and creating properties on PVolume
with them.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: switch remove_vg and remove_pv to process.run

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: allow passing a fail_msg param to process.run

This will allow the user to provide a more detailed error message
on why the command failed.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #20011 into master

* refs/pull/20011/head:
mds: fix some typo

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>