ceph-volume: adds tests for zap --destroy and --osd-id

These are not added for dmcrypt right now because there is a bug related
to zapping encrypted disks that needs fixed first.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: no need to return osd_id from util.prepare.check_id

Now that osd_id is passed to util.prepare.create_id it doesn't
make any sense to return osd_id from check_id anymore as it's
not being used.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: when reusing an osd ID you must give that id to 'osd new'

If you do not then auth will not be created for the new OSD and the
daemon will not be able to start.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: prepare_* methods do not need to recreate fsid or osd_id

The prepare_filestore and prepare_bluestore methods do not need to
attempt to recreate osd_id or fsid because this happens in prepare
already.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph_volume: modify util.prepare.check_id to handle stdout as a list

process.call returns stdout as a list, not a string

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: tests for util.prepare.check_id

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: return osd_id from util.prepare.check_id if it exists

This also changes this so the osd_id is returned as a string so
an ID of 0 would evaluate to True.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: check to see if an OSD id exists before reusing it

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a prepare util for checking OSD ID existance

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20047 from jcsp/wip-prometheus-qa

qa: add new prometheus test to rados/mgr suite

Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19596 from qrGitHub/wip-rgw-lc-scheduleNextStartTime

rgw: optimize next start time for lifecycle

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #20077 from smithfarm/wip-spdk-fast-forward

spdk: update submodule to more recent upstream

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19235 from jcsp/wip-mgr-can-run

mgr: improved module loading for error reporting etc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19122 from smithfarm/wip-lowmem-builder

build/ops: rpm: set build parallelism based on available memory

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

build/ops: cmake: do not guard certain tools with WITH_TESTS

As a follow-up to d7b493a7108a68302bc0f48337bf3c253a720266 we need to stop guarding

    ceph-osdomap-tool
    ceph-monstore-tool

with WITH_TESTS because they have been moved out of the ceph-test package.

(N.B. ceph-kvstore-tool was also moved out of ceph-test, but apparently never
had the guard.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: let ctest benefit from memory calculation

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: override %_smp_mflags if not enough memory

Sometimes the build machine has lots of processor cores and not enough
memory to successfully build Ceph on all of them at once. Calculate
how many parallel build processes we can sustain with the memory we
have and set a lower build parallelism if necessary. Never exceed
the value set by %_smp_mflags even if memory is aplenty.

Credits to Tomáš Chvátal for the original idea and implementation.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: fix Group for rados-objclass-devel subpackage

Fixes RPMLINT warning "non-standard-group Development/Libraries"

Also, the Group: line is only needed for SUSE so put it in an appropriate
distro conditional.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19384 from ivancich/wip-rgw-lazy-cache-put-on-excl-create

rgw: do not update all gateway caches upon creation of system obj w/ exclusive flag

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa: always dump corrupt messages in rados/upgrade

Previously only a service with debug_ms>=1 would dump
corrupt messages: in an upgrade test we're *alway*
interested in a corrupt message.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: refactor get_modules/list_modules

list_modules is really about searching for them
on disk, so it's now probe_modules and private.

Both methods now return values instead of populating
an argument, since when called they were always writing into
a newly constructed container.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: note new mgr module error codes

Signed-off-by: John Spray <john.spray@redhat.com>

mon: expose can_run flag in `module ls` + mgr dump

Enable people to see can_run failures and the explanatory
messages (telling them about a missing dependency)
before trying to enable a module.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: add health checks for failed modules

Signed-off-by: John Spray <john.spray@redhat.com>

qa/mgr: add test for command execution errors

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: load command definitions earlier

...and for all modules, not just the active ones.

This enables us to give better feedback to the user
when they try and use a command from a disabled module,
and also fixes the race between enabling a module and
trying to use its commands.

Fixes: http://tracker.ceph.com/issues/21683
Signed-off-by: John Spray <john.spray@redhat.com>

mgr: evaluate `can_run` method on modules

...and transmit the result to the monitor in
our beacon.

Fixes: http://tracker.ceph.com/issues/21502
Signed-off-by: John Spray <john.spray@redhat.com>

mgr: load all modules (not just active ones)

This is to enable us to learn more about the module
before it is enabled, such as whether its can_run method
return true.

We can also use this to enable loading a module's
commands before it is enabled, to give the user
a better response when they try to use a command
belong to a module that is not loaded.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #20054 from ceph/wip-rm22619

ceph-volume: dmcrypt support for lvm

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

build/ops: rpm: introduce _remote_tarball_prefix macro

With this macro, we can use a single Source0 line for all supported distros.
RH/CentOS/Fedora needs the prefix, while SUSE builds in the OBS use a local
tarball.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: use %_fillupdir macro in openSUSE/SLES

Replace references to /var/adm/fillup-templates with new %_fillupdir macro

Fixes: https://bugzilla.opensuse.org/show_bug.cgi?id=1069468
Signed-off-by: Richard Brown <rbrown@suse.com>

build/ops: rpm: reconstruct SUSE bconds

Disable java build completely. Enable lttng build on SLES only, and only for
certain architectures.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20083 from liewegas/wip-msgr2-goals

doc/dev/msgr2.rst: a few notes on protocol goals

Reviewed-by: Ricardo Dias <rdias@suse.com>

doc/dev/msgr2.rst: a few notes on protocol goals

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20010 from ceph/wip-rm22653

ceph-volume: adds a --destroy flag to ceph-volume lvm zap

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume add lvm/encryption to the index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add dmcrypt support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add docs for zap sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add docs for list sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume update internal LVM metadata information

Includes dmcrypt information, and updated key names to reflect current
code

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add xenial filestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20020 from mikechristie/wip-doc-mnc-update-iscsi-client

doc: update ceph iscsi kernel and package info

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18323 from thmour/thmour_rbdtrash

rbd: replace trash delay option, add rbd trash purge command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20014 from liewegas/wip-rbd-validate

common/options,librbd/Utils: refactor RBD feature validation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

spdk: update submodule to more recent upstream

This commit fast-forwards the SPDK submodule to upstream commit
93bd60e0de6b161567182d965239735bf467e85c to enable the py3-only build.
(Without said commit, SPDK configure script fails to find the python
interpreter in py3-only environment.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rbd trash: replace cli delay option, add rbd trash purge command

Replaced the delay argument for the trash move
command with a string acceptable by /bin/date, e.g.:

$ rbd trash move --pool foo --image bar --expires-in "2 weeks"

Added a "rbd trash purge" command that deletes any expired
image from the trash, has also a command to alter the current
expiration date with the "--older-than" argument which accepts
again a valid argument for /bin/date, e.g.:

$rbd trash purge mypool --older-than "2017-08-20"

There is also the "threshold" argument which tries to remove the
oldest trashed images (by deferment end time) until the pool space
is freed up to a percentage point, e.g.:

$ rbd trash purge mypool --threshold 0.9

If mypool uses 1GB it will try to remove trashed images until the
pool usage becomes equal to or lower than 900MB.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

qa: assign prometheus ports during selftest

This was throwing IOError("Port 9283 not free on '::'",)
when trying to serve, since merging https://github.com/ceph/ceph/pull/19744

It's because the standbys (on the same node as the active) are
now trying to listen too.

Fixes: https://tracker.ceph.com/issues/22755
Signed-off-by: John Spray <john.spray@redhat.com>

qa: add new prometheus test to rados/mgr suite

Added in https://github.com/ceph/ceph/pull/19744

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #20005 from jcsp/wip-mgr-minor

mgr: remove a few junk lines

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: remove a few junk lines

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #19988 from liewegas/wip-22673

osd: only exit if *latest* map(s) say we are destroyed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20059 from ceph/wip-rm22427

doc/ceph-volume OSD use the fsid file, not the osd_fsid

Merge pull request #19979 from k0ste/prometheus_metadata

mgr: prometheus: set metadata metrics value to '1' (#22717).

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #19980 from k0ste/prometheus_latency

mgr: prometheus: added osd commit/apply latency metrics (#22718).

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19986 from tchaikov/wip-denc-size-t-featured

denc: should check element's type not 'size_t'

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20028 from mogeb/steady-clock-rados

rados: use WaitInterval()'s return value instead of manual timing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20044 from tchaikov/wip-22419

osd: update store with options after pg is created

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20032 from liewegas/wip-name-shadowing

mon/OSDMonitor: rename outer name declaration to avoid shadowing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20015 from cbodley/wip-crypto--

crypto: remove cryptopp library

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20029 from tchaikov/wip-osx-build

osdc,os,osd: fix build on osx

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20046 from tchaikov/wip-22711

test: reduce the chance to have degraded PGs

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fix ceph-iscsi-config version number

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc/ceph-volume OSD use the fsid file, not the osd_fsid

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add xenial bluestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add dmcrypt support in tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add centos7 filestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add centos7 bluestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add util.prepare unit tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add fixture helpers for call and run

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util allow prepare module to work with encryption keys

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.activate allow encryption setups for luks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare allow encryption setups for luks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process allow silencing output on file logging for sensitive data

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util create a helper module for encryption

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common add dmcrypt flag

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create an encryption doc explaining dmcrypt workflows

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19960 from gohighsec/default_bluestore

doc: update man page to explain ceph-volume support bluestore

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20034 from linuxbox2/wip-rgw-altlatchfix

rgw_file: alternate fix deadlock on lru eviction

Merge pull request #20030 from tchaikov/wip-doc-cxx11

doc/dev/cxx: add C++11 ABI related doc

Merge pull request #19708 from qrGitHub/wip-rgw-unused-variable4

rgw: drop unused variable bucket_instance_ids

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18717 from shinobu-x/sk_mgr_daemonserver_using_get_val

mgr/DaemonServer.cc: [Cleanup] Change to using  get_val template function

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19362 from myoungwon/wip-manifest-rados-op-promote

osd, librados: add a rados op (TIER_PROMOTE)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20045 from shangzhong/doc-cephfs-multimds

doc: redundant "cephfs" when set the "allow_multimds"

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19744 from jan--f/mgr-prometheus-standby-mondown

pybind/mgr/prometheus: add StandbyModule and handle failed MON cluster

Reviewed-by: John Spray <john.spray@redhat.com>

test: reduce the chance to have degraded PGs

in test_mon_osd_misc(), there is good chance that the cluster chooses
to use an unbalanced weight because of the data distribution at that moment.
but this setting could prevent the CRUSH from choosing enough number of
OSDs for test_mon_cephdf_commands(), where 32 PGs are to be created. so
it's more likely that the CRUSH fails to pick enough OSDs for all PGs.
that's why we have a curr_object_copies_rate = 0.5.

so, in this change, pg=pgp=1 is specified for the new pool.

Fixes: http://tracker.ceph.com/issues/22711
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr: add test_prometheus; smoke tests for prometheus module

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/prometheus: tidy up cherrypy engine start and stop

Calling cherrypy.engine.block() in the stanby module results in a failing
mgr failover.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #19965 from ovh/bp-reorder-mimic-fields

osd_types.cc: reorder fields in serialized pg_stat_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19963 from runsisi/wip-cmake

cmake: do not find bzip2/lz4 for rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19944 from badone/wip-better-prepare_command_pool_set-E2BIG-error-message

mon/OSDMonitor: Better prepare_command_pool_set E2BIG error message

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19934 from david-z/wip-rocksdb-dump-mem

kv/RocksDB: get index and filter blocks memory usage

Reviewed-by: Sage Weil <sage@redhat.com>

doc: redundant "cephfs" when set the "allow_multimds"

Signed-off-by: Shangzhong Zhu <zhu.shangzhong@zte.com.cn>

Merge pull request #19948 from tchaikov/wip-standby-module-get-config

pybind/mgr/mgr_module: add default param for MgrStandbyModule.get_con…

Reviewed-by: John Spray <john.spray@redhat.com>

pybind/mgr/mgr_module: add default param for MgrStandbyModule.get_config()

this matches its counterpart of MgrModule.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19863 from renhwztetecs/renhw-wip-common-time

common/time: add time.h for Alpine build

Reviewed-by: John Coyle <dx9err@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19899 from smithfarm/wip-rip-rcceph

build/ops: rpm: rip out rcceph script

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: update store with options after pg is created

Fixes: http://tracker.ceph.com/issues/22419
Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "osd/PG: pass pool options to ObjectStore on pg create"

pg is not created by the underlying store, hence the
osd->store->set_collection_opts() call always returns -ENOENT.

This reverts commit 37d7fd3b21cab055721504e1546c6b1657a400af.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: print a space after __func__ in log message

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19998 from liewegas/wip-22715

fix broken use of streamstream::rdbuf()

Reviewed-by: Kefu Chai <kchai@redhat.com>