build/ops: rpm: override %_smp_mflags if not enough memory

Sometimes the build machine has lots of processor cores and not enough
memory to successfully build Ceph on all of them at once. Calculate
how many parallel build processes we can sustain with the memory we
have and set a lower build parallelism if necessary. Never exceed
the value set by %_smp_mflags even if memory is aplenty.

Credits to Tomáš Chvátal for the original idea and implementation.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: fix Group for rados-objclass-devel subpackage

Fixes RPMLINT warning "non-standard-group Development/Libraries"

Also, the Group: line is only needed for SUSE so put it in an appropriate
distro conditional.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: introduce _remote_tarball_prefix macro

With this macro, we can use a single Source0 line for all supported distros.
RH/CentOS/Fedora needs the prefix, while SUSE builds in the OBS use a local
tarball.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: use %_fillupdir macro in openSUSE/SLES

Replace references to /var/adm/fillup-templates with new %_fillupdir macro

Fixes: https://bugzilla.opensuse.org/show_bug.cgi?id=1069468
Signed-off-by: Richard Brown <rbrown@suse.com>

build/ops: rpm: reconstruct SUSE bconds

Disable java build completely. Enable lttng build on SLES only, and only for
certain architectures.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20083 from liewegas/wip-msgr2-goals

doc/dev/msgr2.rst: a few notes on protocol goals

Reviewed-by: Ricardo Dias <rdias@suse.com>

doc/dev/msgr2.rst: a few notes on protocol goals

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20010 from ceph/wip-rm22653

ceph-volume: adds a --destroy flag to ceph-volume lvm zap

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20020 from mikechristie/wip-doc-mnc-update-iscsi-client

doc: update ceph iscsi kernel and package info

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18323 from thmour/thmour_rbdtrash

rbd: replace trash delay option, add rbd trash purge command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20014 from liewegas/wip-rbd-validate

common/options,librbd/Utils: refactor RBD feature validation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd trash: replace cli delay option, add rbd trash purge command

Replaced the delay argument for the trash move
command with a string acceptable by /bin/date, e.g.:

$ rbd trash move --pool foo --image bar --expires-in "2 weeks"

Added a "rbd trash purge" command that deletes any expired
image from the trash, has also a command to alter the current
expiration date with the "--older-than" argument which accepts
again a valid argument for /bin/date, e.g.:

$rbd trash purge mypool --older-than "2017-08-20"

There is also the "threshold" argument which tries to remove the
oldest trashed images (by deferment end time) until the pool space
is freed up to a percentage point, e.g.:

$ rbd trash purge mypool --threshold 0.9

If mypool uses 1GB it will try to remove trashed images until the
pool usage becomes equal to or lower than 900MB.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

Merge pull request #20005 from jcsp/wip-mgr-minor

mgr: remove a few junk lines

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: remove a few junk lines

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #19988 from liewegas/wip-22673

osd: only exit if *latest* map(s) say we are destroyed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20059 from ceph/wip-rm22427

doc/ceph-volume OSD use the fsid file, not the osd_fsid

Merge pull request #19979 from k0ste/prometheus_metadata

mgr: prometheus: set metadata metrics value to '1' (#22717).

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #19980 from k0ste/prometheus_latency

mgr: prometheus: added osd commit/apply latency metrics (#22718).

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19986 from tchaikov/wip-denc-size-t-featured

denc: should check element's type not 'size_t'

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20028 from mogeb/steady-clock-rados

rados: use WaitInterval()'s return value instead of manual timing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20044 from tchaikov/wip-22419

osd: update store with options after pg is created

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20032 from liewegas/wip-name-shadowing

mon/OSDMonitor: rename outer name declaration to avoid shadowing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20015 from cbodley/wip-crypto--

crypto: remove cryptopp library

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20029 from tchaikov/wip-osx-build

osdc,os,osd: fix build on osx

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20046 from tchaikov/wip-22711

test: reduce the chance to have degraded PGs

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fix ceph-iscsi-config version number

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc/ceph-volume OSD use the fsid file, not the osd_fsid

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19960 from gohighsec/default_bluestore

doc: update man page to explain ceph-volume support bluestore

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20034 from linuxbox2/wip-rgw-altlatchfix

rgw_file: alternate fix deadlock on lru eviction

Merge pull request #20030 from tchaikov/wip-doc-cxx11

doc/dev/cxx: add C++11 ABI related doc

Merge pull request #19708 from qrGitHub/wip-rgw-unused-variable4

rgw: drop unused variable bucket_instance_ids

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18717 from shinobu-x/sk_mgr_daemonserver_using_get_val

mgr/DaemonServer.cc: [Cleanup] Change to using  get_val template function

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19362 from myoungwon/wip-manifest-rados-op-promote

osd, librados: add a rados op (TIER_PROMOTE)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20045 from shangzhong/doc-cephfs-multimds

doc: redundant "cephfs" when set the "allow_multimds"

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19744 from jan--f/mgr-prometheus-standby-mondown

pybind/mgr/prometheus: add StandbyModule and handle failed MON cluster

Reviewed-by: John Spray <john.spray@redhat.com>

test: reduce the chance to have degraded PGs

in test_mon_osd_misc(), there is good chance that the cluster chooses
to use an unbalanced weight because of the data distribution at that moment.
but this setting could prevent the CRUSH from choosing enough number of
OSDs for test_mon_cephdf_commands(), where 32 PGs are to be created. so
it's more likely that the CRUSH fails to pick enough OSDs for all PGs.
that's why we have a curr_object_copies_rate = 0.5.

so, in this change, pg=pgp=1 is specified for the new pool.

Fixes: http://tracker.ceph.com/issues/22711
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr: add test_prometheus; smoke tests for prometheus module

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/prometheus: tidy up cherrypy engine start and stop

Calling cherrypy.engine.block() in the stanby module results in a failing
mgr failover.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #19965 from ovh/bp-reorder-mimic-fields

osd_types.cc: reorder fields in serialized pg_stat_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19963 from runsisi/wip-cmake

cmake: do not find bzip2/lz4 for rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19944 from badone/wip-better-prepare_command_pool_set-E2BIG-error-message

mon/OSDMonitor: Better prepare_command_pool_set E2BIG error message

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19934 from david-z/wip-rocksdb-dump-mem

kv/RocksDB: get index and filter blocks memory usage

Reviewed-by: Sage Weil <sage@redhat.com>

doc: redundant "cephfs" when set the "allow_multimds"

Signed-off-by: Shangzhong Zhu <zhu.shangzhong@zte.com.cn>

Merge pull request #19948 from tchaikov/wip-standby-module-get-config

pybind/mgr/mgr_module: add default param for MgrStandbyModule.get_con…

Reviewed-by: John Spray <john.spray@redhat.com>

pybind/mgr/mgr_module: add default param for MgrStandbyModule.get_config()

this matches its counterpart of MgrModule.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19863 from renhwztetecs/renhw-wip-common-time

common/time: add time.h for Alpine build

Reviewed-by: John Coyle <dx9err@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19899 from smithfarm/wip-rip-rcceph

build/ops: rpm: rip out rcceph script

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: update store with options after pg is created

Fixes: http://tracker.ceph.com/issues/22419
Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "osd/PG: pass pool options to ObjectStore on pg create"

pg is not created by the underlying store, hence the
osd->store->set_collection_opts() call always returns -ENOENT.

This reverts commit 37d7fd3b21cab055721504e1546c6b1657a400af.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: print a space after __func__ in log message

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19998 from liewegas/wip-22715

fix broken use of streamstream::rdbuf()

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: add ceph iscsi test kernel info

Signed-off-by: Mike Christie <mchristi@redhat.com>

rgw_file: alternate fix deadlock on lru eviction

This change is an alternate fix for two problems found and fixed
by Yao Zongyou <yaozongyou@vip.qq.com>.

The deadlock can be avoided just by not taking it in the recycle
case, which invariantly holds the lock.

The invalidation of the insert iterator by the recyle-path unlink
we'd like to handle as a condition in order to preserve the cached
insertion point optimization we get in the common case.  (The
original behavior was, indeed, incorrect.)

Based on feedback from Yao, removed the RGWFileHandle dtor version
of the unlink check, which I think happened twice.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #19597 from joscollin/wip-lspools-output

mon: align lspools output

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #19936 from linuxbox2/wip-rgw-22121

rgw: URL-decode S3 and Swift object-copy URLs

Merge pull request #20019 from zhouyuan/fix_librbdpy_example

doc: fix librbdpy example

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

doc/dev/cxx: add C++11 ABI related doc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19577 from vshankar/wip-policy-state-machine-rollback

rbd-mirror: rollback state transitions in image policy

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19821 from trociny/wip-nbd-global-options

rbd: properly pass ceph global command line args to subprocess

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19912 from adamemerson/wip-minimax-rbd

librbd: prefer templates to macros

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19937 from trociny/wip-rbd-showmapped

rbd: adjusted "showmapped" JSON and XML formatted output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common: remove typedef for byte

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: remove cryptopp digest impls

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove cryptopp BlockCrypt impls

Signed-off-by: Casey Bodley <cbodley@redhat.com>

auth: remove cryptopp CryptoKeyHandler impl

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: remove cryptopp option

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/options,librbd/Utils: refactor RBD feature validation

Move conversion of RBD features to/from string/uint64_t to helpers, and
use those from the option validator.

Keep the resulting features in integer form to avoid breaking Cinder.
Meh.

Change behavior from prior version:

- fail to parse if any of the specified features is invalid (previously
  we would parse the ones we understood and ignore the others).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19972 from liewegas/wip-mgr-rados

pybind/mgr/mgr_module: make rados handle available to all modules

Reviewed-by: John Spray <john.spray@redhat.com>

mon/OSDMonitor: rename outer name declaration to avoid shadowing

Signed-off-by: Sage Weil <sage@redhat.com>

osdc,os,osd: fix build on osx

on clang on MacOS, size_t is not identical to uint64_t, so we need to do
specify the template parameter if the two parameters' types are
different.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fix librbdpy example

need to connect cluster before opening ioctx

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

rgw: URL-decode S3 and Swift object-copy URLs

Fixes: http://tracker.ceph.com/issues/22121
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

ceph-volume: clean up docstrings for lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: print a message to terminal if --destroy is skipped

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: no need to set PVolume.vg_name explicitly

This will be set automatically in __init__ by looping
through kw.items and creating properties on PVolume
with them.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: switch remove_vg and remove_pv to process.run

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: allow passing a fail_msg param to process.run

This will allow the user to provide a more detailed error message
on why the command failed.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #20011 into master

* refs/pull/20011/head:
mds: fix some typo

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18971 from liewegas/wip-pg-scrub-preempt

osd/PG: allow scrub preemption

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume: tests for lvm zap --destroy

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #19476 from qrGitHub/wip-rgw-lack-timeSkewCheck

rgw: add time skew check in function parse_v4_auth_header

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #19826 from joke-lee/rgw_fix_s3website_redirect_location_string_length

rgw: fix s3website redirect location string length

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19918 from fangyuxiangGL/lc-omap

rgw: abort multipart if upload meta object doesn't exist

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20003 from wjwithagen/fix-PG-protected

common: change routines to public access

Reviewed-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: squelch 'id' arg errors

Several commands use 'id' but for other types (e.g., vector<int64_t>).

Signed-off-by: Sage Weil <sage@redhat.com>

use stringsream::str(), not rdbuf()

Fixes: http://tracker.ceph.com/issues/22715
Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: also remove pvs with lvm zap --destroy

Leaving the pv around doesn't keep us from redeploying on
that device because we also wipefs, but explicitly destroying
it makes the output read better.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds an api.lvm.remove_pv method

Used to remove physical volumes

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: remove the -a flag from api.lvm.get_pvs

When we run `pvs -a` it returns all pvs, even ones that
have been recently deleted. Because of this pvs that have just
been zapped with `lvm zap` show up and can cause issues if a device
is zapped more than once.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: docs for the --destroy flag of lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds the --destroy flag to ceph-volume lvm zap

If you use the --destroy flag and are zapping a raw device
or parition then zap will destroy any vgs or lvs it finds on that
device.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds an api.lvm.remove_vg method

This method can be used to remove volume groups

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: expose vg_name in api.lvm.get_api_pvs

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

rados: use WaitInterval()'s return value instead of manual timing

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

Merge pull request #19949 from ceph/wip-bz1498521

ceph-volume: adds --crush-device-class flag for lvm prepare and create

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19997 from Songweibin/wip-fix-typo

doc/rbd: fix typo s/morror/mirror

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19981 from k0ste/dashboard_iops

mgr: dashboard: show per pool IOPS on health page (#22495).

Reviewed-by: John Spray <john.spray@redhat.com>

mgr: dashboard: show per pool IOPS on health page (#22495).

Signed-off-by: Konstantin Shalygin <k0ste@k0ste.ru>

common: change routines to public access

Clang complains tafter recent changes:
/home/jenkins/workspace/ceph-master/src/osd/OSD.cc:8787:8: error: 'DeleteStart' is a protected member of 'PG'
          PG::DeleteStart())));
              ^
/home/jenkins/workspace/ceph-master/src/osd/PG.h:1870:16: note: declared protected here
  TrivialEvent(DeleteStart)
               ^
/home/jenkins/workspace/ceph-master/src/osd/OSD.cc:9136:6: error: 'DeleteSome' is a protected member of 'PG'
        PG::DeleteSome())),
            ^
/home/jenkins/workspace/ceph-master/src/osd/PG.h:1871:16: note: declared protected here
  TrivialEvent(DeleteSome)
               ^
2 errors generated.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr: prometheus: added osd commit/apply latency metrics (#22718).

Signed-off-by: Konstantin Shalygin <k0ste@k0ste.ru>