common/config: change md_config_obs_impl's template param to ConfigProxy

we were using LockPolicy as this tempalte class's template parameter,
but turns out ConfigProxy is a better parameter for it. as ConfigProxy is
"closer" to md_config_obs_impl than LockPolicy is.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: change md_config_impl<> back to plain class

now that all of the variables that change depending on seastar/alien
have been extracted out of md_config_impl<>, there is no need to make it
a template anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm,install-deps,cmake: install {c-ares,fmt}-devel

* ceph.spec.in: c-ares-devel and fmt-devel are installed as
  dependencies fo seastar.
* install-deps.sh: libfmt-dev is missing in xenial, so we only
  install libc-ares-dev for ubuntu in install-deps.sh .
* cmake: build libfmt if it's not found.

Signed-off-by: Kefu Chai <kchai@redhat.com>

fmt: add libfmt submodule

libfmt is a library for fast printf() support in C++, it is used by
seastar. but it's missing in xenial. so we need to ship it in our dist
tarball and build if if it's not found in system.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: use the modernized cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: sync with upstream

to pick up the commit of 67bb775e, which marks
foreign_ptr::get_owner_shard() const.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: mark do_argv_commands() const

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: extract ObserverMgr from md_config_impl<>

crimson will have sharded<PGShard> (name may vary in future), and
underlying ObjectStore instances. they will be consumers of crimson
ConfigProxy, and will register them as an observer of config changes.
but as always, in crimson osd, world is sharded, each ConfigProxy will
only take care of the sharded services living on the shard which it is
in charge of. in other words, unlike alien's ConfigProxy, the
"observers" data structure is not shared, they are sharded as well. so
we need to extract the observers out of md_config_impl<>, and move it
into ConfigProxy.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: extract "changed" from md_config_impl<> to ConfigValues

for the same reason why we moved ObserverMgr out of md_config_impl<>,
"changed" cannot be shared among shards, we should move "changed" out of
md_config_impl. we can put "changed" into ConfigValues or
ConfigProxy, IMHO, it would be simpler if "changed" resides in
ConfigValues: less fragmented this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: move _apply_changes() calls up into ConfigProxy

see also the previous the commit, to reuse injectargs() and
set_mon_vals() in seastar's ConfigProxy, we need to move apply_changes()
call out of them, as this call is implemented differently in seastar's
ConfigProxy.

finalize_reexpand_meta() is not used by OSD at this moment, so we leave
it as it is for now.

* move _apply_changes() calls up into ConfigProxy
* move update_legacy_vals() out of _apply_changes() into the
callers of the latter. as it changes "values", in seastar's ConfigProxy,
it belongs to the first step of changing a config value

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: promote lock from md_config_t to ConfigProxy

seastar's ConfigProxy and alien's ConfigProxy follow different threading
models and expose different methods. the former updates a setting with 3
steps:
1. create a local copy of current setting, and apply the proposed change
   to the copy
2. populate the updated change with a foreign_ptr<> to all shards
   (including itself)
3. on each shards, call apply_changes() to get the interested observers
   updated, please note, apply_changes() should only update the local
   observers on current shard.

while the alien's ConfigProxy do all the job in a single synchronized
call,
but we can split it into a finer-grained steps:
1. apply the proposed change in-place
2. apply_changes() to get the interested observers updated.

so, to reuse the code across these two implementations, for instance,
set_mon_vals() will be implemented in ConfigProxy instead, so we can
have different behavior in different ConfigProxy classes. if we keep
using the existing single-piece md_config_t::set_mon_vals(), we have no
chance to differentiate the apply_changes() for seastar port. but the
alien implementation requires a grand lock protecting set_val() and
apply_changes(), so we have to move the lock from md_config_t up to
ConfigProxy. it's also simpler this way, as we don't need an extra layer
to have a dummy Mutex for seastar's ConfigProxy. as only the alien's
ConfigProxy requires the lock.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: do not use unique_ptr<> for config

it's not necessary to do so.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22978 from tchaikov/wip-docker-test-cleanup

test/docker-test-helper.sh: move "cp .git/HEAD" out of loop

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23194 from alfredodeza/wip-lvm-docs

ceph-volume: expand on why ceph-disk was replaced

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #23029 from dillaman/wip-24909

osd: do not treat an IO hint as an IOP for PG stats

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23186 from smithfarm/wip-4640

doc/man: mention import and export commands in rados manpage

doc/ceph-volume expand on the ceph-disk replacement reasons

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume reference the new ceph-disk-replacement section

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man: mention import and export commands in rados manpage

Fixes: http://tracker.ceph.com/issues/4640
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23114 from smithfarm/wip-24800

doc: use :command: for subcommands in ceph-bluestore-tool manpage

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #23166 from tchaikov/wip-cmake-cleanups

cmake: cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22875 from smithfarm/wip-backport-script

script: new ceph-backport.sh script

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20401 from smithfarm/wip-22558

build/ops: rpm: provide files moved from ceph-test …

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: extract ceph-dencoder related stuff out

let's modularize the cmake file piecemeal.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: Finddpdk: add dpdk::${component} libs

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23160 from smithfarm/wip-24017

tests: upgrade/luminous-x: whitelist REQUEST_SLOW for rados_mon_thrash

Reviewed-by: Sage Weil <sage@redhat.com>

tests: upgrade/luminous-x: whitelist REQUEST_SLOW for rados_mon_thrash

Fixes: http://tracker.ceph.com/issues/25051
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: provide files moved from ceph-test

d7b493a7108a68302bc0f48337bf3c253a720266 moved three binaries from the
ceph-test package to other packages, causing trouble with upgrades in
test lab environments (users typically do not install the ceph-test
package).

While the original upgrade troubles have already been addressed, RPM
has a "Provides" mechanism for easing the pain, so it makes sense to
use it.

Fixes: http://tracker.ceph.com/issues/22558
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #23139 into master

* refs/pull/23139/head:
mds: avoid using g_conf().get_val<...>(...) in hot path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22977 into master

* refs/pull/22977/head:
qa/tasks/cephfs: add test for discontinuous mdsmap
mds: handle discontinuous mdsmap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23022 into master

* refs/pull/23022/head:
mds: health warning for slow metadata IO

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23063 into master

* refs/pull/23063/head:
mds: increase debug level for dropped client cap msg

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23062 into master

* refs/pull/23062/head:
mds: dump recent events on respawn

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23061 into master

* refs/pull/23061/head:
mds: print mdsmap processed at low debug level

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #23135 into master

* refs/pull/23135/head:
mds: refactor ostream passing

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #22947 into master

* refs/pull/22947/head:
test: validate empty json output during recursive scrub
test: subclass test_scrub.Workload from CephFSTestCase
mds: dump minimal cli output during recursive scrub trigger

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23009 into master

* refs/pull/23009/head:
client: wire up ceph_ll_fallocate

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23151 from neha-ojha/wip-25008

qa/suites/powercycle: whitelist MDS_SLOW_REQUEST

Merge pull request #23123 from yuriw/wip-yuriw-24926-master

qa/tests: Set ansible-version: 2.5

qa/suites/powercycle: whitelist MDS_SLOW_REQUEST

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #22982 into master

* refs/pull/22982/head:
os/bluestore: Add new perfcounter which record latency of

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #23041 into master

* refs/pull/23041/head:
mon: manage paxos_services alloc with unique_ptr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23047 into master

* refs/pull/23047/head:
common: SubProcess needs SIGTERM, from the includes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23053 into master

* refs/pull/23053/head:
mon: emplace mon_caps in Monitor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23060 into master

* refs/pull/23060/head:
test: Only inquire bluestore when it's build

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23078 into master

* refs/pull/23078/head:
spdk: -march=core2 instead of -march=native

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22919 from liewegas/wip-rgw-cls-test

qa/suites/rgw/verify/tasks/cls_rgw: test cls_rgw

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22975 from tone-zhang/vstart-spdk

vstart.sh: Support SPDK in Ceph development deployment

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22790 from joke-lee/WIP-admin-rest-api-user-info-with-access-key

rgw: support admin rest api get user info through user's access-key

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23099 from adamemerson/wip-all-your-fault

rgw: All Your Fault

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #23058 from mkogan1/cont-19282-fix-rgw-multisite-datasync_v2

rgw: add curl_low_speed_limit and curl_low_speed_time config to avoid

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23119 from cbodley/wip-rgw-link-tenant

rgw: RGWBucket::link supports tenant

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23010 from noahdesu/mgr-expose-ec-profiles

mgr: expose ec profiles through manager

Reveiwed-by: John Spray <john.spray@redhat.com>

Merge pull request #23016 from rzarzynski/wip-osd-cleanup-log_op_stats

osd: clean up and avoid extra ref-counting in PrimaryLogPG::log_op_stats.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22993 from jcsp/wip-progress

mgr/progress: introduce the `progress` module

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

mds: avoid using g_conf().get_val<...>(...) in hot path

g_conf->get_val<>()'s overhead is high

Fixes: http://tracker.ceph.com/issues/24820
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

vstart.sh: Support SPDK in Ceph development deployment

Support SPDK in vstart.sh, the developer can debug SPDK within
development deployment with the option
"--bluestore-spdk <vendor>:<device>".

Signed-off-by: tone.zhang <tone.zhang@arm.com>

Merge pull request #23121 from noahdesu/mgr-crash-history

mgr/crash: add hour granularity crash summary

mds: refactor ostream passing

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/crash: json report of recent crashes

reports crashes per entity type with hour granularity. primarily for
consumption by the insights module.

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

mgr/crash: add timestamp filter helper

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #23129 from alfredodeza/wip-rm24952

ceph-volume lvm.listing only include devices if they exist

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #23097 into master

* refs/pull/23097/head:
vstart_runner: pass ctx to FuseMount and load mount info

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #23077 from smithfarm/wip-etc-sudoers

build/ops: rpm: make sudo a build dependency

Reviewed-by: Tim Serong <tserong@suse.com>

build/ops: rpm: make sudo a build dependency

ede36df3253bd67bfe206abc0446f1f156feae64 added sudo as a runtime dependency,
but that commit also causes OBS builds to fail with:

[10666s] ... checking filelist
[10667s] ceph-osd-14.0.0-lp150.1332.1.x86_64.rpm: directories not owned by a package:
[10667s]  - /etc/sudoers.d

Since the "sudo" package is a runtime dependency of ceph-osd, it is guaranteed
to be installed on the target system and there is no reason for ceph-osd to own
this directory.

Having sudo as a build dependency makes OBS happy, and does no harm.

Follow-on fix for https://github.com/ceph/ceph/pull/22745

Signed-off-by: Nathan Cutler <ncutler@suse.com>

script: new ceph_backport.sh script

Once the local backport git branch has been prepared (i.e. after cherry-picking
stage), this script automates the process of opening the backport PR and
updating the backport tracker issue.

Parts of this script were written by Loic Dachary.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23089 from smithfarm/wip-seastar-build-deps

build/ops: rpm: fix seastar build dependencies for SUSE

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: use :command: for subcommands in ceph-bluestore-tool manpage

Older versions of Sphinx, such as the one in CentOS 7, do not render "..
option::" lines correctly if the option contains a hyphen but does not start
with a hyphen. And ceph-bluestore-tool appears to be the only Ceph manpage
affected by this bug.

Fixes: http://tracker.ceph.com/issues/24800
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: fix seastar build dependencies for SUSE

cryptopp-devel is called libcryptopp-devel on SUSE.

/usr/bin/protoc is included in protobuf-devel on SUSE.

numactl-devel is called libnuma-devel on SUSE

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/cephfs: add test for discontinuous mdsmap

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: handle discontinuous mdsmap

There are two cases that a mds can get discontinuous mdsmap:
 - the mdsmap was sent by other mds
 - connection to monitor was reset

Monitor does not preserve old version mdsmap. There is no easy way to
ensure mds always gets continuous mdsmap. Instead, making mds handle
discontinuous mdsmap is not difficult. When failover happens in multimds
cluster, survivor mds may miss the mdsmap that indicates old mds failed
and/or the mdsmap that indicates new mds started to replay. But the
survivor mds always get the mdsmap that indicates the new mds entered
resolve state.

Fixes: http://tracker.ceph.com/issues/24856
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

qa/tests: Set ansible-version: 2.5

Fixes http://tracker.ceph.com/issues/24926
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

ceph-volume lvm.listing only include devices if they exist

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rgw: RGWBucket::link supports tenant

use tenant-aware helper functions to generate metadata keys/oids

this allows 'radosgw-admin bucket link' to succeed for a tenanted user,
provided that the bucket belongs to the same tenant

Fixes: http://tracker.ceph.com/issues/22666
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23117 from ceph/wip-rm24796

ceph-volume: unmount lvs correctly before zapping

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23098 from neha-ojha/wip-limit-pglog

osd: limit pg log length under all circumstances

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22347 from cbodley/wip-24367

rgw: fixes for sync of versioned objects

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23032 from cbodley/wip-24873

radosgw-admin: 'sync error trim' loops until complete

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23116 from alfredodeza/wip-rm24961

ceph-volume tests/functional run lvm list after OSD provisioning

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests/functional run lvm list after OSD provisioning

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rgw: Cache notify fault injection

Allow a configured probability of dropping a notify message on the
floor without processing or acking it.

Fixes: http://tracker.ceph.com/issues/24962
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph-volume: unmount lvs correctly before zapping

If the lv found does not include the meatadata to construct an osd path,
then we should try to unmount it's path instead.

Fixes: http://tracker.ceph.com/issues/24796
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23075 from alfredodeza/wip-24492

ceph-volume batch command

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23083 from dzafman/wip-disable-skip-dd

osd: Change osd_skip_data_digest default to false and make it LEVEL_DEV

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22732 from b-ranto/wip-metric-reset

prometheus: Fix metric resets

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #23040 into master

* refs/pull/23040/head:
ceph-debug-docker: install python3 cephfs/rados packages

Reviewed-by: Rishabh Dave <ridave@redhat.com>

vstart_runner: pass ctx to FuseMount and load mount info

Credit to Venky for finding the problem.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23008 from tchaikov/wip-rbd-silence-gcc-warning

tools/rbd: assert(g_ceph_context) not g_conf

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22902 from alfonsomthd/24778-dashboard-info-cards-refactor

mgr/dashboard: Dashboard info cards refactoring

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23108 from tchaikov/wip-install-deps-seastar

run-make-check: set WITH_SEASTAR with a non-empty string

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

install-deps: set DEBIAN_FRONTEND for apt-get

so it won't complain if the tty is not an interactive one.

Signed-off-by: Kefu Chai <kchai@redhat.com>

run-make-check: set WITH_SEASTAR with a non-empty string

otherwise "test -n $WITH_SEASTAR" fails, and the dependencies won't be
installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #22966 into master

* refs/pull/22966/head:
mds: clarify doc string for mds_beacon_grace

Reviewed-by: Zheng Yan <zyan@redhat.com>

osd/PGLog: allow pg log trim when complete_to is less than trim_to

Signed-off-by: Neha Ojha <nojha@redhat.com>

prometheus: Set the response header for cached response

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Reset the time the data was captured

This was forgotten during the previous patches.

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Format metrics in the collect function

This allows us to clear metrics upon formatting in the collect function.

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Remove the Metrics class

The class was introduced to account for the 'pending' mechanism but that
is no longer being used by the code so we no longer need it since it
serves a purpose of a simple dictionary, now.

This simplifies the code and brings a bit of performance improvements
since the Metric classes in the collect method are created only if they
did not exist before.

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Optimize metrics formatting

It is faster to use join for join multiple strings than the regular
string concatenation since it does not create a new string on each
iteration.

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Use instance instead of inst variable

Signed-off-by: Boris Ranto <branto@redhat.com>