Merge pull request #23186 from smithfarm/wip-4640

doc/man: mention import and export commands in rados manpage

doc/man: mention import and export commands in rados manpage

Fixes: http://tracker.ceph.com/issues/4640
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23114 from smithfarm/wip-24800

doc: use :command: for subcommands in ceph-bluestore-tool manpage

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #23166 from tchaikov/wip-cmake-cleanups

cmake: cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22875 from smithfarm/wip-backport-script

script: new ceph-backport.sh script

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20401 from smithfarm/wip-22558

build/ops: rpm: provide files moved from ceph-test …

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: extract ceph-dencoder related stuff out

let's modularize the cmake file piecemeal.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: Finddpdk: add dpdk::${component} libs

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23160 from smithfarm/wip-24017

tests: upgrade/luminous-x: whitelist REQUEST_SLOW for rados_mon_thrash

Reviewed-by: Sage Weil <sage@redhat.com>

tests: upgrade/luminous-x: whitelist REQUEST_SLOW for rados_mon_thrash

Fixes: http://tracker.ceph.com/issues/25051
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: provide files moved from ceph-test

d7b493a7108a68302bc0f48337bf3c253a720266 moved three binaries from the
ceph-test package to other packages, causing trouble with upgrades in
test lab environments (users typically do not install the ceph-test
package).

While the original upgrade troubles have already been addressed, RPM
has a "Provides" mechanism for easing the pain, so it makes sense to
use it.

Fixes: http://tracker.ceph.com/issues/22558
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #23139 into master

* refs/pull/23139/head:
mds: avoid using g_conf().get_val<...>(...) in hot path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22977 into master

* refs/pull/22977/head:
qa/tasks/cephfs: add test for discontinuous mdsmap
mds: handle discontinuous mdsmap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23022 into master

* refs/pull/23022/head:
mds: health warning for slow metadata IO

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23063 into master

* refs/pull/23063/head:
mds: increase debug level for dropped client cap msg

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23062 into master

* refs/pull/23062/head:
mds: dump recent events on respawn

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23061 into master

* refs/pull/23061/head:
mds: print mdsmap processed at low debug level

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #23135 into master

* refs/pull/23135/head:
mds: refactor ostream passing

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #22947 into master

* refs/pull/22947/head:
test: validate empty json output during recursive scrub
test: subclass test_scrub.Workload from CephFSTestCase
mds: dump minimal cli output during recursive scrub trigger

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23009 into master

* refs/pull/23009/head:
client: wire up ceph_ll_fallocate

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23151 from neha-ojha/wip-25008

qa/suites/powercycle: whitelist MDS_SLOW_REQUEST

Merge pull request #23123 from yuriw/wip-yuriw-24926-master

qa/tests: Set ansible-version: 2.5

qa/suites/powercycle: whitelist MDS_SLOW_REQUEST

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #22982 into master

* refs/pull/22982/head:
os/bluestore: Add new perfcounter which record latency of

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #23041 into master

* refs/pull/23041/head:
mon: manage paxos_services alloc with unique_ptr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23047 into master

* refs/pull/23047/head:
common: SubProcess needs SIGTERM, from the includes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23053 into master

* refs/pull/23053/head:
mon: emplace mon_caps in Monitor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23060 into master

* refs/pull/23060/head:
test: Only inquire bluestore when it's build

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23078 into master

* refs/pull/23078/head:
spdk: -march=core2 instead of -march=native

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22919 from liewegas/wip-rgw-cls-test

qa/suites/rgw/verify/tasks/cls_rgw: test cls_rgw

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22975 from tone-zhang/vstart-spdk

vstart.sh: Support SPDK in Ceph development deployment

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22790 from joke-lee/WIP-admin-rest-api-user-info-with-access-key

rgw: support admin rest api get user info through user's access-key

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23099 from adamemerson/wip-all-your-fault

rgw: All Your Fault

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #23058 from mkogan1/cont-19282-fix-rgw-multisite-datasync_v2

rgw: add curl_low_speed_limit and curl_low_speed_time config to avoid

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23119 from cbodley/wip-rgw-link-tenant

rgw: RGWBucket::link supports tenant

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23010 from noahdesu/mgr-expose-ec-profiles

mgr: expose ec profiles through manager

Reveiwed-by: John Spray <john.spray@redhat.com>

Merge pull request #23016 from rzarzynski/wip-osd-cleanup-log_op_stats

osd: clean up and avoid extra ref-counting in PrimaryLogPG::log_op_stats.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22993 from jcsp/wip-progress

mgr/progress: introduce the `progress` module

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

mds: avoid using g_conf().get_val<...>(...) in hot path

g_conf->get_val<>()'s overhead is high

Fixes: http://tracker.ceph.com/issues/24820
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

vstart.sh: Support SPDK in Ceph development deployment

Support SPDK in vstart.sh, the developer can debug SPDK within
development deployment with the option
"--bluestore-spdk <vendor>:<device>".

Signed-off-by: tone.zhang <tone.zhang@arm.com>

Merge pull request #23121 from noahdesu/mgr-crash-history

mgr/crash: add hour granularity crash summary

mds: refactor ostream passing

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/crash: json report of recent crashes

reports crashes per entity type with hour granularity. primarily for
consumption by the insights module.

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

mgr/crash: add timestamp filter helper

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #23129 from alfredodeza/wip-rm24952

ceph-volume lvm.listing only include devices if they exist

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #23097 into master

* refs/pull/23097/head:
vstart_runner: pass ctx to FuseMount and load mount info

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #23077 from smithfarm/wip-etc-sudoers

build/ops: rpm: make sudo a build dependency

Reviewed-by: Tim Serong <tserong@suse.com>

build/ops: rpm: make sudo a build dependency

ede36df3253bd67bfe206abc0446f1f156feae64 added sudo as a runtime dependency,
but that commit also causes OBS builds to fail with:

[10666s] ... checking filelist
[10667s] ceph-osd-14.0.0-lp150.1332.1.x86_64.rpm: directories not owned by a package:
[10667s]  - /etc/sudoers.d

Since the "sudo" package is a runtime dependency of ceph-osd, it is guaranteed
to be installed on the target system and there is no reason for ceph-osd to own
this directory.

Having sudo as a build dependency makes OBS happy, and does no harm.

Follow-on fix for https://github.com/ceph/ceph/pull/22745

Signed-off-by: Nathan Cutler <ncutler@suse.com>

script: new ceph_backport.sh script

Once the local backport git branch has been prepared (i.e. after cherry-picking
stage), this script automates the process of opening the backport PR and
updating the backport tracker issue.

Parts of this script were written by Loic Dachary.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23089 from smithfarm/wip-seastar-build-deps

build/ops: rpm: fix seastar build dependencies for SUSE

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: use :command: for subcommands in ceph-bluestore-tool manpage

Older versions of Sphinx, such as the one in CentOS 7, do not render "..
option::" lines correctly if the option contains a hyphen but does not start
with a hyphen. And ceph-bluestore-tool appears to be the only Ceph manpage
affected by this bug.

Fixes: http://tracker.ceph.com/issues/24800
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: fix seastar build dependencies for SUSE

cryptopp-devel is called libcryptopp-devel on SUSE.

/usr/bin/protoc is included in protobuf-devel on SUSE.

numactl-devel is called libnuma-devel on SUSE

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/cephfs: add test for discontinuous mdsmap

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: handle discontinuous mdsmap

There are two cases that a mds can get discontinuous mdsmap:
 - the mdsmap was sent by other mds
 - connection to monitor was reset

Monitor does not preserve old version mdsmap. There is no easy way to
ensure mds always gets continuous mdsmap. Instead, making mds handle
discontinuous mdsmap is not difficult. When failover happens in multimds
cluster, survivor mds may miss the mdsmap that indicates old mds failed
and/or the mdsmap that indicates new mds started to replay. But the
survivor mds always get the mdsmap that indicates the new mds entered
resolve state.

Fixes: http://tracker.ceph.com/issues/24856
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

qa/tests: Set ansible-version: 2.5

Fixes http://tracker.ceph.com/issues/24926
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

ceph-volume lvm.listing only include devices if they exist

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rgw: RGWBucket::link supports tenant

use tenant-aware helper functions to generate metadata keys/oids

this allows 'radosgw-admin bucket link' to succeed for a tenanted user,
provided that the bucket belongs to the same tenant

Fixes: http://tracker.ceph.com/issues/22666
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23117 from ceph/wip-rm24796

ceph-volume: unmount lvs correctly before zapping

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23098 from neha-ojha/wip-limit-pglog

osd: limit pg log length under all circumstances

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22347 from cbodley/wip-24367

rgw: fixes for sync of versioned objects

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23032 from cbodley/wip-24873

radosgw-admin: 'sync error trim' loops until complete

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23116 from alfredodeza/wip-rm24961

ceph-volume tests/functional run lvm list after OSD provisioning

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests/functional run lvm list after OSD provisioning

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rgw: Cache notify fault injection

Allow a configured probability of dropping a notify message on the
floor without processing or acking it.

Fixes: http://tracker.ceph.com/issues/24962
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph-volume: unmount lvs correctly before zapping

If the lv found does not include the meatadata to construct an osd path,
then we should try to unmount it's path instead.

Fixes: http://tracker.ceph.com/issues/24796
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23075 from alfredodeza/wip-24492

ceph-volume batch command

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23083 from dzafman/wip-disable-skip-dd

osd: Change osd_skip_data_digest default to false and make it LEVEL_DEV

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22732 from b-ranto/wip-metric-reset

prometheus: Fix metric resets

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #23040 into master

* refs/pull/23040/head:
ceph-debug-docker: install python3 cephfs/rados packages

Reviewed-by: Rishabh Dave <ridave@redhat.com>

vstart_runner: pass ctx to FuseMount and load mount info

Credit to Venky for finding the problem.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23008 from tchaikov/wip-rbd-silence-gcc-warning

tools/rbd: assert(g_ceph_context) not g_conf

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22902 from alfonsomthd/24778-dashboard-info-cards-refactor

mgr/dashboard: Dashboard info cards refactoring

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23108 from tchaikov/wip-install-deps-seastar

run-make-check: set WITH_SEASTAR with a non-empty string

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

install-deps: set DEBIAN_FRONTEND for apt-get

so it won't complain if the tty is not an interactive one.

Signed-off-by: Kefu Chai <kchai@redhat.com>

run-make-check: set WITH_SEASTAR with a non-empty string

otherwise "test -n $WITH_SEASTAR" fails, and the dependencies won't be
installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #22966 into master

* refs/pull/22966/head:
mds: clarify doc string for mds_beacon_grace

Reviewed-by: Zheng Yan <zyan@redhat.com>

osd/PGLog: allow pg log trim when complete_to is less than trim_to

Signed-off-by: Neha Ojha <nojha@redhat.com>

prometheus: Set the response header for cached response

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Reset the time the data was captured

This was forgotten during the previous patches.

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Format metrics in the collect function

This allows us to clear metrics upon formatting in the collect function.

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Remove the Metrics class

The class was introduced to account for the 'pending' mechanism but that
is no longer being used by the code so we no longer need it since it
serves a purpose of a simple dictionary, now.

This simplifies the code and brings a bit of performance improvements
since the Metric classes in the collect method are created only if they
did not exist before.

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Optimize metrics formatting

It is faster to use join for join multiple strings than the regular
string concatenation since it does not create a new string on each
iteration.

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Use instance instead of inst variable

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Make the cache timeout configurable

Signed-off-by: Boris Ranto <branto@redhat.com>

prometheus: Fix metric resets

This patch changes the way we reset metrics when collecting data in the
prometheus exporter module. With this patch, the exported metrics at any
point in time should align with a freshly started ceph-mgr module.

The patch also introduces locking mechanism to serialize the requests
and to avoid overwriting the metrics during multiple scrapes happenning
at the same time.

Signed-off-by: Boris Ranto <branto@redhat.com>

ceph-volume tests validate the new utils for prompts

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util create prompt helpers

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.api update lvm sizing tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify str_to_int utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm convert and use extents for sizing calculations

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util move the str_to_int utility out of lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch update help docs from auto to batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.main rename auto to batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm rename auto to batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.strategies add stubs for filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.strategies add bluestore with validators

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23093 from ceph/wip-rm24044

ceph-volume: an OSD ID must be exist and be destroyed before reuse

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23038 from linuxbox2/wip-rgwfile-deepstat

rgw_file: deep stat handling

Merge pull request #23044 from tchaikov/wip-crimson-connection

crimson/connection: misc changes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23057 from liewegas/wip-12.2.7-notes

doc/release/luminous: v12.2.6 and v12.2.7 release notes

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>