Merge PR #37095 into master

* refs/pull/37095/head:
mds: make purge_queue delete objects asynchronously and keep accepting pushes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37876 into master

* refs/pull/37876/head:
systemd: cephfs-mirror systemd unit files
doc, man: man page for `cephfs-mirror` tool
doc: document mirror daemon internals
cephfs-mirror: switch to using PeerReplayer class
cephfs-mirror: cancel ongoing snapshot syncs on dir removal
cephfs-mirror: display peer snapshot sync stats
cephfs-mirror: carve out (and implement) mirroring snapshots to peers
cephfs-mirror: remove `cephfs_mirror_directory_choose_policy` option
cephfs-mirror: include helper routines to separate source
cephfs-mirror: remove peer only when peer is tracked
cephfs-mirror: typedef ceph_mount_info as MountRef shared pointer
cephfs-mirror: enclose json dump in object section
cephfs-mirror: note current peer set
cephfs-mirror: fix option typo and document certain options
cephfs-mirror: remove unnecessary command line options
cephfs-mirror: default log level 0/5

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38820 into master

* refs/pull/38820/head:
include: use narrower ostream instead of iostream
mds: remove unused fstream header
mds: use narrower ostream header instead of sstream
mds: remove unused iostream header
mds: remove unused unix headers

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38835 into master

* refs/pull/38835/head:
qa: add test for fs rm idempotency
mon/FSCommands: restore idempotent behavior of fs rm
qa: add ceph cmd helper
qa: allow kwargs for raw_cluster_cmd

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38848 into master

* refs/pull/38848/head:
qa: ignore MDS_SLOW_METADATA_IO with osd thrasher

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #38886 from cbodley/wip-rgw-pubsub-warning

rgw: fix initialization of s3_event in pubsub

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #38882 from tchaikov/wip-no-print-nanoseconds

tests/test_bluestore_types: cast duration to ceph::timespan

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge PR #38652 into master

* refs/pull/38652/head:
mgr/volumes: Filter inherited snapshots while listing snapshots

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38785 from sebastian-philipp/cephadm-move-create-daemon-to-serve

mgr/cephadm: move a bunch of methods from module.py to serve.py

Reviewed-by: Michael Fritch <mfritch@suse.com>

rgw: fix initialization of s3_event in pubsub

resolves compiler warning:

rgw_sync_module_pubsub.cc: In constructor ‘RGWPSHandleObjEventCR::RGWPSHandleObjEventCR(RGWDataSyncCtx*, PSEnvRef, const rgw_user&, EventRef<rgw_pubsub_event>&, EventRef<rgw_pubsub_s3_event>&, const TopicsRef&)’:
rgw_sync_module_pubsub.cc:1028:3: warning: ‘RGWPSHandleObjEventCR::s3_event’ is initialized with itself [-Winit-self]
 1028 |   RGWPSHandleObjEventCR(RGWDataSyncCtx* const _sc,
      |   ^~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Casey Bodley <cbodley@redhat.com>

tests/test_bluestore_types: cast duration to ceph::timespan

on FreeBSD + Clang + libc++ + amd64, std::chrono::nanoseconds::rep is `long
long` instead of `long`, so the explictly instantiated template
operator<< operator instances in `src/common/ceph_time.cc` are not able to
cover this use case. so in this change, the duration is casted to
ceph::timespan instead of nanoseconds. so we can ensure that `operator<<` is
always available in this case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38860 from tchaikov/wip-48777

osd/pg_scrubber: do not print log in dtors

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #38724 from tchaikov/wip-install-deps-remove-mirror

install-deps.sh: do not use deb.rug.nl as a mirror anymore

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/volumes: Filter inherited snapshots while listing snapshots

Filter inherited snapshots resulted as part of a snapshot
at ancestor level while listing snapshots of a subvolume
and subvolumegroup

Also, fail the snapshot info on inherited snapshot.

Fixes: https://tracker.ceph.com/issues/48501
Signed-off-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #33692 from majianpeng/osd-remove-useless-code

osd: remove usless check condition.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38765 from tchaikov/wip-mutex-debug-atomic

common/mutex_debug: add memory barrier before load/store nlock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38696 from cyx1231st/wip-seastar-onode-tree-denc-delta

crimson/onode-staged-tree: implement delta encode, decode, replay and tests

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: set indent-tabs-mode to nil

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement get_type() for test extents

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: implement seastore replay test for onode-staged-tree

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38806 from yuriw/wip-yuriw-octopus-x-master

qa/tests: added rbd_import_export workload

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tests: added stress-split-no-cephadm tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #38849 into master

* refs/pull/38849/head:
doc: document ms_mode option in the mount.ceph manpage

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38793 into master

* refs/pull/38793/head:
qa: check for rank hole without racy joinable flag gymnastics
qa: skip client mounts for TestResize

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa: add test for fs rm idempotency

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/FSCommands: restore idempotent behavior of fs rm

Fixes: https://tracker.ceph.com/issues/48808
Fixes: 811e48d6ffd841ecd93a57eb00c6aceae476f4f8
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add ceph cmd helper

A more programmer friendly command to use.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: allow kwargs for raw_cluster_cmd

And refactor.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd/pg_scrubber: do not print log in dtors

PG::recovery_state is defined after PG::m_scrubber, while
PG::gen_prefix() retrieves the osdmap from recovery_state. in
PgScrubber::~PgScrubber(), we print out the osdmap by dereferencing
recovery_state for the prefix of the logging message. in other words,
PgScrubber's destructor is referencing "recovery_state" which is
already destroyed.

in this change, the logging messages are removed from dtors. we can
print out the pgid in a follow-up PR if it helps.

Fixes: https://tracker.ceph.com/issues/48777
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38718 from tchaikov/wip-mgr-health-cleanup

mgr/devicehealth: use helpers to define options and commands

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

systemd: cephfs-mirror systemd unit files

Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc, man: man page for `cephfs-mirror` tool

Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: document mirror daemon internals

... fix some typos and reorganize sections to be reader
friendly. Also include a requirements section.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: switch to using PeerReplayer class

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: cancel ongoing snapshot syncs on dir removal

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: display peer snapshot sync stats

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: carve out (and implement) mirroring snapshots to peers

Remote file synchronization by @mchangir.

Signed-off-by: Milind Changire <mchangir@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: remove `cephfs_mirror_directory_choose_policy` option

Basically, the mirror daemon would process directory list in in a
round-robin fashion. Randomly choosing directories probably does not
make much sense right now.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: include helper routines to separate source

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: remove peer only when peer is tracked

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: typedef ceph_mount_info as MountRef shared pointer

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: enclose json dump in object section

So that mirroring module can decode the JSON object.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: note current peer set

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: fix option typo and document certain options

s/cephfs_mirror_mirror_action_update_interval/cephfs_mirror_action_update_interval/

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: remove unnecessary command line options

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: default log level 0/5

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #38863 from tchaikov/wip-osd-scrubber-tab

osd/pg_scrubber: use tab-width:8

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #34859 into master

* refs/pull/34859/head:
test: Initialize timer
test: Avoid named semaphores on Windows
test: Update cmake files, allowing tests to run on Windows
test: port Ceph tests to Windows

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd/pg_scrubber: use tab-width:8

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38781 from tchaikov/wip-googletest-gcc11

googletest: pick up change to silence error=maybe-uninitialized warning

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38809 from Rethan/fs-status-style

pybind/mgr: change format_bytes unit color from black to yellow

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38814 from tchaikov/wip-fix-throttle-test

common/Throttle: return new count in put() right away

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38811 from majianpeng/optimize-is_blackhole

msg: using switch-case in place of if-else.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38780 from majianpeng/fix-wrong-num_ios

os/bluestore: fix wrong num_ios for read op.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #38837 from vbashkirtsev/master

common/options.h: use uint64_t for TYPE_SIZE options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: document ms_mode option in the mount.ceph manpage

Fixes: https://tracker.ceph.com/issues/48838
Signed-off-by: Jeff Layton <jlayton@redhat.com>

qa/tests: added rbd_import_export tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #38768 from votdev/fix_path_to_dash_devel_doc

mgr/dashboard: Fix path to dash_devel.rst

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38790 into master

* refs/pull/38790/head:
common: remove dead option

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38788 into master

* refs/pull/38788/head:
mount.ceph: collect v2 addresses for non-legacy ms_mode options

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38640 into master

* refs/pull/38640/head:
qa: add test for reserved keyword feature
qa: use no client for required client feature tests
mds: do not allow setting a reserved feature by name
mds: return sv for efficiency

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

qa: ignore MDS_SLOW_METADATA_IO with osd thrasher

Fixes: https://tracker.ceph.com/issues/48834
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38766 from sebastian-philipp/cephadm-service-action-async

mgr/cephadm: make `ceph orch {restart|...}` asynchronous

Reviewed-by: Michael Fritch <mfritch@suse.com>

include: use narrower ostream instead of iostream

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove unused fstream header

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use narrower ostream header instead of sstream

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove unused iostream header

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove unused unix headers

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38255 from linuxbox2/wip-fio-librgw

rgw:fio: introduce libfio_rgw

Merge pull request #36120 from smithfarm/wip-rpath-ceph-diff-sorted

cmake: add empty RPATH to ceph-diff-sorted

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #38671 from rhcs-dashboard/48685-upgrade-grafana

mgr/dashboard/monitoring: upgrade Grafana version due to CVE-2020-13379

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: b-ranto <branto@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: p-se <pseidensal@suse.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #38108 into master

* refs/pull/38108/head:
doc, man: man page for `cephfs-top` utility
doc: document `cephfs-top` utility
test: selftest for `cephfs-top` utility
spec, deb: package cephfs-top utility
cephfs-top: top(1) like utility for Ceph Filesystem
mgr/stats: include kernel version (for kclients) in `perf stats` command output
mgr/stats: include version with `perf stats` output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38836 into master

* refs/pull/38836/head:
client: add debug messages for osdmap wait

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #38141 from Daniel-Pivonka/nodeexporterpullauth

cephadm: fix podman failure to pull authenticated registry image from systemd unit

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _remote_connection to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _deploy_cephadm_binary to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _registry_login to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _get_container_image_info to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _run_cephadm to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _remove_daemon to serve.py

`_remove_daemon` can potentially make the CLI unresponsive and should
only be called from the serve() thread.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _create_daemon to serve.py

`_create_daemon` can potentially make the CLI unresponsive and should
only be called from the serve() thread.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38712 from orozery/librbd-encryption-api

librbd: encryption API

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: add empty RPATH to ceph-diff-sorted

This fixes a transient FTBFS on openSUSE:

[ 5365s] -- Installing: /home/abuild/rpmbuild/BUILDROOT/ceph-15.2.4.337+g55cec95eaf-1.1.x86_64/usr/bin/ceph-diff-sorted
[ 5365s] CMake Error at src/tools/cmake_install.cmake:230 (file):
[ 5365s]   file RPATH_CHANGE could not write new RPATH:
[ 5365s]
[ 5365s]     /usr/lib64/ceph
[ 5365s]
[ 5365s]   to the file:
[ 5365s]
[ 5365s]     /home/abuild/rpmbuild/BUILDROOT/ceph-15.2.4.337+g55cec95eaf-1.1.x86_64/usr/bin/ceph-diff-sorted
[ 5365s]
[ 5365s]   No valid ELF RPATH or RUNPATH entry exists in the file;

Fixes: https://tracker.ceph.com/issues/46553
Signed-off-by: Nathan Cutler <ncutler@suse.com>

librbd: add encryption api

This commit exposes librbd encryption api

Signed-off-by: Or Ozeri <oro@il.ibm.com>

librbd: change remap_extents to lvalue reference

This commit changes the remap_extents api to use lvalue instead of ravlue references

Signed-off-by: Or Ozeri <oro@il.ibm.com>

librbd: fix wrong delete type in DataCryptor

This commit fixes a delete which does not match its allocation

Signed-off-by: Or Ozeri <oro@il.ibm.com>

doc, man: man page for `cephfs-top` utility

Fixes: http://tracker.ceph.com/issues/48602
Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: document `cephfs-top` utility

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: selftest for `cephfs-top` utility

Signed-off-by: Venky Shankar <vshankar@redhat.com>

spec, deb: package cephfs-top utility

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-top: top(1) like utility for Ceph Filesystem

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/stats: include kernel version (for kclients) in `perf stats` command output

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/stats: include version with `perf stats` output

So that `fstop` infers the version of mgr/stats to parse the
JSON command output.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #38804 from sebastian-philipp/cephadm-silence-evict-container

cephadm: silence "Failed to evict container" log msg

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #38615 from adk3798/haproxy-stuff

mgr/cephadm: HA for RGW endpoints

Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Fix path to dash_devel.rst

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #38543 from zjcmszh/cmpext_op

pybind/rados: add WriteOp::cmpext() and ReadOp::cmpext()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38531 from gregsfortytwo/wip-stretch-updates

More stretch updates for overly-aggressive safety checks that broke clients

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

client: add debug messages for osdmap wait

To help debug i47294.

See-also: https://tracker.ceph.com/issues/47294
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: move object create before set alloc hint

This commit fixes the order of ops in write requests to have create before set alloc hint, so that exclusive create will not falsely fail

Signed-off-by: Or Ozeri <oro@il.ibm.com>