Merge pull request #38838 from orozery/rbd-encryption-cli

librbd: rbd encryption cli

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38864 from MahatiC/wip-ssd-fix

librbd/cache/pwl: Rename config options

Reviewed-by: Xiaoyan Li <xiaoyan.li@intel.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38751 from rhcs-dashboard/fix-48591-master

mgr/dashboard: Monitoring alert badge includes suppressed alerts

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38879 from Rethan/remove-osd-heartbeat-option

doc/rados: remove unused osd heartbeat option

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38564 from ronen-fr/wip-ronenf-clangcrims-1

rgw: fixing compilation errors

Reviewed-by: Willem Jan Withagen (@wjwithagen)
Reviewed-by: Matt Benjamin mbenjamin@redhat.com
Reviewed-by: Daniel Gryniewicz dang@redhat.com dang@fprintf.net
Reviewed-by: Adam C. Emerson aemerson@redhat.com
Reviewed-by: Yuval Lifshitz ylifshit@redhat.com

Merge PR #37095 into master

* refs/pull/37095/head:
mds: make purge_queue delete objects asynchronously and keep accepting pushes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37876 into master

* refs/pull/37876/head:
systemd: cephfs-mirror systemd unit files
doc, man: man page for `cephfs-mirror` tool
doc: document mirror daemon internals
cephfs-mirror: switch to using PeerReplayer class
cephfs-mirror: cancel ongoing snapshot syncs on dir removal
cephfs-mirror: display peer snapshot sync stats
cephfs-mirror: carve out (and implement) mirroring snapshots to peers
cephfs-mirror: remove `cephfs_mirror_directory_choose_policy` option
cephfs-mirror: include helper routines to separate source
cephfs-mirror: remove peer only when peer is tracked
cephfs-mirror: typedef ceph_mount_info as MountRef shared pointer
cephfs-mirror: enclose json dump in object section
cephfs-mirror: note current peer set
cephfs-mirror: fix option typo and document certain options
cephfs-mirror: remove unnecessary command line options
cephfs-mirror: default log level 0/5

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38820 into master

* refs/pull/38820/head:
include: use narrower ostream instead of iostream
mds: remove unused fstream header
mds: use narrower ostream header instead of sstream
mds: remove unused iostream header
mds: remove unused unix headers

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38835 into master

* refs/pull/38835/head:
qa: add test for fs rm idempotency
mon/FSCommands: restore idempotent behavior of fs rm
qa: add ceph cmd helper
qa: allow kwargs for raw_cluster_cmd

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38848 into master

* refs/pull/38848/head:
qa: ignore MDS_SLOW_METADATA_IO with osd thrasher

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #38886 from cbodley/wip-rgw-pubsub-warning

rgw: fix initialization of s3_event in pubsub

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #38882 from tchaikov/wip-no-print-nanoseconds

tests/test_bluestore_types: cast duration to ceph::timespan

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

librbd: add encryption cli

This commit adds rbd encryption cli (encryption format and nbd map)

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge PR #38652 into master

* refs/pull/38652/head:
mgr/volumes: Filter inherited snapshots while listing snapshots

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38785 from sebastian-philipp/cephadm-move-create-daemon-to-serve

mgr/cephadm: move a bunch of methods from module.py to serve.py

Reviewed-by: Michael Fritch <mfritch@suse.com>

rgw: fix initialization of s3_event in pubsub

resolves compiler warning:

rgw_sync_module_pubsub.cc: In constructor ‘RGWPSHandleObjEventCR::RGWPSHandleObjEventCR(RGWDataSyncCtx*, PSEnvRef, const rgw_user&, EventRef<rgw_pubsub_event>&, EventRef<rgw_pubsub_s3_event>&, const TopicsRef&)’:
rgw_sync_module_pubsub.cc:1028:3: warning: ‘RGWPSHandleObjEventCR::s3_event’ is initialized with itself [-Winit-self]
 1028 |   RGWPSHandleObjEventCR(RGWDataSyncCtx* const _sc,
      |   ^~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Casey Bodley <cbodley@redhat.com>

tests/test_bluestore_types: cast duration to ceph::timespan

on FreeBSD + Clang + libc++ + amd64, std::chrono::nanoseconds::rep is `long
long` instead of `long`, so the explictly instantiated template
operator<< operator instances in `src/common/ceph_time.cc` are not able to
cover this use case. so in this change, the duration is casted to
ceph::timespan instead of nanoseconds. so we can ensure that `operator<<` is
always available in this case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38860 from tchaikov/wip-48777

osd/pg_scrubber: do not print log in dtors

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #38724 from tchaikov/wip-install-deps-remove-mirror

install-deps.sh: do not use deb.rug.nl as a mirror anymore

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/rados: remove unused osd heartbeat option

Since commit a5f9ca, option osd_mon_ack_timeout was removed, change
doc accordingly.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

mgr/volumes: Filter inherited snapshots while listing snapshots

Filter inherited snapshots resulted as part of a snapshot
at ancestor level while listing snapshots of a subvolume
and subvolumegroup

Also, fail the snapshot info on inherited snapshot.

Fixes: https://tracker.ceph.com/issues/48501
Signed-off-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #33692 from majianpeng/osd-remove-useless-code

osd: remove usless check condition.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38765 from tchaikov/wip-mutex-debug-atomic

common/mutex_debug: add memory barrier before load/store nlock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd/cache/pwl: Rename config options

Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #38696 from cyx1231st/wip-seastar-onode-tree-denc-delta

crimson/onode-staged-tree: implement delta encode, decode, replay and tests

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: set indent-tabs-mode to nil

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement get_type() for test extents

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: implement seastore replay test for onode-staged-tree

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38806 from yuriw/wip-yuriw-octopus-x-master

qa/tests: added rbd_import_export workload

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tests: added stress-split-no-cephadm tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #38849 into master

* refs/pull/38849/head:
doc: document ms_mode option in the mount.ceph manpage

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38793 into master

* refs/pull/38793/head:
qa: check for rank hole without racy joinable flag gymnastics
qa: skip client mounts for TestResize

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa: add test for fs rm idempotency

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/FSCommands: restore idempotent behavior of fs rm

Fixes: https://tracker.ceph.com/issues/48808
Fixes: 811e48d6ffd841ecd93a57eb00c6aceae476f4f8
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add ceph cmd helper

A more programmer friendly command to use.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: allow kwargs for raw_cluster_cmd

And refactor.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd/pg_scrubber: do not print log in dtors

PG::recovery_state is defined after PG::m_scrubber, while
PG::gen_prefix() retrieves the osdmap from recovery_state. in
PgScrubber::~PgScrubber(), we print out the osdmap by dereferencing
recovery_state for the prefix of the logging message. in other words,
PgScrubber's destructor is referencing "recovery_state" which is
already destroyed.

in this change, the logging messages are removed from dtors. we can
print out the pgid in a follow-up PR if it helps.

Fixes: https://tracker.ceph.com/issues/48777
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38718 from tchaikov/wip-mgr-health-cleanup

mgr/devicehealth: use helpers to define options and commands

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

systemd: cephfs-mirror systemd unit files

Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc, man: man page for `cephfs-mirror` tool

Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: document mirror daemon internals

... fix some typos and reorganize sections to be reader
friendly. Also include a requirements section.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: switch to using PeerReplayer class

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: cancel ongoing snapshot syncs on dir removal

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: display peer snapshot sync stats

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: carve out (and implement) mirroring snapshots to peers

Remote file synchronization by @mchangir.

Signed-off-by: Milind Changire <mchangir@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: remove `cephfs_mirror_directory_choose_policy` option

Basically, the mirror daemon would process directory list in in a
round-robin fashion. Randomly choosing directories probably does not
make much sense right now.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: include helper routines to separate source

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: remove peer only when peer is tracked

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: typedef ceph_mount_info as MountRef shared pointer

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: enclose json dump in object section

So that mirroring module can decode the JSON object.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: note current peer set

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: fix option typo and document certain options

s/cephfs_mirror_mirror_action_update_interval/cephfs_mirror_action_update_interval/

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: remove unnecessary command line options

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: default log level 0/5

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #38863 from tchaikov/wip-osd-scrubber-tab

osd/pg_scrubber: use tab-width:8

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #34859 into master

* refs/pull/34859/head:
test: Initialize timer
test: Avoid named semaphores on Windows
test: Update cmake files, allowing tests to run on Windows
test: port Ceph tests to Windows

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd/pg_scrubber: use tab-width:8

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38781 from tchaikov/wip-googletest-gcc11

googletest: pick up change to silence error=maybe-uninitialized warning

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38809 from Rethan/fs-status-style

pybind/mgr: change format_bytes unit color from black to yellow

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38814 from tchaikov/wip-fix-throttle-test

common/Throttle: return new count in put() right away

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38811 from majianpeng/optimize-is_blackhole

msg: using switch-case in place of if-else.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38780 from majianpeng/fix-wrong-num_ios

os/bluestore: fix wrong num_ios for read op.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #38837 from vbashkirtsev/master

common/options.h: use uint64_t for TYPE_SIZE options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: document ms_mode option in the mount.ceph manpage

Fixes: https://tracker.ceph.com/issues/48838
Signed-off-by: Jeff Layton <jlayton@redhat.com>

qa/tests: added rbd_import_export tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #38768 from votdev/fix_path_to_dash_devel_doc

mgr/dashboard: Fix path to dash_devel.rst

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38790 into master

* refs/pull/38790/head:
common: remove dead option

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38788 into master

* refs/pull/38788/head:
mount.ceph: collect v2 addresses for non-legacy ms_mode options

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38640 into master

* refs/pull/38640/head:
qa: add test for reserved keyword feature
qa: use no client for required client feature tests
mds: do not allow setting a reserved feature by name
mds: return sv for efficiency

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

qa: ignore MDS_SLOW_METADATA_IO with osd thrasher

Fixes: https://tracker.ceph.com/issues/48834
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38766 from sebastian-philipp/cephadm-service-action-async

mgr/cephadm: make `ceph orch {restart|...}` asynchronous

Reviewed-by: Michael Fritch <mfritch@suse.com>

include: use narrower ostream instead of iostream

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove unused fstream header

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use narrower ostream header instead of sstream

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove unused iostream header

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove unused unix headers

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38255 from linuxbox2/wip-fio-librgw

rgw:fio: introduce libfio_rgw

Merge pull request #36120 from smithfarm/wip-rpath-ceph-diff-sorted

cmake: add empty RPATH to ceph-diff-sorted

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #38671 from rhcs-dashboard/48685-upgrade-grafana

mgr/dashboard/monitoring: upgrade Grafana version due to CVE-2020-13379

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: b-ranto <branto@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: p-se <pseidensal@suse.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #38108 into master

* refs/pull/38108/head:
doc, man: man page for `cephfs-top` utility
doc: document `cephfs-top` utility
test: selftest for `cephfs-top` utility
spec, deb: package cephfs-top utility
cephfs-top: top(1) like utility for Ceph Filesystem
mgr/stats: include kernel version (for kclients) in `perf stats` command output
mgr/stats: include version with `perf stats` output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38836 into master

* refs/pull/38836/head:
client: add debug messages for osdmap wait

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #38141 from Daniel-Pivonka/nodeexporterpullauth

cephadm: fix podman failure to pull authenticated registry image from systemd unit

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _remote_connection to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _deploy_cephadm_binary to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _registry_login to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _get_container_image_info to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _run_cephadm to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _remove_daemon to serve.py

`_remove_daemon` can potentially make the CLI unresponsive and should
only be called from the serve() thread.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _create_daemon to serve.py

`_create_daemon` can potentially make the CLI unresponsive and should
only be called from the serve() thread.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38712 from orozery/librbd-encryption-api

librbd: encryption API

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: add empty RPATH to ceph-diff-sorted

This fixes a transient FTBFS on openSUSE:

[ 5365s] -- Installing: /home/abuild/rpmbuild/BUILDROOT/ceph-15.2.4.337+g55cec95eaf-1.1.x86_64/usr/bin/ceph-diff-sorted
[ 5365s] CMake Error at src/tools/cmake_install.cmake:230 (file):
[ 5365s]   file RPATH_CHANGE could not write new RPATH:
[ 5365s]
[ 5365s]     /usr/lib64/ceph
[ 5365s]
[ 5365s]   to the file:
[ 5365s]
[ 5365s]     /home/abuild/rpmbuild/BUILDROOT/ceph-15.2.4.337+g55cec95eaf-1.1.x86_64/usr/bin/ceph-diff-sorted
[ 5365s]
[ 5365s]   No valid ELF RPATH or RUNPATH entry exists in the file;

Fixes: https://tracker.ceph.com/issues/46553
Signed-off-by: Nathan Cutler <ncutler@suse.com>

librbd: add encryption api

This commit exposes librbd encryption api

Signed-off-by: Or Ozeri <oro@il.ibm.com>

librbd: change remap_extents to lvalue reference

This commit changes the remap_extents api to use lvalue instead of ravlue references

Signed-off-by: Or Ozeri <oro@il.ibm.com>

librbd: fix wrong delete type in DataCryptor

This commit fixes a delete which does not match its allocation

Signed-off-by: Or Ozeri <oro@il.ibm.com>

doc, man: man page for `cephfs-top` utility

Fixes: http://tracker.ceph.com/issues/48602
Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: document `cephfs-top` utility

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: selftest for `cephfs-top` utility

Signed-off-by: Venky Shankar <vshankar@redhat.com>

spec, deb: package cephfs-top utility

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-top: top(1) like utility for Ceph Filesystem

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/stats: include kernel version (for kclients) in `perf stats` command output

Signed-off-by: Venky Shankar <vshankar@redhat.com>