Merge pull request #38886 from cbodley/wip-rgw-pubsub-warning

rgw: fix initialization of s3_event in pubsub

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #38882 from tchaikov/wip-no-print-nanoseconds

tests/test_bluestore_types: cast duration to ceph::timespan

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge PR #38652 into master

* refs/pull/38652/head:
mgr/volumes: Filter inherited snapshots while listing snapshots

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38785 from sebastian-philipp/cephadm-move-create-daemon-to-serve

mgr/cephadm: move a bunch of methods from module.py to serve.py

Reviewed-by: Michael Fritch <mfritch@suse.com>

rgw: fix initialization of s3_event in pubsub

resolves compiler warning:

rgw_sync_module_pubsub.cc: In constructor ‘RGWPSHandleObjEventCR::RGWPSHandleObjEventCR(RGWDataSyncCtx*, PSEnvRef, const rgw_user&, EventRef<rgw_pubsub_event>&, EventRef<rgw_pubsub_s3_event>&, const TopicsRef&)’:
rgw_sync_module_pubsub.cc:1028:3: warning: ‘RGWPSHandleObjEventCR::s3_event’ is initialized with itself [-Winit-self]
 1028 |   RGWPSHandleObjEventCR(RGWDataSyncCtx* const _sc,
      |   ^~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Casey Bodley <cbodley@redhat.com>

tests/test_bluestore_types: cast duration to ceph::timespan

on FreeBSD + Clang + libc++ + amd64, std::chrono::nanoseconds::rep is `long
long` instead of `long`, so the explictly instantiated template
operator<< operator instances in `src/common/ceph_time.cc` are not able to
cover this use case. so in this change, the duration is casted to
ceph::timespan instead of nanoseconds. so we can ensure that `operator<<` is
always available in this case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38860 from tchaikov/wip-48777

osd/pg_scrubber: do not print log in dtors

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #38724 from tchaikov/wip-install-deps-remove-mirror

install-deps.sh: do not use deb.rug.nl as a mirror anymore

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/volumes: Filter inherited snapshots while listing snapshots

Filter inherited snapshots resulted as part of a snapshot
at ancestor level while listing snapshots of a subvolume
and subvolumegroup

Also, fail the snapshot info on inherited snapshot.

Fixes: https://tracker.ceph.com/issues/48501
Signed-off-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #33692 from majianpeng/osd-remove-useless-code

osd: remove usless check condition.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38765 from tchaikov/wip-mutex-debug-atomic

common/mutex_debug: add memory barrier before load/store nlock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38696 from cyx1231st/wip-seastar-onode-tree-denc-delta

crimson/onode-staged-tree: implement delta encode, decode, replay and tests

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: set indent-tabs-mode to nil

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement get_type() for test extents

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: implement seastore replay test for onode-staged-tree

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38806 from yuriw/wip-yuriw-octopus-x-master

qa/tests: added rbd_import_export workload

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tests: added stress-split-no-cephadm tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #38849 into master

* refs/pull/38849/head:
doc: document ms_mode option in the mount.ceph manpage

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38793 into master

* refs/pull/38793/head:
qa: check for rank hole without racy joinable flag gymnastics
qa: skip client mounts for TestResize

Reviewed-by: Venky Shankar <vshankar@redhat.com>

osd/pg_scrubber: do not print log in dtors

PG::recovery_state is defined after PG::m_scrubber, while
PG::gen_prefix() retrieves the osdmap from recovery_state. in
PgScrubber::~PgScrubber(), we print out the osdmap by dereferencing
recovery_state for the prefix of the logging message. in other words,
PgScrubber's destructor is referencing "recovery_state" which is
already destroyed.

in this change, the logging messages are removed from dtors. we can
print out the pgid in a follow-up PR if it helps.

Fixes: https://tracker.ceph.com/issues/48777
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38718 from tchaikov/wip-mgr-health-cleanup

mgr/devicehealth: use helpers to define options and commands

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #38863 from tchaikov/wip-osd-scrubber-tab

osd/pg_scrubber: use tab-width:8

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #34859 into master

* refs/pull/34859/head:
test: Initialize timer
test: Avoid named semaphores on Windows
test: Update cmake files, allowing tests to run on Windows
test: port Ceph tests to Windows

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd/pg_scrubber: use tab-width:8

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38781 from tchaikov/wip-googletest-gcc11

googletest: pick up change to silence error=maybe-uninitialized warning

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38809 from Rethan/fs-status-style

pybind/mgr: change format_bytes unit color from black to yellow

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38814 from tchaikov/wip-fix-throttle-test

common/Throttle: return new count in put() right away

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38811 from majianpeng/optimize-is_blackhole

msg: using switch-case in place of if-else.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38780 from majianpeng/fix-wrong-num_ios

os/bluestore: fix wrong num_ios for read op.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #38837 from vbashkirtsev/master

common/options.h: use uint64_t for TYPE_SIZE options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: document ms_mode option in the mount.ceph manpage

Fixes: https://tracker.ceph.com/issues/48838
Signed-off-by: Jeff Layton <jlayton@redhat.com>

qa/tests: added rbd_import_export tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #38768 from votdev/fix_path_to_dash_devel_doc

mgr/dashboard: Fix path to dash_devel.rst

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38790 into master

* refs/pull/38790/head:
common: remove dead option

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38788 into master

* refs/pull/38788/head:
mount.ceph: collect v2 addresses for non-legacy ms_mode options

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38640 into master

* refs/pull/38640/head:
qa: add test for reserved keyword feature
qa: use no client for required client feature tests
mds: do not allow setting a reserved feature by name
mds: return sv for efficiency

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #38766 from sebastian-philipp/cephadm-service-action-async

mgr/cephadm: make `ceph orch {restart|...}` asynchronous

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38255 from linuxbox2/wip-fio-librgw

rgw:fio: introduce libfio_rgw

Merge pull request #36120 from smithfarm/wip-rpath-ceph-diff-sorted

cmake: add empty RPATH to ceph-diff-sorted

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #38671 from rhcs-dashboard/48685-upgrade-grafana

mgr/dashboard/monitoring: upgrade Grafana version due to CVE-2020-13379

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: b-ranto <branto@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: p-se <pseidensal@suse.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #38108 into master

* refs/pull/38108/head:
doc, man: man page for `cephfs-top` utility
doc: document `cephfs-top` utility
test: selftest for `cephfs-top` utility
spec, deb: package cephfs-top utility
cephfs-top: top(1) like utility for Ceph Filesystem
mgr/stats: include kernel version (for kclients) in `perf stats` command output
mgr/stats: include version with `perf stats` output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38836 into master

* refs/pull/38836/head:
client: add debug messages for osdmap wait

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #38141 from Daniel-Pivonka/nodeexporterpullauth

cephadm: fix podman failure to pull authenticated registry image from systemd unit

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _remote_connection to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _deploy_cephadm_binary to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _registry_login to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _get_container_image_info to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _run_cephadm to serve.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _remove_daemon to serve.py

`_remove_daemon` can potentially make the CLI unresponsive and should
only be called from the serve() thread.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move _create_daemon to serve.py

`_create_daemon` can potentially make the CLI unresponsive and should
only be called from the serve() thread.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38712 from orozery/librbd-encryption-api

librbd: encryption API

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: add empty RPATH to ceph-diff-sorted

This fixes a transient FTBFS on openSUSE:

[ 5365s] -- Installing: /home/abuild/rpmbuild/BUILDROOT/ceph-15.2.4.337+g55cec95eaf-1.1.x86_64/usr/bin/ceph-diff-sorted
[ 5365s] CMake Error at src/tools/cmake_install.cmake:230 (file):
[ 5365s]   file RPATH_CHANGE could not write new RPATH:
[ 5365s]
[ 5365s]     /usr/lib64/ceph
[ 5365s]
[ 5365s]   to the file:
[ 5365s]
[ 5365s]     /home/abuild/rpmbuild/BUILDROOT/ceph-15.2.4.337+g55cec95eaf-1.1.x86_64/usr/bin/ceph-diff-sorted
[ 5365s]
[ 5365s]   No valid ELF RPATH or RUNPATH entry exists in the file;

Fixes: https://tracker.ceph.com/issues/46553
Signed-off-by: Nathan Cutler <ncutler@suse.com>

librbd: add encryption api

This commit exposes librbd encryption api

Signed-off-by: Or Ozeri <oro@il.ibm.com>

librbd: change remap_extents to lvalue reference

This commit changes the remap_extents api to use lvalue instead of ravlue references

Signed-off-by: Or Ozeri <oro@il.ibm.com>

librbd: fix wrong delete type in DataCryptor

This commit fixes a delete which does not match its allocation

Signed-off-by: Or Ozeri <oro@il.ibm.com>

doc, man: man page for `cephfs-top` utility

Fixes: http://tracker.ceph.com/issues/48602
Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: document `cephfs-top` utility

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: selftest for `cephfs-top` utility

Signed-off-by: Venky Shankar <vshankar@redhat.com>

spec, deb: package cephfs-top utility

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-top: top(1) like utility for Ceph Filesystem

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/stats: include kernel version (for kclients) in `perf stats` command output

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/stats: include version with `perf stats` output

So that `fstop` infers the version of mgr/stats to parse the
JSON command output.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #38804 from sebastian-philipp/cephadm-silence-evict-container

cephadm: silence "Failed to evict container" log msg

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #38615 from adk3798/haproxy-stuff

mgr/cephadm: HA for RGW endpoints

Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Fix path to dash_devel.rst

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #38543 from zjcmszh/cmpext_op

pybind/rados: add WriteOp::cmpext() and ReadOp::cmpext()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38531 from gregsfortytwo/wip-stretch-updates

More stretch updates for overly-aggressive safety checks that broke clients

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

client: add debug messages for osdmap wait

To help debug i47294.

See-also: https://tracker.ceph.com/issues/47294
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: move object create before set alloc hint

This commit fixes the order of ops in write requests to have create before set alloc hint, so that exclusive create will not falsely fail

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #38701 from rosinL/fix-memory-over-use

commom/buffer: use small page aligned when bufferlist prealloc memory

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38747 from tchaikov/wip-scrub-event

osd/scrub_machine: pass an instance of event to post_event()

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

common/options.h: use uint64_t for TYPE_SIZE options regardless of CPU architecture

On 32 bit systems TYPE_SIZE should remain as 64 bit value as large number of default options of TYPE_SIZE easily overflow 32 bit word. Also cluster wide options from 64 bit systems should fit unmodified into options structures on 32 bit systems.

Signed-off-by: Vladimir Bashkirtsev <vladimir@bashkirtsev.com>

librbd: fix crypto related memory leaks

This commit fixes memory leaks introduced by the new librbd encryption feature

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #38796 from MahatiC/wip-ssd-read

librbd/cache: Implement aio_read operation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38272 from vbashkirtsev/master

test/test_arch.cc: search "pni" in CPU flags also when detecting SSE3

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/test_arch.cc: search "pni" in CPU flags also when detecting SSE3

as sometimes a SSE-3 CPU reports "pni" (Prescott New Instructions) in /proc/cpuinfo.

Signed-off-by: Vladimir Bashkirtsev <vladimir@bashkirtsev.com>

do_cmake.sh: do not perform 'git submodule update' if building from tarball

Signed-off-by: Vladimir Bashkirtsev <vladimir@bashkirtsev.com>

Merge pull request #38753 from Rethan/style-ceph-df-output

mon/PGMap: align to same side when output ceph df / ceph df detail

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38775 into master

* refs/pull/38775/head:
qa: remove ceph file systems on completion

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38791 into master

* refs/pull/38791/head:
mon/MDSMonitor: add missing frozen checks

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #38778 into master

* refs/pull/38778/head:
mds: don't use dentry_key_t in C_IO_Dir_Commit_Ops

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #38773 into master

* refs/pull/38773/head:
mds: avoid spurious sleeps

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #38698 into master

* refs/pull/38698/head:
client: fix the cache size check
client: add more detail info for the _put_inode debug log
client: skip putting inodes if map is empty

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38683 into master

* refs/pull/38683/head:
mds: add scrub perf counters support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38642 into master

* refs/pull/38642/head:
rpm/deb/cmake: install mount.fuse.ceph man page

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd: osdmap: do not assert target_v when encoding in stretch mode

Again, this was an overly-strict safety check. It passed testing against
(even old) userspace clients, but the kernel clients don't have all the
same feature bits and so when encoding for them, we see lower target_v values
and assert out erroneously.

As before, monitors and OSDs that actually need to interpret this data
are gated on cluster entry, so we don't need this safety check here.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

cephadm: HA for RGW endpoints

Cephadm deploying keepalived and HAproxy for providing High availability for RGW endpoints

Fixes: https://tracker.ceph.com/issues/45116
Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #37882 from cbodley/wip-rgw-data-sync-notification-fix

rgw: data sync: fix obligation check

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #38720 from bk201/wip-48717

mgr/dashboard: fix Reads/Writes ratio of Clients IOPS donut chart

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #38776 into master

* refs/pull/38776/head:
qa: get pool holding file data from file layout

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38664 into master

* refs/pull/38664/head:
qa: bump scrub timeout
qa: move cephfs_ec_profile under cephfs
qa: do not use ec pools for default data pool
qa: skip check-counters for light workloads
qa: remove empty multimds suite
qa: merge multimds:verify with fs:verify
qa: merge multimds:thrash to fs:thrash
qa: remove dead multimds:basic
qa: move functional multimds tests to fs:functional
qa: migrate multimds workloads to fs:workloads
qa: only run valgrind on cephfs daemons
qa: stop testing filestore on cephfs suites
qa: load data pools before deleting fs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Tested-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38693 into master

* refs/pull/38693/head:
qa: execute scrubs only on rank 0
client: print debug information about resolved MDS
qa: let Client::resolve_mds lookup the rank

Reviewed-by: Xiubo Li <xiubli@redhat.com>

qa: remove ceph file systems on completion

So that we can avoid MDS replacement warnings.

Fixes: https://tracker.ceph.com/issues/48757
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38510 into master

* refs/pull/38510/head:
PendingReleaseNotes: Update about volume/nfs cluster id changes
mgr/volumes/nfs: remove 'ganesha-' prefix from cluster id

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38537 from IlsooByun/fix_ret_invalid_err

rgw: return error when trying to copy encrypted object without key

Merge pull request #38283 from dvanders/dvanders_throttle

rgw: add an outstanding_requests perf counter

Merge pull request #38242 from theanalyst/rgw/reset-stats-1-fix

rgw: cls/user: set from_index for reset stats calls

qa: add test for reserved keyword feature

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38725 from CongMinYin/immutable-obejct-cache-doc

doc/rbd: Immutable object cache doc

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38667 from pcuzner/fix-cephadm-stdout

cephadm: make host add failure message more friendly