osd: handle ceph specific config changes for the mclock scheduler

The below ceph parameters are set automatically
while enabling the mclock scheduler with a built-in profile.
The user in this case will not be able to modify these
ceph specific config options during runtime.

  a. osd_async_recovery_min_cost
  b. osd_recovery_max_active{_hdd,_ssd}
  c. osd_max_backfills
  d. osd_recovery_sleep{_hdd,_ssd,_hybrid}

If the custom profile is enabled for the mclock scheduler,
the user can modify these parameters.

Signed-off-by: Sunny Kumar <sunkumar@redhat.com>

Merge PR #38896 into master

* refs/pull/38896/head:
cephfs: Remove explicit forward declare CephContext

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38912 from xenago/patch-1

common/options: Fix typos for osd_recovery_max_active

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #38891 from yuriw/wip-yuriw-octopus-x-master

qa/tests: added stress-split-erasure-code-no-cephadm tests

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38721 from changchengx/pwl

librbd/cache/pwl: fix parameter type to update argument

Reviewed-by: Xiaoyan Li <xiaoyan.li@intel.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/options: Fix typos for osd_recovery_max_active

This commit fixes a minor typo in the descriptions of osd_recovery_max_active_hdd and osd_recovery_max_active_ssd.

Signed-off-by: Noah Kruiper <nkruiper@gmail.com>

Merge pull request #38907 from rzarzynski/wip-crimson-errorator-handle_exception_type

crimson/common: errorated future exposes handle_exception_type().

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38708 into master

* refs/pull/38708/head:
doc: update multiple mds scrub docs

Reviewed-by: Jeff Layton <jlayton@redhat.com>

qa: remove invalid characters from octopus-x/parallel

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38870 from orozery/rbd-encryption-pybind

librbd: python bindings for encryption api

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tests: fixed yaml format

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #38846 into master

* refs/pull/38846/head:
*: remove legacy ceph_volume_client.py library

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #34858 into master

* refs/pull/34858/head:
mds: do not allow the service to run on Windows
win32*.sh: fix SKIP_TESTS flag
include/win32/win32_errno:h Add missing include
client: Port CephFS client to Windows

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38574 into master

* refs/pull/38574/head:
mds: update defaults for recall configs

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38889 into master

* refs/pull/38889/head:
qa: add delays only for osd/mds

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38858 into master

* refs/pull/38858/head:
client/libcephfs: Don't let one unmount screw up another mount

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/common: errorated future exposes handle_exception_type().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: python bindings for encryption api

This commit adds the python bindings for the new librbd encryption api

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #38850 from liewegas/mgr-handle-old-host-inventory

mgr/cephadm: tolerate old host inventory without 'hostname' key

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38866 from sebastian-philipp/cephadm-pyright

cephadm: Fix type errors reported by pyright

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

cephfs: Remove explicit forward declare CephContext

It is already correctly declared in:
    src/include/common_fwd.h

fixes: https://github.com/ceph/ceph/pull/37876
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #38851 from liewegas/fix-mailmap-sage

.mailmap: map myself to sweil@redhat.com

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd/cache/pwl: fix parameter type to update argument

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #38838 from orozery/rbd-encryption-cli

librbd: rbd encryption cli

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

client/libcephfs: Don't let one unmount screw up another mount

Share threadpools between mounts. Don't shut them down until the last
user goes away, and make sure to start it back up when needed.

Fixes: https://tracker.ceph.com/issues/47294
For real this time.
Maybe.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #38864 from MahatiC/wip-ssd-fix

librbd/cache/pwl: Rename config options

Reviewed-by: Xiaoyan Li <xiaoyan.li@intel.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tests: added stress-split-erasure-code-no-cephadm tests (not using cephadm)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #38751 from rhcs-dashboard/fix-48591-master

mgr/dashboard: Monitoring alert badge includes suppressed alerts

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mds: do not allow the service to run on Windows

Some file related flags and types are stubbed on Windows.
In order to avoid incorrect behavior, we're going to prevent
the service from running on Windows for now.

This does not apply to MDS clients.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #38879 from Rethan/remove-osd-heartbeat-option

doc/rados: remove unused osd heartbeat option

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38564 from ronen-fr/wip-ronenf-clangcrims-1

rgw: fixing compilation errors

Reviewed-by: Willem Jan Withagen (@wjwithagen)
Reviewed-by: Matt Benjamin mbenjamin@redhat.com
Reviewed-by: Daniel Gryniewicz dang@redhat.com dang@fprintf.net
Reviewed-by: Adam C. Emerson aemerson@redhat.com
Reviewed-by: Yuval Lifshitz ylifshit@redhat.com

Merge PR #37095 into master

* refs/pull/37095/head:
mds: make purge_queue delete objects asynchronously and keep accepting pushes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37876 into master

* refs/pull/37876/head:
systemd: cephfs-mirror systemd unit files
doc, man: man page for `cephfs-mirror` tool
doc: document mirror daemon internals
cephfs-mirror: switch to using PeerReplayer class
cephfs-mirror: cancel ongoing snapshot syncs on dir removal
cephfs-mirror: display peer snapshot sync stats
cephfs-mirror: carve out (and implement) mirroring snapshots to peers
cephfs-mirror: remove `cephfs_mirror_directory_choose_policy` option
cephfs-mirror: include helper routines to separate source
cephfs-mirror: remove peer only when peer is tracked
cephfs-mirror: typedef ceph_mount_info as MountRef shared pointer
cephfs-mirror: enclose json dump in object section
cephfs-mirror: note current peer set
cephfs-mirror: fix option typo and document certain options
cephfs-mirror: remove unnecessary command line options
cephfs-mirror: default log level 0/5

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38820 into master

* refs/pull/38820/head:
include: use narrower ostream instead of iostream
mds: remove unused fstream header
mds: use narrower ostream header instead of sstream
mds: remove unused iostream header
mds: remove unused unix headers

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38835 into master

* refs/pull/38835/head:
qa: add test for fs rm idempotency
mon/FSCommands: restore idempotent behavior of fs rm
qa: add ceph cmd helper
qa: allow kwargs for raw_cluster_cmd

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38848 into master

* refs/pull/38848/head:
qa: ignore MDS_SLOW_METADATA_IO with osd thrasher

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #38886 from cbodley/wip-rgw-pubsub-warning

rgw: fix initialization of s3_event in pubsub

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #38882 from tchaikov/wip-no-print-nanoseconds

tests/test_bluestore_types: cast duration to ceph::timespan

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

librbd: add encryption cli

This commit adds rbd encryption cli (encryption format and nbd map)

Signed-off-by: Or Ozeri <oro@il.ibm.com>

qa: add delays only for osd/mds

The delays were applied everywhere and needlessly interfere with test
commands sent to mons from the ceph admin command. Furthermore, the
delays would not affect the kernel client. Now the delays are performed
by the MDS on clients.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38652 into master

* refs/pull/38652/head:
mgr/volumes: Filter inherited snapshots while listing snapshots

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38785 from sebastian-philipp/cephadm-move-create-daemon-to-serve

mgr/cephadm: move a bunch of methods from module.py to serve.py

Reviewed-by: Michael Fritch <mfritch@suse.com>

rgw: fix initialization of s3_event in pubsub

resolves compiler warning:

rgw_sync_module_pubsub.cc: In constructor ‘RGWPSHandleObjEventCR::RGWPSHandleObjEventCR(RGWDataSyncCtx*, PSEnvRef, const rgw_user&, EventRef<rgw_pubsub_event>&, EventRef<rgw_pubsub_s3_event>&, const TopicsRef&)’:
rgw_sync_module_pubsub.cc:1028:3: warning: ‘RGWPSHandleObjEventCR::s3_event’ is initialized with itself [-Winit-self]
 1028 |   RGWPSHandleObjEventCR(RGWDataSyncCtx* const _sc,
      |   ^~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Casey Bodley <cbodley@redhat.com>

tests/test_bluestore_types: cast duration to ceph::timespan

on FreeBSD + Clang + libc++ + amd64, std::chrono::nanoseconds::rep is `long
long` instead of `long`, so the explictly instantiated template
operator<< operator instances in `src/common/ceph_time.cc` are not able to
cover this use case. so in this change, the duration is casted to
ceph::timespan instead of nanoseconds. so we can ensure that `operator<<` is
always available in this case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38860 from tchaikov/wip-48777

osd/pg_scrubber: do not print log in dtors

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #38724 from tchaikov/wip-install-deps-remove-mirror

install-deps.sh: do not use deb.rug.nl as a mirror anymore

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

win32*.sh: fix SKIP_TESTS flag

The "SKIP_TESTS" flag used by the Windows build script is
currently ignored.

This patch will skip some of the make targets when "SKIP_TESTS"
is set.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

include/win32/win32_errno:h Add missing include

We'll need to include "include/int_types.h" for the "__s32" type
definition.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

client: Port CephFS client to Windows

This patch provides a few changes required in order to use the
CephFS client on Windows.

A subsequent patch will make use of it along with Dokan, an
interface similar to Fuse.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

doc/rados: remove unused osd heartbeat option

Since commit a5f9ca, option osd_mon_ack_timeout was removed, change
doc accordingly.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

mgr/volumes: Filter inherited snapshots while listing snapshots

Filter inherited snapshots resulted as part of a snapshot
at ancestor level while listing snapshots of a subvolume
and subvolumegroup

Also, fail the snapshot info on inherited snapshot.

Fixes: https://tracker.ceph.com/issues/48501
Signed-off-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #33692 from majianpeng/osd-remove-useless-code

osd: remove usless check condition.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38765 from tchaikov/wip-mutex-debug-atomic

common/mutex_debug: add memory barrier before load/store nlock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd/cache/pwl: Rename config options

Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #38696 from cyx1231st/wip-seastar-onode-tree-denc-delta

crimson/onode-staged-tree: implement delta encode, decode, replay and tests

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: set indent-tabs-mode to nil

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement get_type() for test extents

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: implement seastore replay test for onode-staged-tree

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38806 from yuriw/wip-yuriw-octopus-x-master

qa/tests: added rbd_import_export workload

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tests: added stress-split-no-cephadm tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #38849 into master

* refs/pull/38849/head:
doc: document ms_mode option in the mount.ceph manpage

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38793 into master

* refs/pull/38793/head:
qa: check for rank hole without racy joinable flag gymnastics
qa: skip client mounts for TestResize

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa: add test for fs rm idempotency

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/FSCommands: restore idempotent behavior of fs rm

Fixes: https://tracker.ceph.com/issues/48808
Fixes: 811e48d6ffd841ecd93a57eb00c6aceae476f4f8
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add ceph cmd helper

A more programmer friendly command to use.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: allow kwargs for raw_cluster_cmd

And refactor.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

*: remove legacy ceph_volume_client.py library

This library is obsolete with the mgr volumes plugin since Nautilus.

The last remaining user of this library was Manila which will be using
the volumes plugin with Pacific and onwards.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd/pg_scrubber: do not print log in dtors

PG::recovery_state is defined after PG::m_scrubber, while
PG::gen_prefix() retrieves the osdmap from recovery_state. in
PgScrubber::~PgScrubber(), we print out the osdmap by dereferencing
recovery_state for the prefix of the logging message. in other words,
PgScrubber's destructor is referencing "recovery_state" which is
already destroyed.

in this change, the logging messages are removed from dtors. we can
print out the pgid in a follow-up PR if it helps.

Fixes: https://tracker.ceph.com/issues/48777
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38718 from tchaikov/wip-mgr-health-cleanup

mgr/devicehealth: use helpers to define options and commands

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

cephadm: Fix type errors reported by pyright

pyright is picky about `except ImportError` constructs
and also checks un-annotated code by default.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

systemd: cephfs-mirror systemd unit files

Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc, man: man page for `cephfs-mirror` tool

Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: document mirror daemon internals

... fix some typos and reorganize sections to be reader
friendly. Also include a requirements section.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: switch to using PeerReplayer class

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: cancel ongoing snapshot syncs on dir removal

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: display peer snapshot sync stats

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: carve out (and implement) mirroring snapshots to peers

Remote file synchronization by @mchangir.

Signed-off-by: Milind Changire <mchangir@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: remove `cephfs_mirror_directory_choose_policy` option

Basically, the mirror daemon would process directory list in in a
round-robin fashion. Randomly choosing directories probably does not
make much sense right now.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: include helper routines to separate source

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: remove peer only when peer is tracked

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: typedef ceph_mount_info as MountRef shared pointer

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: enclose json dump in object section

So that mirroring module can decode the JSON object.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: note current peer set

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: fix option typo and document certain options

s/cephfs_mirror_mirror_action_update_interval/cephfs_mirror_action_update_interval/

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: remove unnecessary command line options

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: default log level 0/5

Signed-off-by: Venky Shankar <vshankar@redhat.com>

.mailmap: map myself to sweil@redhat.com

Signed-off-by: Sage Weil <sweil@redhat.com>

Merge pull request #38863 from tchaikov/wip-osd-scrubber-tab

osd/pg_scrubber: use tab-width:8

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #34859 into master

* refs/pull/34859/head:
test: Initialize timer
test: Avoid named semaphores on Windows
test: Update cmake files, allowing tests to run on Windows
test: port Ceph tests to Windows

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd/pg_scrubber: use tab-width:8

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38781 from tchaikov/wip-googletest-gcc11

googletest: pick up change to silence error=maybe-uninitialized warning

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38809 from Rethan/fs-status-style

pybind/mgr: change format_bytes unit color from black to yellow

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38814 from tchaikov/wip-fix-throttle-test

common/Throttle: return new count in put() right away

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38811 from majianpeng/optimize-is_blackhole

msg: using switch-case in place of if-else.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38780 from majianpeng/fix-wrong-num_ios

os/bluestore: fix wrong num_ios for read op.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #38837 from vbashkirtsev/master

common/options.h: use uint64_t for TYPE_SIZE options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: document ms_mode option in the mount.ceph manpage

Fixes: https://tracker.ceph.com/issues/48838
Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/cephadm: tolerate old host inventory without 'hostname' key

Older cephadm clusters lack the 'hostname' key in the host spec.  e.g.,

 "cpach": {"addr": "cpach", "labels": ["mon"]}, "eutow": {"addr": "eutow", "labels": ["mon"]}, "stud": {"addr": "stud", "labels": ["mon"]}}

Populate hostname from the dict key if necessary for compatibility.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tests: added rbd_import_export tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #38768 from votdev/fix_path_to_dash_devel_doc

mgr/dashboard: Fix path to dash_devel.rst

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>