mgr/ServiceMap: move ServiceMap::Daemon implementation into .cc file

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36103 from dillaman/wip-44311

crush/CrushWrapper: rebuild reverse maps after rebuilding crush map

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #35768 from rhcs-dashboard/fix-45185-master

mgr/dashboard: fix pool usage calculation

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #36095 from orozery/striper_get_file_offset

osdc/Striper: add get_file_offset function

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36091 from tchaikov/wip-46495

mgr/DaemonServer: keep daemon in service map when it has no tasks

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #36051 from Songweibin/wip-rados-setomapval

tools/rados: add an option "--input-file file" to `setomapval` command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35130 from rzarzynski/wip-crimson-backfill

crimson: add support for backfill, part 0

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33770 from majianpeng/osd-simple-logger

osd/OSD: simple perfcounter usage in OSDService

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35989 from bigwhite37/fix-dedup-tool-unused-var-warn

tools/ceph_dedup_tool: fix unused variable warning

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30780 from zhangsw/fix-rgw-multipart-abortdate

rgw: add abort multipart date and rule-id header to init multipart upload response

Merge pull request #35950 from zdover23/wip-doc-dev-guide-add-zdover-to-lead-list

doc/devel: add zdover to leads list

Merge pull request #34537 from ideepika/wip-bluestore-mem-tracking

os/bluestore: mempool split for finer granularity

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #35901 from sebastian-philipp/juanmi-firewall-refactor

cephadm: refactor update_firewalld

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35862 from adk3798/cephadm_45724

mgr/cephadm: check-host should not fail as hard using fqdn

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35471 from sebastian-philipp/cephadm-daemon-spec

mgr/cephadm: Add CephadmDaemonSpec class

Merge pull request #35838 from mgfritch/cephadm-rgw-spec-validate

python-common: RGW service_id might not contain a zone

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/devel: add zdover to leads list

This commit adds "zdover" to the list of
leads as the Docs Lead.

Fixes: https://tracker.ceph.com/issues/46378
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #36096 from BenoitKnecht/doc-osd-scrub-during-recovery

doc/rados: Fix osd_scrub_during_recovery default value

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #36052 from smithfarm/wip-focce

bluestore: fix typo in _flush function declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35991 from votdev/adapt_cephadm_vagrantbox_to_el8

mgr/cephadm: Adapt Vagrantbox to el8

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35315 from weixinwei/master

blk:BlockDevice.cc: use pending_aios instead of iovec size as ios num

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #35844 from wuhongsong/fuse-assert-Dprocess

common:  ignore SIGHUP prior to fork

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36054 from tchaikov/wip-blk-no-unused-label

blk/BlockDevice: use ceph_assertf() for assert

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #35744 from jmolmo/document_unmanaged_use

doc: Add information about unmanaged parameter

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

mgr/cephadm: Add test for _check_daemons()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: refactor `_create_daemon()`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add CephadmDaemonSpec class

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35709 from zdover23/wip-doc-preferring-yaml-to-cli-in-cephadm

doc/cephadm: add yaml info for orch apply mon

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/DaemonServer: keep daemon in service map when it has no tasks

we should not remove a daemon from service map even if it has no task
running, as it is not an evidence that the service is removed from the
cluster.

this change partially reverts 5a7d0632d989536e09b2d916b5ea9713dc8dbea5

Fixes: https://tracker.ceph.com/issues/46495
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/DaemonServer: pass task_status instead of a message

be explicit that the caller is only interested in the task_status

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29283 from myoungwon/wip-refcount-snap

osd: refcounting chunks for snapshotted manifest object

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com> (portions Sam wrote)

crush/CrushWrapper: rebuild reverse maps after rebuilding crush map

The Objecter will crash when localized reads are enabled and two threads
attempt to rebuild the (invalidated) reverse maps concurrently. This
should address the issue for the Objecter use-case without the need to
add additional locking.

Fixes: https://tracker.ceph.com/issues/44311
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35955 from jschmid1/deprecate_filstore

mgr/cephadm: do not accept filestore osds in osdspecs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

doc/cephadm: add yaml infor for orch apply mon

This commit breaks one of the long sentences into
three shorter sentences, and adds a parenthetical
comment walking the reader through a series of commands,
explaining what each command does and why only the last
of the commands ends up having a lasting effect on the Ceph
environment.

Fixes: https://tracker.ceph.com/issues/46133
Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/cephadm: check-host should not fail as hard using fqdn

Print error message instead of traceback when check-host
fails in due to host address not being found

Fixes: https://tracker.ceph.com/issues/45724
Signed-off-by: Adam King <adking@redhat.com>

cephadm: refactor update_firewalld

* split into three different methods
* untangle service and port lists

This helps in making our firewall handling more dynamic.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Co-authored-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

doc/rados: Fix osd_scrub_during_recovery default value

Since 8dca17c, `osd_scrub_during_recovery` defaults to `false`, but the
documentation was still stating that its default value is `true`.

Fixes: https://tracker.ceph.com/issues/46531
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

osdc/Striper: add get_file_offset function

This commit adds a get_file_offset translating (object_no, object_off) -> file_offset.
This is useful for encryption object dispatch layer in librbd
to comply with disk-encryption standards that require the file offset as input.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #36013 from Devp00l/wip-46376

cephadm: Make Vagrantfile more flexible

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36072 from zdover23/wip-doc-dev-word-choice-guidance

doc/dev: remove extraneous word "guidance".

Merge pull request #35931 from tchaikov/wip-mds-denc

mds,denc: prepare for the enabling of denc support of EMetaBlob::fullbit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35755 into master

* refs/pull/35755/head:
mgr/volumes: Deprecate protect/unprotect CLI calls for subvolume snapshots

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Victoria Martinez de la Cruz <vkmc@redhat.com>
Reviewed-by: Goutham Pacha Ravi <gouthamr@redhat.com>

Merge pull request #35970 from athanatos/sjust/wip-fix-lba-addrs

crimson/seastore: fix lba btree internal references

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34246 into master

* refs/pull/34246/head:
mds: add request to batch_op before taking auth pins and locks
mds: move MDRequestImpl::batch_reqs into Batch_Getattr_Lookup
mds: track which map batch_op is in-use in MDRequest

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #34785 into master

* refs/pull/34785/head:
ceph-fuse: show fuse helper options for libfuse >= 3.0
ceph-fuse: add splice read/write support to reduce the memory copy
ceph-fuse: add connection args parsing support for libfuse > 3.0
ceph-fuse: switch to fuse_reply_iov to reduce the memory copy

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #35327 into master

* refs/pull/35327/head:
client: fix directory inode can not call release callback

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35690 into master

* refs/pull/35690/head:
mds: fix hang issue when accessing a file under a lost parent directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35848 into master

* refs/pull/35848/head:
mds: do not submit omap_rm_keys if the dir is the basedir of merge.

Reviewed-by: Zheng Yan <zyan@redhat.com>

os/Bluefs: fix file reader/writer buffer mempool

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

bluestore: _get_used_bytes should reflect new mempool fields

with bluestore_cache_other split further, the _get_used_bytes should
take into account the new mempool splits as well.

see also: 0ce8ee4b123eacfc2bfb14f554487e928512415a
  68ee931fd4b6bf748f95b14dae31d11738fbaf1e

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

common/mempool: More detailed bluestore stats

Splits bluestore bluestore_cache_other, exposes accounting of some internals.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Introduce separate mempool for FileReaderBuffer and FileWriter

FileReaderBuffer and FileWriter uses Bufferlists which needs to be
tracked by mempool to get the proper memory usage.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

split mempool allocation for bluestore_cache_other

While doing root cause analysis it bluestore_cache_other gives a bit of
a crude estimate, something more helpful would be to have it split into
the following fields:

- bluestore_cache_onode
- bluestore_cache_meta
- bluestore_onode_attrs

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge PR #35951 into master

* refs/pull/35951/head:
qa/cephfs: set omit_sudo to False for wait_until_mounted() in...

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os/seastore: fix internal relative btree addrs

Internal relative addrs must be block relative on disk and record
relative in deltas.  Use maybe_generate_relative as intended.

Signed-off-by: Samuel Just <sjust@redhat.com>

seastore/test_transaction_manager: add a test which forces an lba split

Signed-off-by: Samuel Just <sjust@redhat.com>

seastore/test_btree_lba_manager: ensure record and block addrs are different

Adds an extra extent to each transaction in the test to ensure that
record and block relative addresses differ as they do in real usage.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #34920 from ofriedma/ofriedma-civetweb-timeout

rgw: increase civetweb request timeout to 65 seconds

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

crimson/osd: automate state logging in the backfill FSM.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: refactor needs_recovery() and needs_backfill() of PeeringState.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: synchronize BackfillRecovery with PeeringEvent.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: recovery locking uses ceph_assert_always().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement MOSDPGBackfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement BackfillRecoveryPipeline.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: improve debugs in PrimaryLogPG::do_scan().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement BackfillListener::backfilled().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement BackfillListener::enqueue_push().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: short-circuit budget_available().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: wire-up pg scanning on replicas.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: wire-up BackfillState and pg scanning on primary.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: introduce PGRecovery::start_backfill_recovery().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: settle BackfillState in PGRecovery.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: PGRecovery provides dummy BackfillListener.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: make BackgroundRecovery abstract for the sake of backfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: decouple backfill from PG::start_recovery_ops().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: bring the boost::statechart-based FSM for backfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: refactor PrimaryLogPG::recover_backfill() a little bit.

This is change is intended to only refactor the code.
The main theme is just declaring the `backfill_pos` closer
to its usage.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: refactor PeeringState::find_best_info() a little bit.

This is change is intended to only refactor the code. All
it does is just declaring the `min_last_update_acceptable`
closer to its usage.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: sketch the backfill machine and its interfaces.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: send MOSDPGTemp on peering and map advance.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: rework handling of MOSDAlive.

1. Migrate `MOSDAlive` crafting from `OSD` to `ShardServices`.
2. `send_alive()` takes the `want` epoch from callers
   instead of assuming it matches current OSDMap.
   Callers pass `pg_history_t::same_interval_since`.
   The passed epoch is handled similarly to `queue_want_up_thru()`
   in the classical.
   The message is sent once and when the desired epoch is greater
   than the one returned by `OSDMap::get_up_thru()`.
3. Send the message also when completing the `PeeringEvent`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: link up the documentation for last_epoch_started.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement PG scanning.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/dev: remove extraneous word "guidance".

This bug changes the sentence "If your patches emit
messages to the Ceph cluster log, please consult this
guidance:" to "If your patches emit messages to the
Ceph cluster log, please consult this:". This new sentence
is more natural and idiomatic.

Fixes: https://tracker.ceph.com/issues/46363
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #36055 from zli091/master

 rgw: lc: fix Segmentation Fault when the tag of the object was not f…

Reviewed-by: Or Friedmann <ofriedma@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

cephadm: Make Vagrantfile more flexible

Now you can use a JSON or pass multiple variable to vagrant in order to
configure the outcome of VMs you get. Similar to vstart.sh you can use
OSDS, MGRS and MONS as arguments to pass. As OSDS behave a bit different
in this scenario you can also specify the amount of extra disks and OSD
VM has.

Fixes: https://tracker.ceph.com/issues/46376
Signed-off-by: Stephan Müller <smueller@suse.com>

tools/rados: add a parameter "-i infile" to `setomapval` command

Signed-off-by: songweibin <song.weibin@zte.com.cn>

common: ignore SIGHUP prior to fork

Otherwise, the ceph-fuse process is terminated by the logratote task and will produce one Uninterruptible Sleep process.

Fixes: http://tracker.ceph.com/issues/46269
Signed-off-by: hzwuhongsong <hzwuhongsong@corp.netease.com>

blk/zoned: do not use g_ceph_context for reading conf

use local `cct` instead, more consistent this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

blk/BlockDevice: use ceph_assertf() for assert

also silences a warning like

src/blk/BlockDevice.cc:158:1: warning: label 'out_fail' defined but not used [-Wunused-label]
  158 | out_fail:
      | ^~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSD: simple perfcounter usage in OSDService

Move create_logger()/create_recoverystate_perf() into construct to
avoid check logger in OSDService. And avoid in destructor delete
nullptr.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rgwlc: fix Segmentation Fault related to tags.

It is found that rgw crashes when putting a tag beyond the rule to the object, which can be solved by adding the return value judgment.

Fixes: https://tracker.ceph.com/issues/46485
Signed-off-by: zhuo li <lizhuo@chinatelecom.cn>
Signed-off-by: yupeng chen <chenyupeng@chinatelecom.cn>

Merge pull request #36008 from idryomov/wip-krbd-58-options

rbd: recognize crush_location, read_from_replica and compression_hint map options

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mds: do not decode EMetaBlob::fullbit struct_v < 8

the struct_v of EMetaBlob::fullbit::encode was bumped from 7 to 8 in
d62c67ca2041797fc7aa212ae4a37ba91e5620e7, the oldest release containing
this change was hammer rc1.

`EMetaBlob::fullbit` is used to persist dentry + inode as part of
journal, after the journal is flushed to the metadata pool, and applied to
dirty metadata, the flushed segments are trimmed. also, it's not allowed
to upgrade a Ceph cluster across 2 releases. so, there is no need
to decode an on-disk struct which will be trimmmed as part of journal
recycling, and that struct was encoded in hammer.

Signed-off-by: Kefu Chai <kchai@redhat.com>

denc: add boost::container::small_vector support

i could use `_denc::container_base<>` and `_denc::pushback_details<>` to
implement the traits for boost::container::small_vector (bcs for short),
but the second template parameter of bcs is a value of type size_t, so
the value it not a type, forunately, C++17 allows us to declare non-type
template parameter using `auto`, but when instantiating
_denc::container_base<>, the compiler complains that it expects a
constatnt of `auto` but `T` is passed in. because i have to change the
signature of `container_base` to something like
```
template<template<auto...> class C, typename Details, auto ...Ts>
  struct container_base {
```

so, in this change, the traits for bsc is implemented from scratch.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36041 from tchaikov/wip-mds-EMetaBlob

ceph-object-corpus: pick up changes to drop backward compatibility of…

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-object-corpus: pick up changes to drop backward compatibility

to drop backward compatibility of EMetaBlob

see also https://github.com/ceph/ceph-object-corpus/pull/12

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35747 from adk3798/cephadm-44877

mgr/cephadm: allow custom dashboard grafana url

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #32852 from ifed01/wip-ifed-simplify-pin

os/bluestore: simplify Onode pin/unpin logic.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35973 from mlausch/mgr-progress-perf

mgr/progress: Skip pg_summary update if _events dict is empty.

Reviewed-by: Mykola Golub <mgolub@suse.com>