osd: fix reference leak

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

test: use existing replicated tier instead of using ec_overwrite

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

test: fix to use set-chunk (with reference)

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

tool, test: make with-reference default when called set-chunk

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: chunk_info must hold a reference

Every mapping in chunk_map must have a corresponding refcount

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

test: adjust test cases based on a new flush scheme

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: make checking intersect range simple

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd, test: remove all things related to manifest_flush

Due to a plan to re-work based on CDC-based flush,
remove manifest_flush()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

test: add refcount increment test on snapshot

add refcount increment test on snapshotted manifest object
when using set-chunk

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: increment refcount if adjacent clones do not have a same chunk

Upon set-chunk, check a previous snapshot and increment the target
reference if there is no the same chunk in the previous snapshot

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

test: Consider a tier_promote case deleting redirect info

Since calling tier_promote triggers to remove redirect
manifest info in the target object, unset_manifest can get
the response with EOPNOTSUPP

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: remove existing object before rename is called

In redirected manifest case, the object exists in the upper tier.
So, to avoid a conflict when rename() is called, remove existing
object first.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: drop the reference if redirect has the reference

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: clear <off, chunk_info>  if the chunk is dedup chunk

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

test: add tests to cover overlapped cases when using
calc_refs_to_drop_on_modify()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

test: remove unused code and fix test scenario

Fix test scenario due to the reference decrement
upon modification

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: make dec_refcount simple

Use hobject_t as a input paramter instead of ObjectContextRef

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

src/test: clean up ec test cases

Use write_full to create an object and adjust test
scenario because all dirty chunks do not lead to flush

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd, test: do not increment refcount if a previous snapshot have a same chunk

When flush is started, check a previous snapshot and making target chunk's
state clean if there is the same chunk in the previous snapshot

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

src/test: adjust test scenario based on modified refcount scheme

Based on the reference decrement upon modification,
test scenario is adjusted with cleanups

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

src/test: remove unused code

no need to configure existing tier mode

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

src/test: create a instance and use tier-flush

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

src/test: use tier-flush explicitly

flush no longer occurs when all chunks are dirty

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

src/test: prevent truncate from using write_full

Use write instead of write_full to avoid truncate
even if set-chunk is set

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

src/test: add a test that can verify dropping the reference when modification occurs

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: drop the reference right after modification

To simply chunk_info_t state and keep unreferenced chunk minimal,
we prefer to drop the reference and remove manifest pointer.
Also, FLAG_DIRTY will be no longer used in chunk_info_t.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: add a test to check that the redirect is clear after promote

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: clear redirect after promote

We do want PROMOTE to be able to atomically replace
a redirect with the actual object, so the solution
is to clear the redirect at the end of the promote.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

src/test: verify that set-chunk is working right on the clone

1. check whether set-chunk triggers a clone
2. verify that set-chunk make a chunk on a clone

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: do not trigger clone when set_chunk is called

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: remove dead code

the code is no longer useful

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: do not make the chunk state missing upon set_chunk

Since the missing state causes a promote on clone
after set_chunk is set, we remove the missing state.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: return ERANGE if set_chunk tries to set large range
over the object size

We allow existing snapshot to be set as a chunk. So,
return ERAMGE if set_chunk's range is over the object
size

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: make set_chunk as ReadOperation

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

Merge pull request #35989 from bigwhite37/fix-dedup-tool-unused-var-warn

tools/ceph_dedup_tool: fix unused variable warning

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30780 from zhangsw/fix-rgw-multipart-abortdate

rgw: add abort multipart date and rule-id header to init multipart upload response

Merge pull request #35950 from zdover23/wip-doc-dev-guide-add-zdover-to-lead-list

doc/devel: add zdover to leads list

Merge pull request #34537 from ideepika/wip-bluestore-mem-tracking

os/bluestore: mempool split for finer granularity

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #35901 from sebastian-philipp/juanmi-firewall-refactor

cephadm: refactor update_firewalld

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35862 from adk3798/cephadm_45724

mgr/cephadm: check-host should not fail as hard using fqdn

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35471 from sebastian-philipp/cephadm-daemon-spec

mgr/cephadm: Add CephadmDaemonSpec class

Merge pull request #35838 from mgfritch/cephadm-rgw-spec-validate

python-common: RGW service_id might not contain a zone

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/devel: add zdover to leads list

This commit adds "zdover" to the list of
leads as the Docs Lead.

Fixes: https://tracker.ceph.com/issues/46378
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #36096 from BenoitKnecht/doc-osd-scrub-during-recovery

doc/rados: Fix osd_scrub_during_recovery default value

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #36052 from smithfarm/wip-focce

bluestore: fix typo in _flush function declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35991 from votdev/adapt_cephadm_vagrantbox_to_el8

mgr/cephadm: Adapt Vagrantbox to el8

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35315 from weixinwei/master

blk:BlockDevice.cc: use pending_aios instead of iovec size as ios num

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #35844 from wuhongsong/fuse-assert-Dprocess

common:  ignore SIGHUP prior to fork

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36054 from tchaikov/wip-blk-no-unused-label

blk/BlockDevice: use ceph_assertf() for assert

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #35744 from jmolmo/document_unmanaged_use

doc: Add information about unmanaged parameter

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

mgr/cephadm: Add test for _check_daemons()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: refactor `_create_daemon()`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add CephadmDaemonSpec class

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35709 from zdover23/wip-doc-preferring-yaml-to-cli-in-cephadm

doc/cephadm: add yaml info for orch apply mon

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #29283 from myoungwon/wip-refcount-snap

osd: refcounting chunks for snapshotted manifest object

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com> (portions Sam wrote)

Merge pull request #35955 from jschmid1/deprecate_filstore

mgr/cephadm: do not accept filestore osds in osdspecs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

doc/cephadm: add yaml infor for orch apply mon

This commit breaks one of the long sentences into
three shorter sentences, and adds a parenthetical
comment walking the reader through a series of commands,
explaining what each command does and why only the last
of the commands ends up having a lasting effect on the Ceph
environment.

Fixes: https://tracker.ceph.com/issues/46133
Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/cephadm: check-host should not fail as hard using fqdn

Print error message instead of traceback when check-host
fails in due to host address not being found

Fixes: https://tracker.ceph.com/issues/45724
Signed-off-by: Adam King <adking@redhat.com>

cephadm: refactor update_firewalld

* split into three different methods
* untangle service and port lists

This helps in making our firewall handling more dynamic.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Co-authored-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

doc/rados: Fix osd_scrub_during_recovery default value

Since 8dca17c, `osd_scrub_during_recovery` defaults to `false`, but the
documentation was still stating that its default value is `true`.

Fixes: https://tracker.ceph.com/issues/46531
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

Merge pull request #36013 from Devp00l/wip-46376

cephadm: Make Vagrantfile more flexible

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36072 from zdover23/wip-doc-dev-word-choice-guidance

doc/dev: remove extraneous word "guidance".

Merge pull request #35931 from tchaikov/wip-mds-denc

mds,denc: prepare for the enabling of denc support of EMetaBlob::fullbit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35755 into master

* refs/pull/35755/head:
mgr/volumes: Deprecate protect/unprotect CLI calls for subvolume snapshots

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Victoria Martinez de la Cruz <vkmc@redhat.com>
Reviewed-by: Goutham Pacha Ravi <gouthamr@redhat.com>

Merge pull request #35970 from athanatos/sjust/wip-fix-lba-addrs

crimson/seastore: fix lba btree internal references

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34246 into master

* refs/pull/34246/head:
mds: add request to batch_op before taking auth pins and locks
mds: move MDRequestImpl::batch_reqs into Batch_Getattr_Lookup
mds: track which map batch_op is in-use in MDRequest

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #34785 into master

* refs/pull/34785/head:
ceph-fuse: show fuse helper options for libfuse >= 3.0
ceph-fuse: add splice read/write support to reduce the memory copy
ceph-fuse: add connection args parsing support for libfuse > 3.0
ceph-fuse: switch to fuse_reply_iov to reduce the memory copy

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #35327 into master

* refs/pull/35327/head:
client: fix directory inode can not call release callback

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35690 into master

* refs/pull/35690/head:
mds: fix hang issue when accessing a file under a lost parent directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35848 into master

* refs/pull/35848/head:
mds: do not submit omap_rm_keys if the dir is the basedir of merge.

Reviewed-by: Zheng Yan <zyan@redhat.com>

os/Bluefs: fix file reader/writer buffer mempool

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

bluestore: _get_used_bytes should reflect new mempool fields

with bluestore_cache_other split further, the _get_used_bytes should
take into account the new mempool splits as well.

see also: 0ce8ee4b123eacfc2bfb14f554487e928512415a
  68ee931fd4b6bf748f95b14dae31d11738fbaf1e

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

common/mempool: More detailed bluestore stats

Splits bluestore bluestore_cache_other, exposes accounting of some internals.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Introduce separate mempool for FileReaderBuffer and FileWriter

FileReaderBuffer and FileWriter uses Bufferlists which needs to be
tracked by mempool to get the proper memory usage.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

split mempool allocation for bluestore_cache_other

While doing root cause analysis it bluestore_cache_other gives a bit of
a crude estimate, something more helpful would be to have it split into
the following fields:

- bluestore_cache_onode
- bluestore_cache_meta
- bluestore_onode_attrs

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge PR #35951 into master

* refs/pull/35951/head:
qa/cephfs: set omit_sudo to False for wait_until_mounted() in...

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os/seastore: fix internal relative btree addrs

Internal relative addrs must be block relative on disk and record
relative in deltas.  Use maybe_generate_relative as intended.

Signed-off-by: Samuel Just <sjust@redhat.com>

seastore/test_transaction_manager: add a test which forces an lba split

Signed-off-by: Samuel Just <sjust@redhat.com>

seastore/test_btree_lba_manager: ensure record and block addrs are different

Adds an extra extent to each transaction in the test to ensure that
record and block relative addresses differ as they do in real usage.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #34920 from ofriedma/ofriedma-civetweb-timeout

rgw: increase civetweb request timeout to 65 seconds

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc/dev: remove extraneous word "guidance".

This bug changes the sentence "If your patches emit
messages to the Ceph cluster log, please consult this
guidance:" to "If your patches emit messages to the
Ceph cluster log, please consult this:". This new sentence
is more natural and idiomatic.

Fixes: https://tracker.ceph.com/issues/46363
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #36055 from zli091/master

 rgw: lc: fix Segmentation Fault when the tag of the object was not f…

Reviewed-by: Or Friedmann <ofriedma@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

cephadm: Make Vagrantfile more flexible

Now you can use a JSON or pass multiple variable to vagrant in order to
configure the outcome of VMs you get. Similar to vstart.sh you can use
OSDS, MGRS and MONS as arguments to pass. As OSDS behave a bit different
in this scenario you can also specify the amount of extra disks and OSD
VM has.

Fixes: https://tracker.ceph.com/issues/46376
Signed-off-by: Stephan Müller <smueller@suse.com>

common: ignore SIGHUP prior to fork

Otherwise, the ceph-fuse process is terminated by the logratote task and will produce one Uninterruptible Sleep process.

Fixes: http://tracker.ceph.com/issues/46269
Signed-off-by: hzwuhongsong <hzwuhongsong@corp.netease.com>

blk/zoned: do not use g_ceph_context for reading conf

use local `cct` instead, more consistent this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

blk/BlockDevice: use ceph_assertf() for assert

also silences a warning like

src/blk/BlockDevice.cc:158:1: warning: label 'out_fail' defined but not used [-Wunused-label]
  158 | out_fail:
      | ^~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgwlc: fix Segmentation Fault related to tags.

It is found that rgw crashes when putting a tag beyond the rule to the object, which can be solved by adding the return value judgment.

Fixes: https://tracker.ceph.com/issues/46485
Signed-off-by: zhuo li <lizhuo@chinatelecom.cn>
Signed-off-by: yupeng chen <chenyupeng@chinatelecom.cn>

Merge pull request #36008 from idryomov/wip-krbd-58-options

rbd: recognize crush_location, read_from_replica and compression_hint map options

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mds: do not decode EMetaBlob::fullbit struct_v < 8

the struct_v of EMetaBlob::fullbit::encode was bumped from 7 to 8 in
d62c67ca2041797fc7aa212ae4a37ba91e5620e7, the oldest release containing
this change was hammer rc1.

`EMetaBlob::fullbit` is used to persist dentry + inode as part of
journal, after the journal is flushed to the metadata pool, and applied to
dirty metadata, the flushed segments are trimmed. also, it's not allowed
to upgrade a Ceph cluster across 2 releases. so, there is no need
to decode an on-disk struct which will be trimmmed as part of journal
recycling, and that struct was encoded in hammer.

Signed-off-by: Kefu Chai <kchai@redhat.com>

denc: add boost::container::small_vector support

i could use `_denc::container_base<>` and `_denc::pushback_details<>` to
implement the traits for boost::container::small_vector (bcs for short),
but the second template parameter of bcs is a value of type size_t, so
the value it not a type, forunately, C++17 allows us to declare non-type
template parameter using `auto`, but when instantiating
_denc::container_base<>, the compiler complains that it expects a
constatnt of `auto` but `T` is passed in. because i have to change the
signature of `container_base` to something like
```
template<template<auto...> class C, typename Details, auto ...Ts>
  struct container_base {
```

so, in this change, the traits for bsc is implemented from scratch.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36041 from tchaikov/wip-mds-EMetaBlob

ceph-object-corpus: pick up changes to drop backward compatibility of…

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-object-corpus: pick up changes to drop backward compatibility

to drop backward compatibility of EMetaBlob

see also https://github.com/ceph/ceph-object-corpus/pull/12

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35747 from adk3798/cephadm-44877

mgr/cephadm: allow custom dashboard grafana url

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #32852 from ifed01/wip-ifed-simplify-pin

os/bluestore: simplify Onode pin/unpin logic.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35973 from mlausch/mgr-progress-perf

mgr/progress: Skip pg_summary update if _events dict is empty.

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35915 from jmolmo/fix_rook_create_osd

mgr/rook: Fix error creating OSDs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35968 from optimistyzy/73_debug

NVMEDevice: Fix the issue of multiple OSDs when using SPDK

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32707 from petrutlucian94/windows.4

Windows support [part 4]

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35971 from tchaikov/wip-addr-parsing-strncpy

common/addr_parsing: refactor safe_cat()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35990 from votdev/fix_cephadm_py_code

cephadm: Fix Python formating issues

Reviewed-by: Kefu Chai <kchai@redhat.com>