crimson/os/seastore/transaction_manager: complete dec_ref

Previously, dec_ref didn't handle actually retiring the
extent from the cache.  dec_ref will now reach into the
cache and mark the extent retired if it exists either
in the cache or in the current transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: clarify ref count operation return

Previously, we returned a refcount from inc_ref and dec_ref.  Now,
return the paddr as well for future code accounting for released
extents.

In addition, replumb btree_lba_manager to return an enoent error if
the mapping does not exist, and the resulting refcount, paddr
otherwise with a refcount of 0 indicating that the mapping has
been removed.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: don't mark invalid extents clean

In the event that an extent is created and removed in the same
transaction (invalid extent in fresh_block_list), update
block specific metadata but don't add to cache and especially
do not mark clean.  LBAManager::complete_commit implementations
are meant to use skip based on state in the same way.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cached_extent: add print_detail for logical extents

This way we can always see laddr in debug output.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore: add fake paddr for better test debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: link in LBAPin after read

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: initialize dirty extents after replay

lba_manager needs to preload BtreeLBAPin and parent nodes.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: improved debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: wire in btree pin set

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: add lba pin structure

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: actually use LBAManager::complete_transaction

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/: add a meta field to FixedKVNodeLayout and use in LBANode*

Also removes LBANode::depth.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: move BtreeLBAPin to btree_range_pin.h

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: init laddr for LogicalCachedExtent on replay

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cached_extent: add has_pin and is_logical helpers

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: move LogicalCachedExtent to cached_extent.h

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore_types: initialize crcs

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/fixed_kv_node_layout: remove unnecessary/wrong friend declaration

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree/lba_btree_node: remove dup include

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/sesatore/lba_manager/btree: let leaf be depth 1

This way, we can let depth 0 be the actual logical extents.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: ignore invalid extents during commit

This happens when a transaction mutates and then retires an
extent.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: retain read set

We're going to be using the read set refs to ensure cache residence
until commit.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: drop retired extents from write_set and invalidate

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: add helpers

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: move transaction out of cache

Transaction is going to gain some knowledge of logical
addresses.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add const cast() method

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: replumb to pass op_context_t

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore: add TestBlockPhysical

TestBlock doesn't work without the rest of the TransactionManager
machinery.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #36128 from xxhdx1985126/wip-crimson-fix-osd-shutdown

crimson/osd: fix osd shutdown problem

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36142 from tchaikov/wip-crimson-cleanup

crimson: do not capture unused variables

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35987 from tspmelo/wip-update-lodash

mgr/dashboard: Fix some npm audit warnings

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #36138 into master

* refs/pull/36138/head:
doc/cephfs/nfs: Update about deploying nfs-ganesha with cephadm in vstart

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35960 from votdev/issue_46395_required_field_message

mgr/dashboard: Use same required field message accross the UI

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35777 from ofriedma/rgw-enoent-cache

rgw: add negative cache to the system object

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #36070 from tspmelo/wip-fix-async-unittest

mgr/dashboard: Block the import of async from @angular/core/testing

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #36113 from trociny/wip-snap-create-error-msg

librbd: fix snapshot create error message

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35974 from tspmelo/wip-clean-headers

mgr/dashboard: Clean custom header styles

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

crimson: do not capture unused variables

this change silences warnings like:

mson/osd/backfill_state.cc:131:11: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
      [&, this](const pg_log_entry_t& e) {
        ~~^~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36099 from dillaman/wip-rbd-nbd-hook

rbd-nbd: quisce hook should be installed to /usr/libexec/...

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35605 from ceph/osd-compact-on-start

osd: Add config option to compact OSD's object store on start

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/cephfs/nfs: Update about deploying nfs-ganesha with cephadm in vstart

Fixes: https://tracker.ceph.com/issues/46571
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #36122 from zdover23/wip-doc-rados-redundant-a-typo

doc/rados: remove a redundant "a" from a sentence

This sentence removes an "a" from a sentence, which makes the sentence grammatically correct.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: Block the import of async from @angular/core/testing

We no longer need this and it was causing unexpected results in some tests.

Fixes: https://tracker.ceph.com/issues/46500
Signed-off-by: Tiago Melo <tmelo@suse.com>

osd: Add config option to compact OSD's object store on start

By setting this configuration option an OSD will compact it's
store's OMAP on start.

This way admin's can trigger an offline compaction by setting this
configuration value to 'true' and then restarting the OSD.

This is easier than using tools like 'ceph-kvstore-tool' with requires
more manual work on the CLI and might be more difficult for users.

Signed-off-by: Wido den Hollander <wido@42on.com>

crimson/osd: fix osd shutdown problem

when doing full-system shutdown, monitors may go down before OSDs in which case
the osd shutdown hangs waiting for monc to successfully send the markmedown msg
to monitors

Fixes: https://tracker.ceph.com/issues/46564
Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

Merge pull request #36103 from dillaman/wip-44311

crush/CrushWrapper: rebuild reverse maps after rebuilding crush map

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #35768 from rhcs-dashboard/fix-45185-master

mgr/dashboard: fix pool usage calculation

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #36095 from orozery/striper_get_file_offset

osdc/Striper: add get_file_offset function

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36091 from tchaikov/wip-46495

mgr/DaemonServer: keep daemon in service map when it has no tasks

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #36051 from Songweibin/wip-rados-setomapval

tools/rados: add an option "--input-file file" to `setomapval` command

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rados: remove a redundant "a" from a sentence

This commit changes the (not quite) sentence "Once
you have a deployed a Ceph Storage Cluster, you may
begin operating your cluster." to "Once you have
deployed a Ceph Storage Cluster, you may begin
operating your cluster."

Fixes: https://tracker.ceph.com/issues/46554
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #35130 from rzarzynski/wip-crimson-backfill

crimson: add support for backfill, part 0

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33770 from majianpeng/osd-simple-logger

osd/OSD: simple perfcounter usage in OSDService

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35989 from bigwhite37/fix-dedup-tool-unused-var-warn

tools/ceph_dedup_tool: fix unused variable warning

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30780 from zhangsw/fix-rgw-multipart-abortdate

rgw: add abort multipart date and rule-id header to init multipart upload response

Merge pull request #35950 from zdover23/wip-doc-dev-guide-add-zdover-to-lead-list

doc/devel: add zdover to leads list

Merge pull request #34537 from ideepika/wip-bluestore-mem-tracking

os/bluestore: mempool split for finer granularity

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #35901 from sebastian-philipp/juanmi-firewall-refactor

cephadm: refactor update_firewalld

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35862 from adk3798/cephadm_45724

mgr/cephadm: check-host should not fail as hard using fqdn

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

librbd: fix snapshot create error message

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35471 from sebastian-philipp/cephadm-daemon-spec

mgr/cephadm: Add CephadmDaemonSpec class

Merge pull request #35838 from mgfritch/cephadm-rgw-spec-validate

python-common: RGW service_id might not contain a zone

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/devel: add zdover to leads list

This commit adds "zdover" to the list of
leads as the Docs Lead.

Fixes: https://tracker.ceph.com/issues/46378
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #36096 from BenoitKnecht/doc-osd-scrub-during-recovery

doc/rados: Fix osd_scrub_during_recovery default value

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #36052 from smithfarm/wip-focce

bluestore: fix typo in _flush function declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35991 from votdev/adapt_cephadm_vagrantbox_to_el8

mgr/cephadm: Adapt Vagrantbox to el8

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35315 from weixinwei/master

blk:BlockDevice.cc: use pending_aios instead of iovec size as ios num

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #35844 from wuhongsong/fuse-assert-Dprocess

common:  ignore SIGHUP prior to fork

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36054 from tchaikov/wip-blk-no-unused-label

blk/BlockDevice: use ceph_assertf() for assert

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #35744 from jmolmo/document_unmanaged_use

doc: Add information about unmanaged parameter

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

mgr/cephadm: Add test for _check_daemons()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: refactor `_create_daemon()`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add CephadmDaemonSpec class

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35709 from zdover23/wip-doc-preferring-yaml-to-cli-in-cephadm

doc/cephadm: add yaml info for orch apply mon

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/DaemonServer: keep daemon in service map when it has no tasks

we should not remove a daemon from service map even if it has no task
running, as it is not an evidence that the service is removed from the
cluster.

this change partially reverts 5a7d0632d989536e09b2d916b5ea9713dc8dbea5

Fixes: https://tracker.ceph.com/issues/46495
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/DaemonServer: pass task_status instead of a message

be explicit that the caller is only interested in the task_status

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29283 from myoungwon/wip-refcount-snap

osd: refcounting chunks for snapshotted manifest object

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com> (portions Sam wrote)

crush/CrushWrapper: rebuild reverse maps after rebuilding crush map

The Objecter will crash when localized reads are enabled and two threads
attempt to rebuild the (invalidated) reverse maps concurrently. This
should address the issue for the Objecter use-case without the need to
add additional locking.

Fixes: https://tracker.ceph.com/issues/44311
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-nbd: quisce hook should be installed to /usr/libexec/...

The rbd-nbd daemon expects the rbd-nbd_quiesce script to be installed
in the libexec directory on all distros.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35955 from jschmid1/deprecate_filstore

mgr/cephadm: do not accept filestore osds in osdspecs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

doc/cephadm: add yaml infor for orch apply mon

This commit breaks one of the long sentences into
three shorter sentences, and adds a parenthetical
comment walking the reader through a series of commands,
explaining what each command does and why only the last
of the commands ends up having a lasting effect on the Ceph
environment.

Fixes: https://tracker.ceph.com/issues/46133
Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/cephadm: check-host should not fail as hard using fqdn

Print error message instead of traceback when check-host
fails in due to host address not being found

Fixes: https://tracker.ceph.com/issues/45724
Signed-off-by: Adam King <adking@redhat.com>

cephadm: refactor update_firewalld

* split into three different methods
* untangle service and port lists

This helps in making our firewall handling more dynamic.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Co-authored-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

doc/rados: Fix osd_scrub_during_recovery default value

Since 8dca17c, `osd_scrub_during_recovery` defaults to `false`, but the
documentation was still stating that its default value is `true`.

Fixes: https://tracker.ceph.com/issues/46531
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

osdc/Striper: add get_file_offset function

This commit adds a get_file_offset translating (object_no, object_off) -> file_offset.
This is useful for encryption object dispatch layer in librbd
to comply with disk-encryption standards that require the file offset as input.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

mgr/dashboard: Fix some npm audit warnings

Version of lodash that was use by all packages was causing some warnings.

Fixes: https://tracker.ceph.com/issues/46431
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36013 from Devp00l/wip-46376

cephadm: Make Vagrantfile more flexible

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36072 from zdover23/wip-doc-dev-word-choice-guidance

doc/dev: remove extraneous word "guidance".

Merge pull request #35931 from tchaikov/wip-mds-denc

mds,denc: prepare for the enabling of denc support of EMetaBlob::fullbit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35755 into master

* refs/pull/35755/head:
mgr/volumes: Deprecate protect/unprotect CLI calls for subvolume snapshots

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Victoria Martinez de la Cruz <vkmc@redhat.com>
Reviewed-by: Goutham Pacha Ravi <gouthamr@redhat.com>

Merge pull request #35970 from athanatos/sjust/wip-fix-lba-addrs

crimson/seastore: fix lba btree internal references

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34246 into master

* refs/pull/34246/head:
mds: add request to batch_op before taking auth pins and locks
mds: move MDRequestImpl::batch_reqs into Batch_Getattr_Lookup
mds: track which map batch_op is in-use in MDRequest

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #34785 into master

* refs/pull/34785/head:
ceph-fuse: show fuse helper options for libfuse >= 3.0
ceph-fuse: add splice read/write support to reduce the memory copy
ceph-fuse: add connection args parsing support for libfuse > 3.0
ceph-fuse: switch to fuse_reply_iov to reduce the memory copy

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #35327 into master

* refs/pull/35327/head:
client: fix directory inode can not call release callback

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35690 into master

* refs/pull/35690/head:
mds: fix hang issue when accessing a file under a lost parent directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35848 into master

* refs/pull/35848/head:
mds: do not submit omap_rm_keys if the dir is the basedir of merge.

Reviewed-by: Zheng Yan <zyan@redhat.com>

os/Bluefs: fix file reader/writer buffer mempool

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

bluestore: _get_used_bytes should reflect new mempool fields

with bluestore_cache_other split further, the _get_used_bytes should
take into account the new mempool splits as well.

see also: 0ce8ee4b123eacfc2bfb14f554487e928512415a
  68ee931fd4b6bf748f95b14dae31d11738fbaf1e

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

common/mempool: More detailed bluestore stats

Splits bluestore bluestore_cache_other, exposes accounting of some internals.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Introduce separate mempool for FileReaderBuffer and FileWriter

FileReaderBuffer and FileWriter uses Bufferlists which needs to be
tracked by mempool to get the proper memory usage.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>