osd: do not assert() in the case of no obc

Upon rollback, we should handle ENOENT case,
so what we should do here is to return NOOP.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: remove unnecessary ref handling in _delete_oid

Let's consider the following case when handling a delete op.
1. Delete --> whiteouted
2. Make clone

In this case, current code clears chunk_map and calls dec_all_manifest_refcount()
in _delete_oid() even if the clone still has the references.

To fix this, This commit remove unnecessary ref handling in _delete_oid, and
makes finish_ctx() to handle ref handling, which can aware of whether the
clone is created or not.

Also, remove oi.size == 0 condition in finish_ctx() to handle ref. counting
upon a delete op with whitedouted clone.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

qa: fix typo to call rollback op

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: clear manifest info to fix reference mismatch

If the head is deleted in rollback(), manifest info also needs to be clear.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: remove unnecessary log

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: reset flushed to false when updating object

The object updated by the Ops should be set unflushed.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: fix not updating the object state in the error case

This commit prevent updating wrong state, which happens when
TierFlush receives error values.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: generate useful log regarding ENOENT

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: add ManifestRollbackRefcount test

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: fix reference mismatch on rollback

Upon rollback, we should update new chunk_map in head.
To do so, all entries in the chunk_map need to be updated
via inc_refcount_by_set().

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: inc_refcount_by_set() refactor

current inc_refcount_by_set only supports a case
where a single entry is updated via SET_CHUNK.
This commit will make existing inc_refcount_by_set to handle
multiple entries.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: _do_rollback_to() refactor

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: check flush hasn't been called regarding EBUSY

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: add manifest info when duplicating head upon manifest object

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: move check condition into maybe_handle_manifest_detail()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: add concrete condition regarding ENOENT in TierFlush/TierEvict

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: add comments to explain why EINVAL can be returned on TierEvict()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: use pair instead of reference variable

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: clean-ups

It is more easier to undertand if PromoteCallback is only used in promote_object,
and PromoteManifestCallback is used in TIER_PROMOTE

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: update user_version if redirect object on promote()

If the object is redirect manifest object, need to change
user_version to the target object on promote()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: fix not to decrement num_objects_dirty in finish_set_dedup

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: clear chunk_map if the object is whiteouted

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: add dec_all_refcount when the head is deleting in _rollback_to()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: fix when calling rollback from manifest object to normal object

do not invoke maybe_handle_manifest_detail if the object is not manifest

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: add rollback unit test on manifest object

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: add handling ENOENT when object is deleted

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: fix to call correct ObjectContext

During trim_object(), oid from oi differs with oid from ctx
in dec_all_refcount(). To avoid this, use oi's oid when retreiving
objectcontext.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

qa: add missing argument parser

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: add a function in ceph_test_rados to check refcount is correct

Comparing refcounts between manifest object and chunk object,
ceph_test_rados will check refcount is correct at the end of test

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: add ManifestEvictPromote unit test

To verify that the manifest object works correctly when
read is called after tier_evict is done, ManifestEvictPromote is added

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: add snap to send TierEvictOp to snapshotted manifest object

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: prevent being a normal object during ManifestFlushSnap (unit test)

Now, modifying object when creating snapshot triggers to remove
the corresponding chunk in chunk_map.
This results in tranforming a manifest object to a normal object
if there are no chunks in chunk_map.
To avoid this, another set_chunk() is added.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

qa/workuints: add ceph_dedup_tool test for scrub refcount

Based on CDC flush, a test, which calls get/put reference,
is added.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: fix wrong calculation to continue copy chunks

If the last offset, which indicate the last copy completed, is less then
the offset of the last entry,
copy_some_manifest should be called to copy remaining chunks.
This commit fixes miscalculation to conitnue copy chunks.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: fix wrong promotion after writing snapshot

Let's assume the following scenario.
snap_create() -> write() -> tier_promote() -> read().
write() trigger to create a snapshot with current chunk_map, but
it does not update chunk_map even if the write() overwrites the region.
So, tier_promote() will overwrite the region by writing old chunk to there.

To avoid this, update_chunk_map_by_dirty is added by splitting
dec_refcount_by_dirty

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: allow EBUSY when TierFlushOp

If the snapshot is not oldest, EBUSY can be returned

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: fix wrong link between manifest and chunk object

manifest object will generate fingerprint oid, otherwise
ceph_test_rados will use prefix+oid

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: make TierFlushOp to set snapshot

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: make SetChunkOp to set snapshot

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

qa: add dedup-io-snaps test

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: fix wrong iteration

last_offset means the last entry to be copied,
so it should be last_offset <= iter->first

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

qa: add dedup-io-mixed test

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: allow EOPNOTSUPP in SetChunkOp

Because tier-flush creates new chunks in the manifest object using CDC,
SetChunkOp from ceph_test_rados can issue overlapped offset

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: use fingerprint OID when calling SetChunkOp

To align with CDC, which uses fingerprint OID,
this commit makes SetChunkOp to use fingerprint OID

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: allow ENOENT when calling SetChunkOp

The oid targeting the low tier can be deleted during
mixed test, which sends tier_evict, tier_promote, and tier flush
because the reference of the target oid can be decreased.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: add dedup options (chunking algorithm, chunking size) to run dedup test

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: add CDC chunking when dedup is enabled

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: add TierEvictOp to ceph_test_rados

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: add SetChunkOp as a TestOp

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: use uint32_t referring to chunk_info_t

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: cleanups in SetChunkOp

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

src/test: remove unused values in SetChunkOp

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #40445 from liewegas/cephadm-upgrade-progress-version

mgr/cephadm: make upgrade progress bar mention target version, not repo digest

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #40447 from Daniel-Pivonka/cephadmdocapplyrgwflags

mgr/orchestrator: fix rgw realm and zone flags

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #40379 from adk3798/no-overwrite

mgr/cephadm: --no-overwite flag for apply command

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39969 from batrick/i49684

qa: wait for daemons to come up via cephadm

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #40396 from maydaycc/wip-49968

mgr: attr_list reassigned before use in diskprediction_local module

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40421 from badone/wip-fix-typo-get_device_id

common/blkdev: Remove trailing single quote

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40450 from smithfarm/wip-50007-alt

rpm: drop extraneous explicit sqlite-libs runtime dependency

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rpm: drop extraneous explicit sqlite-libs runtime dependency

Commit 75980798f19b8c11efd75ba4aae3e491d4c99f98 introduced a new package,
libcephsqlite, with a hard RPM dependency on a package "sqlite-libs" which
does not exist in openSUSE.

Since the runtime library dependencies of libcephsqlite are handled by RPM
transparently, this line is not needed.

Fixes: https://tracker.ceph.com/issues/50007
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #40442 from Thingee/update-foundation-page

doc: Update Foundation page

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Updating Foundation membership page

Signed-off-by: Mike Perez <thingee@gmail.com>

Merge pull request #40408 from varshar16/wip-doc-update-ninja

doc/install: update about ninja

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40345 from rzarzynski/wip-crimson-doc-waitstates-follow-up

doc/crimson: improve wording about the pipeline / wait states.

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38732 into master

* refs/pull/38732/head:
client: fire the finish_cap_snap() after buffer being flushed
client: simplify the iterating code
client: remove unused _flushed_cap_snap
client: clean up the code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40326 into master

* refs/pull/40326/head:
client: rebuild bl to avoid too many vector(> IOV_MAX)

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #40394 into master

* refs/pull/40394/head:
osd: handle MOSDPGInfo
osd/PeeringState: init last_require_osd_release in ctor

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/cephadm: fix rgw realm and zone flags

Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>

Merge pull request #40415 from sseshasa/wip-qa-wait-for-clean-check

qa/tasks: Add wait_for_clean() check prior to initiating scrubbing.

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: make upgrade progress bar mention target version, not repo digest

The repo digest is super long and meaningless for a human user.  Instead,
use the target version (as soon as we know what it is--until then, use
the target image name).

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #40438 from idryomov/wip-crimson-reopen-session

crimson/monc: discard active/pending connections when reopening

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/monc: discard active/pending connections when reopening

Otherwise pending_conns vector just keeps growing with redundant
connections all trying to reach the same set of monitors.  When one
of the attempts finally succeeds, _finish_auth() will pick the first
connection with a matching entity_addr_t, designate it as active and
close all others.  The match is very likely to be wrong and hence the
actual authenticated connection gets closed, leaving the OSD with
a bogus active_con and no monitor session.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #40404 into master

* refs/pull/40404/head:
auth: require CEPHX_V2 by default

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40321 into master

* refs/pull/40321/head:
cephadm: fix parsing of keepalived version (drop leading 'v')
cephadm: keepalived needs --cap-add=NET_RAW
cephadm: fix --cap-add=NET_ADMIN
cephadm: fix quoting for keepalived env var
mgr/cephadm: ha-rgw: use correct port

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #40015 into master

* refs/pull/40015/head:
cephadm: validate fsid during cephadm shell command

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge PR #40341 into master

* refs/pull/40341/head:
os/bluestore: separate omap per-pool vs per-pg alerts

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #40429 from varshar16/wip-fix-fs-top-error

src/tools/cephfs-top: fix flake8 line too long error

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #40401 from lnussel/master

ceph.spec: prepare openSUSE usrmerge (boo#1029961)

src/tools/cephfs-top: fix flake8 line too long error

Fixes: https://tracker.ceph.com/issues/50005
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #40170 from Rethan/fix-sync-pause-max-lat

os/FileStore: fix sync_pause_max_lat which should not be inced

Reviewed-by: Kefu Chai <kchai@redhat.com>

README.md: remove extra 'the' and format para related to ninja

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/install: replace do_cmake note with link to ceph README

Ceph README.md doc already has the do_cmake note and other build related info.

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/install: Update about ninja

'ninja' replaced 'make' in this PR[1]. This patch updates the doc about it.

[1] https://github.com/ceph/ceph/pull/39826

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #40334 from dvanders/pick

common/ipaddr: skip loopback interfaces named 'lo' and test it

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40292 from tchaikov/wip-39191

debian,cmake,cephsqlite: hide non-public symbols

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40171 from swegener/bluefs-assert-always-true

bluefs: fix assert always being true

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40304 from tchaikov/wip-telegraph-ann

pybind/mgr/telegraf: add type annotations

Reviewed-by: Wido den Hollander <wido@widodh.nl>

Merge pull request #40403 from vshankar/wip-cephfs-top-add-metrics

cephfs-top: include additional metrics reported by `fs perf stats`

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #40167 from singuliere/wip-49781

common/mempool: only fail tests if sharding is very bad

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common/blkdev: Remove trailing single quote

Example output:

e1 collect_metadata :  no unique device id for : fallback method has no
model nor serial'

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #40331 from athanatos/sjust/wip-seatore-gc-refactor

crimson/seastore: refactor gc into concurrent task

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #40336 from tchaikov/wip-cmake-fio

cmake: pass compile options by fio interface library

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40323 from aclamk/wip-onode-resiliant-to-split-cache

os/bluestore: Make Onode::put/get resiliant to split_cache

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

mgr/volumes: do not overwrite existant mds specs

Signed-off-by: Adam King <adking@redhat.com>

crimson/os/seastore/segment_cleaner: bump reclaim_ratio limits

GC behavior still needs a ton of work, but this adjustment should
reduce gc activity for now until we have time to refine the params
and introduce the ability to write cold extents to a segment other
than the current journal segment.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../btree_range_pin: use ceph_assert

There's a performance overhead, but seastore is quite immature and
detecting these logic errors saves a ton of debugging time.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../block.cc: add error message on alloc error to segment_write

Signed-off-by: Samuel Just <sjust@redhat.com>

mgr/cephadm: no-overwite flag for apply command

no-overwrite flag makes it so cephadm will only apply the spec
if the spec does not already exist.

Signed-off-by: Adam King <adking@redhat.com>

qa/tasks: Add wait_for_clean() check prior to initiating scrubbing.

Fixes: https://tracker.ceph.com/issues/49983
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #39776 from alimaredia/wip-rgw-log-req-latency

rgw: add latency to the request summary of an op