fixup: don't store ref-to-on-stack-temporary in RecoveryBackend::coll

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

fixup: ECRcoveryBackend don't goes to disk as well

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

fixup: don't use obc.attr_cache; move back to disk

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

fixup: create in get_hash_info()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: abstract EC recovery's dispatch_recovery_messages() from IO

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: abstract EC read pipeline's check_recovery_sources() from IO

For the sake of crimson.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: move templated ReadPipeline::check_recovery_sources() to .h

Behind shluffing, this commit includes also adding `std` namespaces
to dissecting the `schedule_recovery_work()` to keep it in .cc.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: make ECCommon compatible with libfmt

For the crimson's sake.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: dissect the recovery-related parts out from ECBackend

For the sake of sharing with crimson.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: shuffle UnstableHashInfoRegistry to ECCommon

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: dissect get_hash_info() with the registry into a dedicated class

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: ECBackend::get_hash_info() takes external object size

This allows to reduce the interactions with `ObjectStore`
by letting callers `get_hash_info()` to provide the object
size from external source, from a cache in partiular.

The same idea will be reused in crimson but with an extra
benefit: no need for green threads within `submit_transaction()`
of `ECBackend`.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit 98347a6013c9244c8768b1e570d36d0e17cd45d3)

osd: attrs are obligatory param for ECBackend::get_hash_info()

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit 149c61c180fadeae7fe5a1574aa8eba01bc0550b)

osd: ECBackend::get_hash_info() makes use of the obc::attr_cache

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit cf07db1a09044c689f5b595d5088a61e3d098e15)

mon/OSDMonitor: let crimson handle ECPools

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: improve comments in ECBackend::objects_read_async()

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: refactor ECTransaction with C++17 structured bindings

For better understandability.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: finish shuffling ECCommon::ReadPipeline to ECCommon.cc

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: make {PG,EC}Transaction.h consumable for crimson

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: consolidate RMWPipeline code

This commit just moved the code to have all methods of
`RMWPipeline` groupped in the file.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: shuffle the ECCommon

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: decouple classical and dummy ops for RMWPipeline::try_finish_rmw()

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: make ECBackend::kick_reads() private

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: settle RMWPipeline and RMWPipeline within ECBackend

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: dissect ECCommon and ECListener into separated header

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd, fixup: reindent ECListener

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: move ReadPipeline from ECBackend to shareable ECCommon

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: decouple RMWPipeline from ECBackend

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: ReadPipeline doesn't depend on classical ECBackend at all

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: dissect crimson-shareable parts of ECBackend into ECCommon

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: lay down the ECListener for ECBackend::RMWPipeline

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: lay down the ECListener for ECBackend::ReadPipeline

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: drop unneeded get_hash_info() from EC's RMWPipeline

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: drop the lambda wrapping from objects_read_and_reconstruct()

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: ReadPipeline doesn't depend on ECBackend anymore

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: rework the callback infrastructure around read_pipeline

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: refactor accessors of RecoveryBackend::temp_contents

1. Move some of them to .cc and
2. switch their implementations to use lower-layer methods
   instead of touching `temp_contents` directly.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

fix a crash due to RecoveryMessages life-time mgmt

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: decouple reads and recovery in ECBackend::check_recovery_sources()

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: dissect the EC read pipeline from ECBackend into dedicated class

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: drop unused param name in ECBackend::FinishReadOp::finish()

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: decouple client and recovery reads in ECBackend, part 2

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: decrease the coupling between client and recovery reads in ECBackend

`complete_read_op()` being aware about `RecoveryMessages` was too much.

TODO:
  * rename or rework `RecoveryMessages`. Since now it's also
    a callack for `ReadOp::on_complete`. I don't like that.
  * drop the `pair<RecoveryMessages*, read_request_t&>`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: s/pg/backend/ in ECBackend::OnRecoveryReadComplete

`pg` for `ECBackend*` is simply misleading.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: s/read/recovery_read/ in ECBackend::RecoveryMessages

`read()` is simply too common for grep. Also, this commit
will ensure there is no unnoticed new call site which
could be problematic in the spot of the upcoming recovery
read rework.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #54829 from awojno-bloomberg/wip-fix-role-delete-replication

rgw/iam: fix role deletion replication

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53156 from qiuxinyidian/rgw-dev

rgw: modify string match_wildcards with fnmatch

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55009 from ronen-fr/wip-rf-actpr-requeue

osd/scrub: remove scrub_clear_state()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54870 from Svelar/crimson_g_conf

crimson: add set_val and rm_val to global conf

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52315 from ronen-fr/wip-rf-rgw-tasserts

test/rgw: annotating variables with maybe_unused

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #55088 from cbodley/wip-make-dist-wget

Merge pull request #54587 from cbodley/wip-45736

rgw: add headers to guide cache update in 304 response

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #54715 from rkachach/fix_issue_63678

mgr/rook: fixing inconsistencies on the dashboard 'services' view

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54676 from rkachach/fix_list_devices_issue

mgr/rook: fixing devices listing on rook

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

make-dist: don't use --continue option for wget

the boost jfrog mirror is broken and returns an HTML error page instead
of the archive. the file size of this page is 11534 bytes

when download_from() retries the download from download.ceph.com, the -c
option tells it to resume the download of the existing file. the
resulting boost_1_82_0.tar.bz2 ends up with the correct total file size
of 121325129 bytes, but the first 11534 bytes still correspond to the
HTML from jfrog. that causes the sha256sum mismatch

remove the -c option so that wget fetches the archive in its entirety

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55081 from petrutlucian94/boost_mirror

win32_deps_build.sh: change Boost URL

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55075 from zdover23/wip-doc-2024-01-07-radosgw-admin-quota

doc/radosgw: edit admin.rst - quota

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit admin.rst - quota

Edit the "Quota Management" section (all of it) in
doc/radosgw/admin.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

win32_deps_build.sh: change Boost URL

The Boost download URL is broken again [1], we'll switch to a
working mirror.

[1] https://github.com/boostorg/boost/issues/842

Fixes: https://tracker.ceph.com/issues/63952
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #54525 from myoungwon/wip-delta-overwrite-inplace-replay

crimson/os/seastore:  introduce inplace rewrite for RBM

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #54613 from Matan-B/wip-matanb-crimson-build-inc-maps

crimson/osd: Support incremental maps

Reviewed-by: Samuel Just <sjust@redhat.com>

osd/scrub: remove scrub_clear_state()

Its functionality is now handled by the FSM when existing
'Session' or 'ActiveReplicaOp' states.
For a PG that is not active - no need to do anything.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54996 from ronen-fr/wip-rf-active-primary

osd/scrub: add a "clean primary" base state

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55069 from ceph/wip-rgw-adminops-user-info-doc-update

doc: update rgw admin api req params for get user info

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc: update rgw admin api req params for get user info

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #54993 from ljflores/wip-balancer-status-detail-documentation

Merge pull request #54890 from kamoltat/wip-ksirivad-fix-63389

src/mon/OSDMonitor.cc: more descriptive loggings for crc mismatch

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #54708 from amathuria/wip-amathuria-bz-2134786-fix-norecover

mgr/pg_autoscaler: add check for norecover flag

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

Merge pull request #52380 from kamoltat/wip-ksirivad-fix-tracker-59564

mon/ConnectionTracker.cc: disregard connection scores from mon_rank = -1

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #48175 from amathuria/wip-add-test-case-bz-2011756

DaemonServer.cc: fix config show command for RGW daemons

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: avoid "over clearing" queued_or_active flag

If two StartScrub messages are received in quick succession, the earlier
one might clear the queued_or_active flag as it fails for being from an old
interval.
When that happens - a 3'rd scrub request will actually be allowed to go
through, while the scrubber is still handling the second one.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: manage queue registration lifetime in the FSM

As the state of 'being registered in the OSDs scrub queue'
corresponds to the PrimaryActive FSM state.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55065 from zdover23/wip-doc-2024-01-05-radosgw-admin-add-remove-admin-caps

doc/radosrgw: edit admin.rst

doc/radosrgw: edit admin.rst

Edit "Add / Remove Admin Capabilities" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/cache: return false explicitly if delta is not replayed

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: check correct crc for inplace-rewritten extents  after replay is done

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

test/crimson/seastore: fix overflow issue and make len > 0 when set_contents is called

set_contents causes the overflow at times because
alloc_extent is allowed to use uint32_t.

Specifically, in random_writes case, PADDING_SIZE is 256<<10,
whereas set_contents's len is uint16_t.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/epm: detect transaction confict during rbm write using OOL

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: introduce modified_region in DATA_BLOCK to keep track of modified region

It has a limitation to keep track of the modified region using the existing
deltas because we can not get the correct region in two cases: 1) a case where replay
is done and 2) duplicate_for_write. This commit introduces modified region
to solve the problem.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: write only modified region when rewriting the extent in an inplace manner

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

test/crimson/seastore: add a random overwrite test to test_objec_date_handler

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/cache: reset version to 1 if inplace write occurs before in prepare_record

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #54425 from linuxbox2/wip-lcnotify-cleanup

rgwlc:  additional lc notify cleanups

Merge pull request #54889 from phlogistonjohn/jjm-cephadm-build-with-wheels

cephadm: build with wheels, enabling pyyaml dependency

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #54722 from phlogistonjohn/jjm-cephadm-sidecars

cephadm: add support for sidecar containers

Reviewed-by: Adam King <adking@redhat.com>

rgwlc:  even current object versions have a unique instance

Fixes: https://tracker.ceph.com/issues/63546
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc:  additional lc notify cleanups

Follow the same formula to build up obj_state and version_id
in all call sites.

Resolves: rhbz#2163667

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #50055 from cbodley/wip-rgw-polymorphic-executor

rgw/beast: switch back to polymorphic executor

Reviewed-by: Mark Kogan <mkogan@ibm.com>

Merge pull request #54903 from cbodley/wip-63829

radosgw-admin: 'zone set' won't overwrite existing default-placement

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #54505 from adamemerson/wip-lazyfifo-no-mutex-over-coro

rgw/multisite: Fix deadlock by not holding mutex over coroutine

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49594 from Matan-B/wip-matanb-crimson-calc-subsets

crimson: Support Partial Object Recovery

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55022 from yuvalif/wip-yuval-63901

rgw/kafka: make the connection idle timeout configurable

reviwed-by: cbodley

Merge pull request #54967 from yuvalif/wip-yuval-63859

rgw/notifications/lc: handle publish_commit() failures as warnings

reviewed-by: dang, thotz

crimson/osd: Keep track of modified_ranges

* `modifies_ranges` interval_set is added to osd_op_params_t
* keep track of modified_ranges while executing relevant ops
* Add `osd_op_params` parameter to `PGBackend::remove()`.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/replicated_recovery_backend: handle_pull_response() - recalc_subsets()

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

mson/osd/replicated_recovery_backend: Introduce prep_push_to_repli…
…ca()

Intelligently push an object to a replica.  make use of existing
clones/heads and dup data ranges where possible.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/replicated_recovery_backend: seperate prep_push_to_replica and prep_push

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

rename prep_push to prep_push_to_replica

dismiss head_ctx

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/replicated_recovery_backend: prepare_pull to support calc_clone_subsets

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/object_metadata_helper: Introduce calc_*_subsets

Signed-off-by: Matan Breizman <mbreizma@redhat.com>