crimson/osd: SnapMapper::record_purged_snaps() takes abstratced store and txn

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: convert SnapMapper::record_purged_snaps() to OSTransaction

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: initialize OSDriver in SnapMapper::record_purged_snaps()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common, osd: switch SnapMapper::_lookup_purged_snap to use OSDriver

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: fix debug in Orderer::clear_and_cancel()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: don't build unnecessary parts of SnapMapper for crimson

This commits skips building two call-sites of `omap_get_iterator()`
of `ObjectStore` which are bypassing the `OSDriver` abstraction
layer.

1. `SnapMapper::convert_legacy()` likely never be needed by
crimson and we can leave it unchanged.

2. `SnapMapper::Subber` isn't needed right now but will be
needed in the future.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: bring SnapTrimObjSubEvent

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: wire SnapTrimEvent with SnapMapper up

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: bring SubOpBlocker to SnapTrimEvent

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement the basics of SnapTrimEvent

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd/osd_operations: flatten indentation of InternalClientRequest

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: drop the residue of CompoundPeeringRequest

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: simplify ShardServices::snap_request_reservation()

`on_preempt` is not used by snap trimming; we can left its default.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: bring the skeleton of SnapTrimEvent

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: settle snap_reserver within ShardServices

This bit will be used by the upcoming `SnapTrimRequest`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: move the SnapTrimmer-related handlers of PeeringState from .h to .cc

This commit is solely about code shuffling.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #49836 from rkachach/fix_issue_58548

cephadm: using ip instead of short hostname for prometheus urls

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #50083 from adk3798/pids-limit

cephadm: set pids-limit unlimited for all ceph daemons

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge PR #50268 into main

* refs/pull/50268/head:
mds: memset the head by default to fix the readable.sh test failure

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46601 from rkachach/fix_issue_55814

mgr/cephadm: adding support for a secure ceph monitoring stack

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #50006 from idryomov/wip-58647

mon/MgrMap: dump last_failure_osd_epoch and active_clients at top level

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #50285 from ceph/epuertat-patch-1

.github: update add-to-project action

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #50299 from rhcs-dashboard/fix-broken-styles-crud

mgr/dashboard: minor fixes to crud form UI

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #50085 from myoungwon/wip-fix-initialization-vstart-rbm

crimson/os/seastore: RBM fixes during vstart startup time

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #50197 into main

* refs/pull/50197/head:
cephfs-top: navigate to home screen while selecting sort field, if no filesystems.
cephfs-top: display_menu() isn't triggered when all fs' are removed

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard: minor fixes to crud form UI

Fixes: https://tracker.ceph.com/issues/58864
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #50295 from zdover23/wip-doc-2023-02-28-radosgw-multisite-zonegroup-correction-and-formatting

doc/radosgw: s/zone group/zonegroup/g et alia

Merge pull request #50273 from cyx1231st/wip-seastore-fix-submitter-roll-segment

crimson/os/seastore/journal: allow pending i/o in a full record_submitter when rolling the segment 2

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

doc/radosgw: s/zone group/zonegroup/g et alia

s/zone group/zonegroup/ where simple greps failed to find instances of
"zone group" that were spread across two lines; break a paragraph into
two paragraphs so that each paragraph has a thematic idea of its own.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50276 from zdover23/wip-doc-2023-02-27-radosgw-multisite-caption-zone-data-replication

doc/rgw: caption a diagram

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48010 from ljflores/wip-lflores-primary-balancer

osd, tools, test/osd: implement read balancer

Merge pull request #49482 from amathuria/wip-amathuria-mclock-high-prio-queue

mClock: Add ability to handle high priority operations

mon/MgrMap: dump last_failure_osd_epoch and active_clients at top level

Currently last_failure_osd_epoch and active_clients are dumped in the
always_on_modules dictionary in "ceph mgr dump" output.  This goes back
to when these fields were added in commits f2986a4400bb ("mon/MgrMonitor:
blacklist previous instance") and df507cde8d71 ("mgr: forward RADOS
client instances for potential blacklist") but is wrong as these fields
have nothing to do with always-on modules.

Fixes: https://tracker.ceph.com/issues/58647
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #48706 from Matan-B/wip-matanb-max-oldest-map

OSD: Fix check_past_interval_bounds()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49990 from rzarzynski/wip-osd-mon-public_bind_in_bindv

mon, osd: rework the public_bind_addr support. Bring it to OSD

Merge pull request #50047 from rhcs-dashboard/post-css-error-teuth

mgr/dashboard: fix conflicting peer dependency: postcss@8.4.21

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #49507 from Matan-B/wip-matanb-check_pg_num-edit

mon/OSDMontior: Simplify check_pg_num()

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #48192 from amathuria/amathuria-edit-pg-deletion-cost

osd: Change default value of osd_pg_delete_cost

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #48657 from alimaredia/wip-rgw-labeled-perf-counters-cache

Labeled Perf Counters

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rgw: caption a diagram

Add an explanation of a diagram that shows how object data is replicated
between zones within a zonegroup.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

.github: update add-to-project action

Signed-off-by: Ernesto Puerta <37327689+epuertat@users.noreply.github.com>

Merge pull request #50274 from zdover23/wip-doc-2023-02-27-radosgw-multisite-remove-tertiary

doc/rgw: remove "tertiary", link to procedure

Merge pull request #50230 from idryomov/wip-58833

librbd: call apply_changes() after setting librados_thread_count

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #50232 from idryomov/wip-qa-workunits-windows-ownership

CODEOWNERS: assign qa/workunits/windows to RBD

Reviewed-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

mds: memset the head by default to fix the readable.sh test failure

Introduced-by: cbd7e304020(ceph_fs.h: add 32 bits extended num_retry and num_fwd support)
Fixes: https://tracker.ceph.com/issues/58853
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #50264 from zdover23/wip-doc-2023-02-27-radosgw-multisite-infernalis-changes

doc/radosgw: multisite - edit "functional changes"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

crimson/os/seastore/journal: move segment rolling asserts to ceph_asserts

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/journal: allow pending i/o in a full record_submitter when rolling the segment

Fixes: https://tracker.ceph.com/issues/58824
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/rgw: remove "tertiary", link to procedure

Remove the term "tertiary zone" and replace it with "second secondary
zone" (because there is no such thing as a tertiary zone). Link to the
procedure for creating a secondary zone in a place where such a link is
helpful to the reader.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/radosgw: multisite - edit "functional changes"

Improve the wording of the section "Functional Changes from Infernalis"
in doc/radosgw/multisite.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #50178 into main

* refs/pull/50178/head:
client: add comment about how quota affect space usage report

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #44240 into main

* refs/pull/44240/head:
qa/cephfs: upgrade xfstests_dev.py for FUSE mounts

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #50238 from rhcs-dashboard/fix-fix-58827_2-main

mgr/dashboard: fix constraints.txt again

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #50236 from ronen-fr/wip-rf-total-dump

test/osd-scrub-dump.sh: fix scrub chunk size

Reviewed-by: Samuel Just <sjust@redhat.com>-

Merge pull request #50267 from zdover23/wip-doc-2023-02-27-glossary-zone

doc/glossary: Add "zone" to glossary.rst

doc/glossary: Add "zone" to glossary.rst

Add "Zone" to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

CODEOWNERS: assign qa/workunits/windows to RBD

Assume ownership of qa/workunits/windows.  Despite the generic name,
currently it has just rbd-wnbd tests.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49299 from weirdwiz/status-update

rbd-mirror: add information about the last snapshot sync to image status

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

test/osd-scrub-dump.sh: fix scrub chunk size

The test performs shallow scrubs, intentionally using small chunk
sizes to allow dump commands time to check specific details.
Following commit ffda64119fc6165cebffd0b26619ad47b13daf51
(PR#44749), shallow scrubs chunks are controlled by a separate
configuration parameter. This PR fixes the test to use the
correct parameter.

An additional minor change is an adjustment to the test loop sleep time:
it is now reduced to guarantee that a dump followed by a counter
increase will be performed in more-or-less the scrubs frequency.

Fixes: https://tracker.ceph.com/issues/58797
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #50260 from zdover23/wip-doc-2023-02-26-glossary-client

doc/glossary: add "client" to glossary

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: add "client" to glossary

Add a definition of "client" to doc/glossary.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50257 from zdover23/wip-doc-2023-02-25-glossary-application

doc/glossary: add "application" to the glossary

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: add "application" to the glossary

Add an admittedly-stub definition of "application" to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

qa/cephfs: upgrade xfstests_dev.py for FUSE mounts

Add support for testing CephFS mounted with FUSE using xfstests-dev.

Fixes: https://tracker.ceph.com/issues/55354
Signed-off-by: Rishabh Dave <ridave@redhat.com>

test/crimson/seastore/rbm: use normal initialization path instead of open and set_block_size for test

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: add do_mount() and use it in derived classes

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: only set block_size at mkfs time

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: move RBMDevice related functions to rbm_device.cc

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: drop set_device_id() in RBMDevice

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: remove read_rbm_header in BlockRBManager::open(), and use mount to load superblock

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: reset stream_index_to_open in NVMeBlockDevice at close()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: use do_with() to prevent from freeing allocated memory

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: add mount() to NVMeBlockDevice

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: In mkfs, add open() before write_rbm_header()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: generalize shared logic and set cbjournal_size in mkfs()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: do not set block_size in open()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: fix stat_deivce() to return correct device size

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: create random_block_device using make_device at startup time

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #50142 from rishabh-d-dave/fs-xfstests_dev-distro

qa/cephfs: add 'rhel' to family of RH OS in xfstest_dev.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #50250 from tnyeanderson/fix-service-spec-link

doc/cephadm/host-management: add service spec link

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50191 from xxhdx1985126/wip-weak-backref-retrieval-trans-issue-fix

crimson/os/seastore: make weak transactions conflict with others on ROOT

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/cephadm/host-management: add service spec link

The old "(below)" text is not accurate, the service spec definition is
not in the same file at this point in time. This commit adds a ref link
to the actual service specification section.

Signed-off-by: thomas <me@thomasanderson.cloud>

Merge pull request #48516 from athanatos/sjust/wip-57801

crimson,mon: add guards to avoid accidental crimson deployment and to avoid usage of unsupported features with crimson

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #50246 from zdover23/wip-doc-2023-02-24-glossary-bucket-2

doc/glossary: add AWS links to bucket info

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: add AWS/OpenStack bucket info

Add links to AWS's documentation of buckets, in accordance with Casey
Bodley's suggestions here:
https://github.com/ceph/ceph/pull/50221#discussion_r1115900879

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50202 from rzarzynski/wip-librbd-drop-cxx11-from-example

examples/librbd: drop -std=c++11 from CXXFLAGS

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

common: Add labeled perf counters

Add the ability to dump labeled perf counters
for a daemon. Labeled perf counters are stored
in a CephContext's PerfCountersCollection.

Labeled and unlabeled perf counters are dumped
to the admin socket via `counters dump` command.

The schema for labeled and unlabeled perf
counters are dumped to the admin socket via
`counters schema` command.

This commit includes docs and additional unit tests

Signed-off-by: Ali Maredia <amaredia@redhat.com>

common: add abstraction for label-aware perf counter keys

a flat representation of a set of prometheus labels, returned as a
std::string. this string can either be used for sorting an ordered
container of perf counters, or for hashing an unordered container

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/cephfs: add 'rhel' to family of RH OS in xfstest_dev.py

The string returned by get_system_type() for RHEL 8.4 recently wasn't
one of the strings that xfstests_dev.py expects. The string returned
this time was instead "rhel". Add this string to the list of strings
that xfstests_dev.py expects for Red Hat OS family so that this failure
won't occur again in future. Also log the detected distro and version so
that it is easier to debug when this issue occurs next time.

Fixes: https://tracker.ceph.com/issues/58726
Signed-off-by: Rishabh Dave <ridave@redhat.com>

mgr/dashboard: fix constraints.txt again

The previous attempt to fix this issue
(https://github.com/ceph/ceph/pull/50207) was not successful, since the
PIP syntax for compatible versions (~=) behaves differently compared to
other package management tools (e.g.: npm).

In this case, package~=x.y.z equasl to package>=x.y.z;package==x.y.*,
instead of the desired package>=x.y.z;package==x.*.

Fixes: tracker.ceph.com/issues/58827
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #50098 from soumyakoduri/wip-skoduri-cloud-trans-azure

rgw/cloud: Add custom headers for objects transitioned to cloud

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: call apply_changes() after setting librados_thread_count

Otherwise the setting doesn't take effect.  While at it, replace
home-grown stringify() with standard to_string().

Fixes: https://tracker.ceph.com/issues/58833
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #46574 from rishabh-d-dave/xfstests-dev-improvements

qa/cephfs: improvements for xfstests_dev.py

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49528 from NitzanMordhai/wip-nitzan-filestore-removal

Reef: filestore removal

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #50176 from xxhdx1985126/wip-seastore-throttle

crimson/os/seastore: throttle concurrent transactions in seastore

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #50226 from anthonyeleven/anthonyeleven/tweak-cephadm-troubleshooting

doc: preen cephadm/troubleshooting.rst and radosgw/placement.rst

doc: preen cephadm/troubleshooting.rst and radosgw/placement.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #50221 from zdover23/wip-doc-2023-02-23-glossary-bucket

doc/glossary: add "Bucket"

Merge pull request #50220 from tnyeanderson/fix-word-repeat

doc: remove word repeat in docs

doc/glossary: add "Bucket"

Add "Bucket" to the glossary. The definition added here is just a first
attempt, but a man's reach must exceed his grasp, or what's a heaven
for?

Signed-off-by: Zac Dover <zac.dover@proton.me>