tasks/.../test_erasure_code_profile: assertSubset in test_create_plugin

Newly added profile options may break this test otherwise.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/cli/osdmaptool/crush.t: adjust --import-crush size output

Signed-off-by: Samuel Just <sjust@redhat.com>

test/cli/crushtool/choose-args.t: add msr related json output

Signed-off-by: Samuel Just <sjust@redhat.com>

qa/erasure-code: modify jerasure 4/2 ec test case to use msr

Signed-off-by: Samuel Just <sjust@redhat.com>

doc/rados/operations: add CRUSH MSR documentation

Signed-off-by: Samuel Just <sjust@redhat.com>

erasure-code: add support for multiple osds in a single failure domain

Adds support for crush-osds-per-failure-domain and
crush-num-failure-domains via MSR rules.

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart.sh: add --require-osd-and-client-version and --use-crush-tunables flags

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart.sh: add --osds-per-host

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crush/crush.cc: add tests specifically for MSR

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crush/crush.cc: add test variants for firstn rules

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crush/crush.cc: convert indep test cases to test MSR as well

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crush/crush.cc: s/NULL/nullptr/g

Signed-off-by: Samuel Just <sjust@redhat.com>

mon/OSDMonitor: generalize rule type check for pools

Add rule_valid_for_pool_type to CrushWrapper to generalize
rule type <-> pool type mapping to include the new MSR
types.

Signed-off-by: Samuel Just <sjust@redhat.com>

doc/dev/crush-msr.rst: add developer summary of crush msr

Signed-off-by: Samuel Just <sjust@redhat.com>

crush/mapper: add support for MSR types

Signed-off-by: Samuel Just <sjust@redhat.com>

crush: add choosemsr rule

Signed-off-by: Samuel Just <sjust@redhat.com>

crush: add msr tunables

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/OSDMap: minor formatting nit

Signed-off-by: Samuel Just <sjust@redhat.com>

crush: add msr rule types and CrushWrapper infrastructure

Signed-off-by: Samuel Just <sjust@redhat.com>

src/crush: add a rule type enum

We're going to add more rule types that don't map directly onto
pool types.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #55323 from ceph/dependabot-github_actions-gregsdennis-dependencies-action-1.3.2

.github: Bump gregsdennis/dependencies-action from 1.2.3 to 1.3.2

Merge pull request #55331 from ceph/revert-55096-sjust/for-review/wip-crush-msr

Revert "crush: add multistep retry rules"

Revert "crush: add multistep retry rules"

This PR was merged by accident before it was ready.
Let's revert for now and open a new PR.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #55096 from athanatos/sjust/for-review/wip-crush-msr

crush: add multistep retry rules

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #55303 from cbodley/wip-63130-debug

cmake/arrow: don't treat warnings as errors

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #55234 from ajarr/wip-64063

rbd-nbd: use netlink interface by default

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55121 from zhscn/fix-ambiguous-error

common: fix ambiguous error when using gcc 13

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #54896 from cyx1231st/wip-crimson-save-conn-foreign-copy

crimson/osd: drop a foreign-copy to shard-0 for every pg operation

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #52496 from adamemerson/wip-rgw-surface-neorados

rgw: Surface neorados

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

.github: Bump gregsdennis/dependencies-action from 1.2.3 to 1.3.2

Bumps [gregsdennis/dependencies-action](https://github.com/gregsdennis/dependencies-action) from 1.2.3 to 1.3.2.
- [Release notes](https://github.com/gregsdennis/dependencies-action/releases)
- [Commits](https://github.com/gregsdennis/dependencies-action/compare/80b5ffec566913b1494d5a8577ab0d60e476271d...f98d55eee1f66e7aaea4a60e71892736ae2548c7)

---
updated-dependencies:
- dependency-name: gregsdennis/dependencies-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #55308 from ljflores/wip-dependabot

Merge pull request #55307 from zdover23/wip-doc-2024-01-25-radosgw-admin-usage

doc/radosgw: edit "Usage" admin.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit "Usage" admin.rst

Edit "Usage" in doc/radosgw/admin.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55315 from cbodley/wip-moncommand-dencoder

mon: zero-initialize MonCommand::flags

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

rbd-nbd: map using netlink interface by default

Mapping rbd images to nbd devices using ioctl interface is not
robust. It was discovered that the device size or the md5 checksum
of the nbd device was incorrect immediately after mapping using
ioctl method. When using the nbd netlink interface to map RBD images
the issue was not encountered. Switch to using nbd netlink interface
for mapping.

Fixes: https://tracker.ceph.com/issues/64063
Signed-off-by: Ramana Raja <rraja@redhat.com>

mon: zero-initialize MonCommand::flags

causing failures in check-generated.sh:

2 MonCommand
/tmp/typ-qmnDZ1ahR /tmp/typ-lIgJTZiUe differ: byte 100, line 6
**** MonCommand test 1 dump_json check failed ****
   ceph-dencoder type MonCommand select_test 1 dump_json > /tmp/typ-qmnDZ1ahR
   ceph-dencoder type MonCommand select_test 1 encode decode dump_json > /tmp/typ-lIgJTZiUe
6c6
<     "flags": 94661933599904
---
>     "flags": 94342467308704

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55287 from ajarr/wip-64139

rbd-nbd: fix resize of images mapped using netlink

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55270 from afreen23/fix-cap-inconsistency-multisite

mgr/dashboard: Fix inconsistency in capitalisation of "Multi-site"

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: rosinL <NOT@FOUND>

Merge pull request #55182 from rkachach/fix_issue_64029

mgr/rook: adding some basic rook e2e testing

Merge pull request #55266 from athanatos/sjust/wip-63996

crimson: retain map references in OSDSingletonState::store_maps

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/shard_services: retain map references in OSDSingletonState::store_maps

Introduced: 3f11cd94
Fixes: https://tracker.ceph.com/issues/63996
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/shard_service.cc: convert to newer logging machinery

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd.cc: migrate logging to new style

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #55288 from athanatos/sjust/wip-64140

Revert "crimson/os/alienstore/alien_log: _flush concurrently"

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

rgw: Thread site config into RADOS driver

Make this available so multisite sync doesn't have to rely on the zone
service.

Signed-off-by: Adam Emerson <aemerson@redhat.com>

rgw: Asio frontend shares `io_context` with the rest of RGW

Pull the `io_context` and threads out of `AsioFrontend`, pass in a
reference to `io_context_pool` so it can be shut down at `AsioFrontend::join`

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #54987 from batrick/i63822

pybind/mgr/devicehealth: skip legacy objects that cannot be loaded

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge pull request #54491 from jianwei1216/fix_osd_pg_stat_report_interval_max_cmain

fix: resolve inconsistent judgment of osd_pg_stat_report_interval_max

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #53250 from YiteGu/add-perfcount-for-allocator

os/bluestore: add perfcount for bluestore/bluefs allocator

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #52530 from amathuria/wip-amat-fix-59531

osd: Add memstore to unsupported objstores for QoS

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

rgw: SAL drivers take `boost::asio::io_context`

`RadosDriver` needs it, and since SAL generally uses `optional_yield`,
other stores are going to need it to implement that sensibly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph/async: `io_context_pool` constructor/start takes init function

Add overloads of the constructor and `start` that take an init
function, for setting up the thread before dropping into
`io_context::run`.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/async: `io_context_pool` constructor takes int64_t

Just because this is what Ceph's config uses and it saves a narrowing
conversion. If we want to set a max value on the thread count, we
should do it in config.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

include/inline_memory.h: Add missing includes to fix build

Signed-off-by: Adam Emerson <aemerson@redhat.com>

.github: fix length of dependabot commit message prefix

Followup to commit 265f727481c729fbe78dce7410def7de6fe869d1.

Fixes this error mesage under GitHub Insights (https://github.com/ceph/ceph/network/updates):

```
The property '#/updates/1/commit-message/prefix' was not of a maximum string length of 15
```

Signed-off-by: Laura Flores <lflores@ibm.com>

rbd-nbd: log errors during netlink_resize() using derr

When using rbd CLI to map the images to NBD devices via netlink,
any errors that arose during image resizing in netlink_resize()
were not logged. Switching the error logging from using cerr to
derr helps log the errors from netlink_resize().

Signed-off-by: Ramana Raja <rraja@redhat.com>

rbd_nbd: fix resize of images mapped using netlink

Include device identifier or cookie in the message sent to the kernel
to resize images mapped to NBD devices using netlink. Otherwise,
netlink_resize() fails and the size of the device isn't updated.

Fixes: https://tracker.ceph.com/issues/64139
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #49462 from rzarzynski/wip-bug-53789

osdc: fix the ENOCONN normalization in Objecter::_linger_reconnect()

Merge pull request #55219 from samarahu/rgw_asio_frontend_asserts

rgw/asio: Add asserts to rgw_asio_frontend.cc

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

mgr/rook: increase minikube mem to 6GB to avoid stability issues

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #55192 from dparmar18/fix_docstrings_ceph_test_case

qa: typo fixes in ceph_test_case docstrings

Merge pull request #53320 from jzhu116-bloomberg/wip-62710

rgw/multisite: maintain endpoints connectable status and retry the requests to them when appropriate

Reviewed-by: Mark Kogan <mkogan@ibm.com>

Merge pull request #54941 from samsungceph/vstart_network_v2

vstart: Pick only CIDR-formatted routes when cephadm enabled

Reviewed-by: Adam King <adking@redhat.com>

cmake/arrow: don't treat warnings as errors

with arrow's default BUILD_WARNING_LEVEL, -Werror is added to cflags and
debug builds fail on a warning about _FORTIFY_SOURCE

Fixes: https://tracker.ceph.com/issues/63130
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53668 from mdw-at-linuxbox/wip-master-update-kmip-1

Update libkmip to pull in some portability changes.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/rook: adding some basic rook e2e testing
Fixes: https://tracker.ceph.com/issues/64029
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Revert "crimson/os/alienstore/alien_log: _flush concurrently"

While submitting the log line asyncronously is reasonable,
with this implementation the EntryVector &q parameter does
not necessarily outlive the submission continuation.

This reverts commit 511af83e2747361350b60ce0ce88e67a726d9343.

Fixes: https://tracker.ceph.com/issues/64140
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #55183 from galsalomon66/s3select_fixes_QE_bugs

rgw/s3select: bug fixes per QE recent defects

Update libkmip submodule to pull in some portability changes.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #55217 from ronen-fr/wip-rf-old-reserv

osd/scrub: check reservation replies for relevance

Reviewed-by: Samuel Just <sjust@redhat.com>-

Merge pull request #55067 from yaarith/telemetry-pool-flags

mgr/telemetry: add pool flags

Merge pull request #55240 from rosinL/wip-fix-64032

install-deps: Force remove ceph-libboost* packages

Merge pull request #55278 from Himura2la/patch-2

Docs: Specify correct fs type for mkfs on volume creation

Reviewed-by: Zac Dover <zac.dover@proton.me>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

bug fixes per QE recent defects
update for the engine_version message
s3select submodule

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

Merge pull request #55277 from yuvalif/wip-yuval-63578

rgw/lua: fix compilation issue when lua packages are disabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: specify correct fs type for mkfs

The default value is ext2, which is not supported (anymore?).

Signed-off-by: Vladislav Glagolev <vladislav.glagolev@devexpress.com>

rgw/lua: fix compilation issue when lua packages are disabled

Fixes: https://tracker.ceph.com/issues/63578#change-253102
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

mgr/dashboard: Fix inconsistency in capitalisation of "Multi-site"

fixes https://tracker.ceph.com/issues/64125

Across the dashboard, two instances are present: Multi-site and
Multi-Site.
Making it consistent all over by using Multi-site.

Signed-off-by: Afreen <afreen23.git@gmail.com>

osd/scrub: check reservation replies for relevance

Compare a token (nonce) carried in the reservation reply with the remembered
token of the reservation request.  If they don't match, the reply is
stale and should be ignored (and logged).

Fixes: https://tracker.ceph.com/issues/64052
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55107 from ronen-fr/wip-rf-rm-penaltyq

osd/scrub: remove the 'penalty queue' from the scrubber

Reviewed-by: Samuel Just <sjust@redhat.com>-

Merge pull request #55269 from zdover23/wip-doc-2024-01-23-radosgw-admin-read-write-global-rate-limit-config

doc/radosgw: edit "read/write global rate limit" admin.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit "read/write global rate limit" admin.rst

Edit "Reading/Writing Global Rate Limit Configuration" in
doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55223 from athanatos/sjust/wip-64055

crimson: clear obc_registry on interval change

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

rgw/multisite: add multisite test cases with some rgw instances down

Signed-off-by: Juan Zhu <jzhu4@dev-10-34-20-139.pw1.bcc.bloomberg.com>

Merge pull request #55070 from pdvian/wip-fix-progressevent

mon: initialize ProgressEvent::add_to_ceph_s

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #53154 from ifed01/wip-ifed-no-death-tests

test/store_test: get rid off assert_death.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #49415 from ljflores/wip-update-telemetry-upgrade

qa/workunits: update telemetry quincy workunits with `basic_pool_options_bluestore` collection

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

osd/scrub: update job's NB on failure

When a scrub job fails, update its NB to the current time plus a
fixed delay.  This prevents the job from being scheduled again
immediately.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: fix set_last_deep_scrub_stamp()

The call should update last_scrub_stamp, too, without
requiring an extra call to on_scrub_schedule_input_change()

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: introduce a 'not before' attribute for scrub jobs

The NB enables the OSD to delay the next attempt to schedule a specific
scrub job.  This is useful for jobs that have failed for whatever
reason, especially if the primary has failed to acquire the replicas.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: remove the 'penalized jobs' queue

The 'penalized jobs' queue was used to track scrub jobs that had failed
to acquire their replicas, and to prevent those jobs from being retried
too quickly.  This functionality will be replaced by a
simple 'not before' delay (see the next commits).

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson/osd/osd_operations/scrub_events: fix to use get_peering_request_pipeline()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: drop a foreign-copy to shard-0 for every pg operation

By using ConnectionRef before pg submission, and after that, change to
use ConnectionXcoreRef.

The intent is to drop the foreign copy of the connection to shard 0 at
pg submission time. This should remove two pairs of crosscore
communications in shard 0 for each I/O, one for connection-ref foreign
copy, another for connection-ref destruction.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #55262 from Matan-B/wip-matanb-crimson-bluestore-submit

crimson/os/alienstore/alien_log: _flush concurrently

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operations/client_request: don't use and move pgref at the same time

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/osd_operations/client_request: cleanup shard_services

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/alienstore/alien_log: _flush concurrently

In continuation to c15e56e386251403a876454f6a4aa186284565e1

Authored-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #55190 from zdover23/wip-doc-2024-01-16-radosgw-admin-enable-disable-bucket-rate-limit

doc/radosgw: edit "Enable/Disable Bucket Rate Limit"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #53288 from rzarzynski/wip-crimson-dont-shadow-store-in-ecbackend

crimson: drop store from ECBackend to not shadow PGBackend::store

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54813 from amathuria/wip-crimson-amat-fix-config-set-cmd

src/crimson: Add support for the OSD to receive config changes

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>