librbd: Reduce use of atomics in librbd throttling

Signed-off-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>

Merge pull request #58722 from cbodley/wip-common-async-shared-mutex-impl

common/async: fix duplicate definition errors from SharedMutexImpl

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #58838 from idryomov/wip-rbd-lock-incompat-note

doc/rbd/rbd-exclusive-locks: mention incompatibility with advisory locks

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/rbd/rbd-exclusive-locks: mention incompatibility with advisory locks

This is due to both using the same name for the lock at the RADOS level
("rbd_lock").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58348 from cbodley/wip-async-co-algorithms

common/async: add primitives for structured concurrency with c++20 coroutines

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #58686 from clwluvw/rgw-noprinc-allow

rgw: donot allow NotPrincipal with Allow Effect

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #58406 from mohit84/issue_42884

test: Create ParallelPGMapper object before start threadpool

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57223 from myoungwon/wip-rename-global-values-ceph-dedup-daemon

tool: renaming and refactoring global variables in ceph-dedup-daemon

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #58781 from idryomov/wip-67097

qa/suites/krbd: stress test for recovering from watch errors for -o exclusive

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58766 from athanatos/sjust/wip-66294-collection-race

crimson: access coll_map under alien tp with a lock

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58464 from athanatos/sjust/wip-66316-async-reserver

crimson: peering event processing fixes,  wait for async operations started during peering events

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57776 from rzarzynski/wip-bug-61948

mon, osd, *: expose upmap-primary in OSDMap::get_features()

Merge pull request #56932 from ceph/wip-yuriw-release-18.2.3-main

doc: reef 18.2.4 release note

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

doc/releases: add release note about thread creation incompatibility

Signed-off-by: Laura Flores <lflores@ibm.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

common/async: add parallel_for_each() algorithm

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/async: add co_spawn_group template for fork-join parallelism

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/async: implement max_concurrent_for_each() for awaitable

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/async: add co_throttle for bounded concurrency with c++20 coroutines

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58535 from anoopcs9/mgr-nfs-fix-export-create-conf-obj

mgr/nfs: Do not ignore clusters from rados pool conf objects

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Ponnuvel Palaniyappan <pponnuvel@gmail.com>

Merge pull request #58533 from rhcs-dashboard/shared-folder-pythonV

cephadm: update shared folder python packages dir

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58153 from anoopcs9/modify-smb-dot-conf

mgr/smb: Make appropriate changes to global and share specific options

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58670 from ivancich/wip-bolster-reshard-testing

test/rgw: address potential race condition in reshard testing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #58056 into main

* refs/pull/58056/head:
qa: suppress __trans_list_add valgrind warning
qa: generate valgrind suppressions

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58041 from vmackie/class-param-bug

mon: fix inconsistencies in class param

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge PR #58768 into main

* refs/pull/58768/head:
script/backport-create-issue: retry without if assignee is invalid

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #58152 from adam-lj/aiocompletion-perf-changes

librbd: Modify locks and atomics in AioCompletion to improve performance

librbd: Modify locks and atomics in AioCompletion to improve performance

Signed-off-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>

Merge PR #58222 into main

* refs/pull/58222/head:
qa: ignore container checkpoint/restore related selinux denials for centos9

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #58463 from athanatos/sjust/wip-66827-interrupt-epoch

crimson: audit and correct epoch captured by IOInterruptCondition

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/krbd: override extra_system_packages directly on install task

Follow commit c61cb16223fa ("qa/suites/rbd: override
extra_system_packages directly on install task").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: stress test for recovering from watch errors for -o exclusive

This is based on a test added in commit 237aa221ebad ("qa/suites/krbd:
stress test for recovering from watch errors") for regular mappings.

Fixes: https://tracker.ceph.com/issues/67097
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58710 from rhcs-dashboard/multisite-test-case-fix

mgr/dashboard: multisite e2e fixes

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #58716 from rhcs-dashboard/expand-cluster-back-btn-fix

mgr/dashboard: expand-cluster back navigation fix

Merge pull request #58677 from rhcs-dashboard/fix-cephfs-e2e-error

mgr/test_orchestrator: add apply_mds command in the test_orch

Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #58524 from zdover23/wip-doc-2024-07-11-cephfs-index

doc/cephfs: streamline a paragraph

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #58572 from VVoidV/lbr-fix-head-chunked-encoding

[RGW] Fix the handling of HEAD requests that do not comply with RFC standards

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58665 from cbodley/wip-67012

rgw/iam: CreateRole ignores EEXIST after forwarded request

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

script/backport-create-issue: retry without if assignee is invalid

Resolves:

    WARNING:root:Missing issues will be created in Backport tracker of the relevant Redmine project
    INFO:root:Redmine key was read from '$REDMINE_API_KEY'; using it
    INFO:root:Processing 35 issues with status Pending Backport
    Traceback (most recent call last):
      File "/home/runner/work/ceph/ceph/src/script/backport-create-issue", line 411, in <module>
        iterate_over_backports(redmine, issues, dry_run=args.dry_run)
      File "/home/runner/work/ceph/ceph/src/script/backport-create-issue", line 354, in iterate_over_backports
        update_relations(r, issue, dry_run)
      File "/home/runner/work/ceph/ceph/src/script/backport-create-issue", line 242, in update_relations
        other = r.issue.create(project_id=issue['project']['id'],
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/redminelib/managers/base.py", line 187, in create
        response = self.redmine.engine.request(self.resource_class.http_method_create, url, data=request)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/redminelib/engines/base.py", line 83, in request
        return self.process_response(self.session.request(method, url, **kwargs))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/redminelib/engines/base.py", line 178, in process_response
        raise exceptions.ValidationError(', '.join(': '.join(e) if isinstance(e, list) else e for e in errors))
    redminelib.exceptions.ValidationError: Assignee is invalid
    Examining issue#66797 (1/35)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #58214 into main

* refs/pull/58214/head:
qa/cephfs: ignore variant of MDS_UP_LESS_THAN_MAX

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #58327 into main

* refs/pull/58327/head:
cephfs-shell: excute cmd 'rmdir_helper' reported error

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #57857 into main

* refs/pull/57857/head:
qa: set debug_ms=2 for cephfs tests
msg: decrease the level to print the sending msgs from the protocol

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58703 from kshtsk/wip-fix-unused-in-erasure-code

erasure-code/clay: cleanup unused but set variable

crimson/osd: execute PGListener async operations in order

- Adds ShardServices::singleton_orderer_t mechanism to ensure that
  OSDSingleton calls are completed in order.
- Updates ShardServices accessors invoked from PeeringListener handlers
  to use orderer.
- Updates PGListener handlers and complete_rctx to use orderer.

Fixes: https://tracker.ceph.com/issues/66316
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../alien_store: access coll_map under coll_map_lock

Fixes: https://tracker.ceph.com/issues/66294
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../alien_store: clear coll_map on stop()

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../alien_store: update coll_map from alien tp

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../alien_store: factor out get_alien_coll_ref

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../alien_store: add comment explaining coll_map

Signed-off-by: Samuel Just <sjust@redhat.com>

qa: suppress __trans_list_add valgrind warning

Fixes: https://tracker.ceph.com/issues/65779
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58215 from badone/wip-tracker-59380-admin-socket-injectfull

qa/suites/rados: Cancel injectfull to allow cleanup

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #57951 from mohit84/issue_66231

AsyncMessenger: Don't decrease l_msgr_active_connections if it is negative

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57797 from ajarr/wip-66011

rbd: add `group snap info` command

Reviewed-by: N Balachandran <nibalach@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #57485 from sseshasa/wip-fix-validator-osd-down-grace-tmout

qa/suites/rados/verify/validater: increase heartbeat grace timeout

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

crimson/.../shard_services: add logging to local and remote reservers

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../pg: add logging to most PeeringState::Listener implementations

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #58276 into main

* refs/pull/58276/head:
mds/QuiesceDbManager: get requested state of members before iterating through each quiesce set members

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa: generate valgrind suppressions

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58709 from kshtsk/wip-assert-equal-please

mgr/dashboard: use assertEqual in test_iscsi

Merge pull request #58735 from yuvalif/wip-yuval-63786

test/cls_2pc_queue: fix multi-threaded access to non-atomic counter

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58745 from cyx1231st/wip-fix-blk-aio-compile-issue

blk/aio: fix compile issue when HAVE_LIBURING isn't defined

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/cls_2pc_queue: fix multi-threaded access to non-atomic variables

Fixes: https://tracker.ceph.com/issues/63786
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #58367 from xxhdx1985126/wip-seastore-lba-mapping-iterator

crimson/os/seastore/cached_extent: add the "refresh" ability to lba mappings

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

blk/aio: fix compile issue when HAVE_LIBURING isn't defined

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57782 from myoungwon/wip-nvme-pi

crimson/os/seastore: add checksum offload to RBM

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #58705 from gitkenan/crimsonDocChanges

doc/dev/crimson: making it more user-friendly, re-arranging chapters and fixing typos

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58275 from NitzanMordhai/wip-nitzn-host-thraser-fix-min-in-checks

suites: host thrasher should check min_in before thrashing host

Merge pull request #58393 from NitzanMordhai/wip-nitzan-old-client-thrasher-slow-ops-ignorelist

suites: thrash old client missing SLOW_OPS in the ignore-list

Merge pull request #58277 from NitzanMordhai/wip-nitzan-upgrade-suites-quincy-ignore-list-adding

suites/upgrade/quincy-x: update the ignore list

Merge pull request #58600 from zdover23/wip-doc-2024-07-16-dev-guide-basic-workflow

doc/dev: add formatting to basic workflow

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rbd: add group snap info command

... to show information about a group snapshot.

And also include group snap ID in `group snap ls` output.

Fixes: https://tracker.ceph.com/issues/66011
Signed-off-by: Ramana Raja <rraja@redhat.com>

Fixes: https://tracker.ceph.com/issues/67070
Signed-off-by: Naman Munet <nmunet@redhat.com>

Merge pull request #58725 from zdover23/wip-doc-2024-07-23-dev-dev-guide-basic-workflow

doc/dev_guide: add needs-upgrade-testing label info

Reviewed-by: Laura Flores <lflores@ibm.com>

doc/dev_guide: add needs-upgrade-testing label info

Add an explanation of the "needs-upgrade-testing" label and explain when
to use it.

Laura is the true author of this text.

Co-authored-by: Laura Flores <lflores@ibm.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/dev/crimson: clarify and rearrange for userability

Signed-off-by: gitkenan <kenan.mth@gmail.com>

Merge pull request #56517 from gukaifeng/doc-amend

doc/start/os-recommendations: remove 16.2.z support for CentOS 7

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #58353 from pereman2/bad-collection-objectstore

tools/objectstore: check for wrong coll open_collection

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57002 from aclamk/wip-aclamk-bs-storetest-expand-synthetic

Improved structure for objectstore unit tests.

Fixes: https://tracker.ceph.com/issues/67074
Signed-off-by: Naman Munet <nmunet@redhat.com>

doc/start/os-recommendations: remove 16.2.z support for CentOS 7

Merge pull request #58639 from zdover23/wip-doc-2024-07-17-start-os-recommendations

doc/start: add tested container host oses

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: add tested container host oses

Add a list of OSes that have been tested as container hosts. This commit
is added in response to a request made by Josh Durgin on Slack.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: use assertEqual in test_iscsi

Python asks in multiple places to use assertEqual instead of
assertEquals:

  DeprecationWarning: Please use assertEqual instead.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

crimson/os/seastore/btree: drop unused lines

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cached_extent: add the "refresh" ability to lba
mappings

So that we don't have to re-iterate through the whole lba tree to get
a "parent-viewable" mapping in most cases.

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: rename PhysicalNodeMapping::is_parent_valid() to
PhysicalNodeMapping::is_parent_viewable()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/btree: add copy_source->pending links to FixedKVNode

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cbj: skip crc calculation for cbj header if device supports the checksum offload

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: disable crc calculation if end to end data protection is enabled

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: add handle_exception() to pass_through_io()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: remove multistream related codes

Current codes allow the device to allocate multiple namespace without specific policy
if the nvme device report that it is capable of mutistream functionality.
So, this commit removes the multistream related code, leaving it as a TODO.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: add nvme_read and nvme_write to be used when end to end data protection is enabled

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: turn on end-to-end-data-protection during mkfs if possible

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: replace rbm_metadata_header_t with rbm_superblock_t

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

common/async: fix duplicate definition errors from SharedMutexImpl

the linker complains that several SharedMutexImpl functions are
duplicated, if "common/async/shared_mutex.h" is included by more than
one translation unit

added 'inline' to the function definitions so they're shared instead of
duplicated

Signed-off-by: Casey Bodley <cbodley@redhat.com>

erasure-code/clay: cleanup unused but set variable count_retrieved_sub_chunks

Addresses clang 17 error:

/home/rocky/ceph/src/erasure-code/clay/ErasureCodeClay.cc:481:7: error: variable 'count_retrieved_sub_chunks' set but not used [-Werror,-Wunused-but-set-variable]

Signed-off-by: kshtsk <kyrylo.shatskyy@clyso.com>

Merge pull request #58452 from Matan-B/wip-matanb-seastar-july7

seastar: update submodule

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #58256 from NitzanMordhai/wip-nitzan-wait-osd-admin-command

crimson/osd: adding osdmap subscribe

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58700 from kshtsk/wip-fix-unused-in-msg-async-rdma

msg/async/rdma: cleanup unused variables

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #58688 from idryomov/wip-67051

qa/workunits/rbd: avoid caching effects in luks-encryption.sh

Reviewed-by: Mykola Golub <mgolub@suse.com>

msg/async/rdma: cleanup unused variables

Fixes: 2fb8be43fa2a758cc7934f7a82a48ad2d47d1112
Signed-off-by: kshtsk <kyrylo.shatskyy@clyso.com>