Merge pull request #58641 from hualongfeng/qat_doc

doc/radosgw/qat-accel: Update and Add QATlib information

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/radosgw/qat-accel: Update and Add QATlib information

Signed-off-by: Feng, Hualong <hualong.feng@intel.com>

Merge pull request #58798 from cbodley/wip-common-async-spawn-throttle-spawn

common/async: spawn_throttle wraps call to asio::spawn()

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #58449 from cbodley/wip-rgw-gc-list-async

cls/rgw: gc_list uses ObjectOperation instead of IoCtx

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #58046 from hualongfeng/wip_qat_zlib_data_format

Compressor: Add data format(QZ_DEFLATE_GZIP_EXT) for QAT Zlib

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #57546 from clwluvw/vault-token

rgw: eliminate vault token perm for group read

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

Merge pull request #58689 from clwluvw/sse-c-policy

rgw: add support sse-c to bucket policy conditions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58534 from afreen23/wip-nvmeof-initiators

mgr/dashboard: Add initiators

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #58844 from ceph/login-text-ui

mgr/dashboard: Changes for Sign out text to Login out

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Add initiators add/update in dashboard

Fixes  https://tracker.ceph.com/issues/66907

- add one or more initiators
- remove one or more initiators
- introduces two new UI routers for the above two

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

mgr/dashboard: Changes for Sign out text

Fixes: https://tracker.ceph.com/issues/67075
Signed-off-by: Prachi Goel <prachi@li-f0e8f2cc-27e0-11b2-a85c-b2772164128d.ibm.com>

Merge pull request #58722 from cbodley/wip-common-async-shared-mutex-impl

common/async: fix duplicate definition errors from SharedMutexImpl

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #58838 from idryomov/wip-rbd-lock-incompat-note

doc/rbd/rbd-exclusive-locks: mention incompatibility with advisory locks

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/rbd/rbd-exclusive-locks: mention incompatibility with advisory locks

This is due to both using the same name for the lock at the RADOS level
("rbd_lock").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58348 from cbodley/wip-async-co-algorithms

common/async: add primitives for structured concurrency with c++20 coroutines

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #58686 from clwluvw/rgw-noprinc-allow

rgw: donot allow NotPrincipal with Allow Effect

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #58406 from mohit84/issue_42884

test: Create ParallelPGMapper object before start threadpool

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57223 from myoungwon/wip-rename-global-values-ceph-dedup-daemon

tool: renaming and refactoring global variables in ceph-dedup-daemon

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #58781 from idryomov/wip-67097

qa/suites/krbd: stress test for recovering from watch errors for -o exclusive

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58766 from athanatos/sjust/wip-66294-collection-race

crimson: access coll_map under alien tp with a lock

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58464 from athanatos/sjust/wip-66316-async-reserver

crimson: peering event processing fixes,  wait for async operations started during peering events

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57776 from rzarzynski/wip-bug-61948

mon, osd, *: expose upmap-primary in OSDMap::get_features()

Merge pull request #56932 from ceph/wip-yuriw-release-18.2.3-main

doc: reef 18.2.4 release note

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

common/async: spawn_throttle wraps call to asio::spawn()

cancellation of the parent must immediately cancel its children, which
only works if the children are on the same executor as the parent

prohibit child coroutines from being spawned on a different executor by
wrapping the call to asio::spawn() in a new spawn_throttle::spawn()
interface

expose an overload for asio::spawn()'s optional StackAllocator argument

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/releases: add release note about thread creation incompatibility

Signed-off-by: Laura Flores <lflores@ibm.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

common/async: add parallel_for_each() algorithm

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/async: add co_spawn_group template for fork-join parallelism

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/async: implement max_concurrent_for_each() for awaitable

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/async: add co_throttle for bounded concurrency with c++20 coroutines

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58535 from anoopcs9/mgr-nfs-fix-export-create-conf-obj

mgr/nfs: Do not ignore clusters from rados pool conf objects

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Ponnuvel Palaniyappan <pponnuvel@gmail.com>

Merge pull request #58533 from rhcs-dashboard/shared-folder-pythonV

cephadm: update shared folder python packages dir

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58153 from anoopcs9/modify-smb-dot-conf

mgr/smb: Make appropriate changes to global and share specific options

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58670 from ivancich/wip-bolster-reshard-testing

test/rgw: address potential race condition in reshard testing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #58056 into main

* refs/pull/58056/head:
qa: suppress __trans_list_add valgrind warning
qa: generate valgrind suppressions

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58041 from vmackie/class-param-bug

mon: fix inconsistencies in class param

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge PR #58768 into main

* refs/pull/58768/head:
script/backport-create-issue: retry without if assignee is invalid

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #58152 from adam-lj/aiocompletion-perf-changes

librbd: Modify locks and atomics in AioCompletion to improve performance

librbd: Modify locks and atomics in AioCompletion to improve performance

Signed-off-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>

Merge PR #58222 into main

* refs/pull/58222/head:
qa: ignore container checkpoint/restore related selinux denials for centos9

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #58463 from athanatos/sjust/wip-66827-interrupt-epoch

crimson: audit and correct epoch captured by IOInterruptCondition

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/krbd: override extra_system_packages directly on install task

Follow commit c61cb16223fa ("qa/suites/rbd: override
extra_system_packages directly on install task").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: stress test for recovering from watch errors for -o exclusive

This is based on a test added in commit 237aa221ebad ("qa/suites/krbd:
stress test for recovering from watch errors") for regular mappings.

Fixes: https://tracker.ceph.com/issues/67097
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58710 from rhcs-dashboard/multisite-test-case-fix

mgr/dashboard: multisite e2e fixes

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #58716 from rhcs-dashboard/expand-cluster-back-btn-fix

mgr/dashboard: expand-cluster back navigation fix

Merge pull request #58677 from rhcs-dashboard/fix-cephfs-e2e-error

mgr/test_orchestrator: add apply_mds command in the test_orch

Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #58524 from zdover23/wip-doc-2024-07-11-cephfs-index

doc/cephfs: streamline a paragraph

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #58572 from VVoidV/lbr-fix-head-chunked-encoding

[RGW] Fix the handling of HEAD requests that do not comply with RFC standards

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58665 from cbodley/wip-67012

rgw/iam: CreateRole ignores EEXIST after forwarded request

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

script/backport-create-issue: retry without if assignee is invalid

Resolves:

    WARNING:root:Missing issues will be created in Backport tracker of the relevant Redmine project
    INFO:root:Redmine key was read from '$REDMINE_API_KEY'; using it
    INFO:root:Processing 35 issues with status Pending Backport
    Traceback (most recent call last):
      File "/home/runner/work/ceph/ceph/src/script/backport-create-issue", line 411, in <module>
        iterate_over_backports(redmine, issues, dry_run=args.dry_run)
      File "/home/runner/work/ceph/ceph/src/script/backport-create-issue", line 354, in iterate_over_backports
        update_relations(r, issue, dry_run)
      File "/home/runner/work/ceph/ceph/src/script/backport-create-issue", line 242, in update_relations
        other = r.issue.create(project_id=issue['project']['id'],
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/redminelib/managers/base.py", line 187, in create
        response = self.redmine.engine.request(self.resource_class.http_method_create, url, data=request)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/redminelib/engines/base.py", line 83, in request
        return self.process_response(self.session.request(method, url, **kwargs))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/redminelib/engines/base.py", line 178, in process_response
        raise exceptions.ValidationError(', '.join(': '.join(e) if isinstance(e, list) else e for e in errors))
    redminelib.exceptions.ValidationError: Assignee is invalid
    Examining issue#66797 (1/35)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #58214 into main

* refs/pull/58214/head:
qa/cephfs: ignore variant of MDS_UP_LESS_THAN_MAX

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #58327 into main

* refs/pull/58327/head:
cephfs-shell: excute cmd 'rmdir_helper' reported error

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #57857 into main

* refs/pull/57857/head:
qa: set debug_ms=2 for cephfs tests
msg: decrease the level to print the sending msgs from the protocol

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58703 from kshtsk/wip-fix-unused-in-erasure-code

erasure-code/clay: cleanup unused but set variable

crimson/osd: execute PGListener async operations in order

- Adds ShardServices::singleton_orderer_t mechanism to ensure that
  OSDSingleton calls are completed in order.
- Updates ShardServices accessors invoked from PeeringListener handlers
  to use orderer.
- Updates PGListener handlers and complete_rctx to use orderer.

Fixes: https://tracker.ceph.com/issues/66316
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../alien_store: access coll_map under coll_map_lock

Fixes: https://tracker.ceph.com/issues/66294
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../alien_store: clear coll_map on stop()

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../alien_store: update coll_map from alien tp

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../alien_store: factor out get_alien_coll_ref

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../alien_store: add comment explaining coll_map

Signed-off-by: Samuel Just <sjust@redhat.com>

qa: suppress __trans_list_add valgrind warning

Fixes: https://tracker.ceph.com/issues/65779
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58215 from badone/wip-tracker-59380-admin-socket-injectfull

qa/suites/rados: Cancel injectfull to allow cleanup

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #57951 from mohit84/issue_66231

AsyncMessenger: Don't decrease l_msgr_active_connections if it is negative

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57797 from ajarr/wip-66011

rbd: add `group snap info` command

Reviewed-by: N Balachandran <nibalach@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #57485 from sseshasa/wip-fix-validator-osd-down-grace-tmout

qa/suites/rados/verify/validater: increase heartbeat grace timeout

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

crimson/.../shard_services: add logging to local and remote reservers

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../pg: add logging to most PeeringState::Listener implementations

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #58276 into main

* refs/pull/58276/head:
mds/QuiesceDbManager: get requested state of members before iterating through each quiesce set members

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa: generate valgrind suppressions

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58709 from kshtsk/wip-assert-equal-please

mgr/dashboard: use assertEqual in test_iscsi

cls/rgw: gc_list uses ObjectOperation instead of IoCtx

clean up the only gc function that was hidden with CLS_CLIENT_HIDE_IOCTX

this allows rgw to use it asynchonously with rgw_rados_operate() and
optional_yield, and warn about blocking calls that should be async

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58735 from yuvalif/wip-yuval-63786

test/cls_2pc_queue: fix multi-threaded access to non-atomic counter

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58745 from cyx1231st/wip-fix-blk-aio-compile-issue

blk/aio: fix compile issue when HAVE_LIBURING isn't defined

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/cls_2pc_queue: fix multi-threaded access to non-atomic variables

Fixes: https://tracker.ceph.com/issues/63786
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #58367 from xxhdx1985126/wip-seastore-lba-mapping-iterator

crimson/os/seastore/cached_extent: add the "refresh" ability to lba mappings

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

blk/aio: fix compile issue when HAVE_LIBURING isn't defined

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57782 from myoungwon/wip-nvme-pi

crimson/os/seastore: add checksum offload to RBM

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #58705 from gitkenan/crimsonDocChanges

doc/dev/crimson: making it more user-friendly, re-arranging chapters and fixing typos

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58275 from NitzanMordhai/wip-nitzn-host-thraser-fix-min-in-checks

suites: host thrasher should check min_in before thrashing host

Merge pull request #58393 from NitzanMordhai/wip-nitzan-old-client-thrasher-slow-ops-ignorelist

suites: thrash old client missing SLOW_OPS in the ignore-list

Merge pull request #58277 from NitzanMordhai/wip-nitzan-upgrade-suites-quincy-ignore-list-adding

suites/upgrade/quincy-x: update the ignore list

Merge pull request #58600 from zdover23/wip-doc-2024-07-16-dev-guide-basic-workflow

doc/dev: add formatting to basic workflow

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rbd: add group snap info command

... to show information about a group snapshot.

And also include group snap ID in `group snap ls` output.

Fixes: https://tracker.ceph.com/issues/66011
Signed-off-by: Ramana Raja <rraja@redhat.com>

Fixes: https://tracker.ceph.com/issues/67070
Signed-off-by: Naman Munet <nmunet@redhat.com>

Merge pull request #58725 from zdover23/wip-doc-2024-07-23-dev-dev-guide-basic-workflow

doc/dev_guide: add needs-upgrade-testing label info

Reviewed-by: Laura Flores <lflores@ibm.com>

doc/dev_guide: add needs-upgrade-testing label info

Add an explanation of the "needs-upgrade-testing" label and explain when
to use it.

Laura is the true author of this text.

Co-authored-by: Laura Flores <lflores@ibm.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/dev/crimson: clarify and rearrange for userability

Signed-off-by: gitkenan <kenan.mth@gmail.com>

Merge pull request #56517 from gukaifeng/doc-amend

doc/start/os-recommendations: remove 16.2.z support for CentOS 7

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #58353 from pereman2/bad-collection-objectstore

tools/objectstore: check for wrong coll open_collection

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57002 from aclamk/wip-aclamk-bs-storetest-expand-synthetic

Improved structure for objectstore unit tests.

Fixes: https://tracker.ceph.com/issues/67074
Signed-off-by: Naman Munet <nmunet@redhat.com>

doc/start/os-recommendations: remove 16.2.z support for CentOS 7

Merge pull request #58639 from zdover23/wip-doc-2024-07-17-start-os-recommendations

doc/start: add tested container host oses

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: add tested container host oses

Add a list of OSes that have been tested as container hosts. This commit
is added in response to a request made by Josh Durgin on Slack.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: use assertEqual in test_iscsi

Python asks in multiple places to use assertEqual instead of
assertEquals:

  DeprecationWarning: Please use assertEqual instead.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

crimson/os/seastore/btree: drop unused lines

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cached_extent: add the "refresh" ability to lba
mappings

So that we don't have to re-iterate through the whole lba tree to get
a "parent-viewable" mapping in most cases.

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: rename PhysicalNodeMapping::is_parent_valid() to
PhysicalNodeMapping::is_parent_viewable()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/btree: add copy_source->pending links to FixedKVNode

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cbj: skip crc calculation for cbj header if device supports the checksum offload

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: disable crc calculation if end to end data protection is enabled

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>