doc/rados: add "pgs not deep scrubbed in time" info

Add a procedure to doc/rados/operations/health-warnings.rst that
explains how to remedy the "X PGs not deep-scrubbed in time" health
warning.

This procedure was developed by Eugen Block, and is at the time of this
commit available on his blog at
https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-time/

Co-authored-by: Eugen Block <eblock@nde.ag>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit d620a51c306d1ebcdfc974dc0b85dc46cfde8d40)

Merge pull request #58221 from guits/wip-66025-quincy

quincy: ceph-volume: create LVs when using partitions

Merge pull request #58999 from guits/wip-64643-quincy

quincy: src/ceph-volume/ceph_volume/devices/lvm/listing.py : lvm list filters with vg name

Merge pull request #59702 from zdover23/wip-doc-2024-09-10-backport-59675-to-quincy

quincy: doc/README.md: improve formatting

Merge pull request #59684 from zdover23/wip-doc-2024-09-10-backport-59677-to-quincy

quincy: doc/man: edit ceph-bluestore-tool.rst

doc/README.md: improve formatting

Improve the formatting of the section "Building Ceph" in the file
README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 7171f73c81d603f8ff62198fc1e5f50c20a0aa05)

doc/man: edit ceph-bluestore-tool.rst

Edit the section "bluefs-bdev-migrate" in
doc/man/8/ceph-bluestore-tool.rst to add the information that this
operation expands the target storage by updating its size label, making
"bluefs-bdev-expand" unnecessary.

Improve the subject-verb agreement in this section, and supply some
absent definite articles.

Co-authored-by: Peter Gervai <grin@drop.grin.hu>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 6b34707f827b2b197f53fe2e430d173b30b81401)

Merge pull request #59665 from zdover23/wip-doc-2024-09-09-backport-59642-to-quincy

quincy: doc/cephadm: add malformed-JSON removal instructions

Merge pull request #59639 from zdover23/wip-doc-2024-09-06-backport-59630-to-quincy

quincy: doc/README.md: edit "Build Prerequisites"

doc/cephadm: add malformed-JSON removal instructions

Add instructions for removing malformed JSON that causes the cephadm
module to fail to start, which in turn makes the orchestrator fail to
start.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 93d90ea358cf17b08e502caf8529061a10008fad)

doc/README.md: edit "Build Prerequisites"

Make the sentences in the "Build Prerequisites" section of README.md
more direct.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 0b277fe3809c2eae7bfced002ac85f7e1629fa30)

Merge pull request #59600 from idryomov/wip-67752-quincy

quincy: rbd: amend "rbd {group,} rename" and "rbd mirror pool" command descriptions

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #56616 from prazumovsky/wip-63620

quincy: rgw/swift: preserve dashes/underscores in swift user metadata names

Merge pull request #56145 from yuvalif/wip-64793-quincy

quincy: rgw/notification: Kafka persistent notifications not retried and removed even when the broker is down

Merge pull request #56447 from pr0ton11/wip-65005

quincy: RGW: allow user disabling presigned urls in rgw configuration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rbd: mention namespace in "rbd mirror pool" command descriptions

Commit 5e64748927d0 ("doc/rbd: add namespace information for mirror
commands") did this for the man page, update the built-in help as well.

The "by default" bit in the description of "rbd mirror pool enable" and
"rbd mirror pool disable" commands is specific to pool mode which is in
turn specific to journal-based mirroring, so it's removed.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 7e02946a3f60da0dcc99e0b4a4b6b3d359db1c15)

rbd: clarify "rbd {group,} rename" command description

Make it clearer that, despite a full image or group spec being taken
for source and destination, an image or a group can be renamed only
within its pool or namespace.

Rename across pools or namespaces within the same pool is unsupported.

Fixes: https://tracker.ceph.com/issues/67752
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 5f064a52707af58fd6007734a768899aeb5a32fc)

Merge pull request #59584 from zdover23/wip-doc-2024-09-04-backport-59546-to-quincy

quincy: doc/rados/operations: Improve health-checks.rst

Merge pull request #59587 from zdover23/wip-doc-2024-09-04-backport-59577-to-quincy

quincy: doc/mds: improve wording

Merge pull request #59561 from zdover23/wip-doc-2024-09-02-backport-59556-to-quincy

quincy: doc: Correct link to Prometheus docs

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/mds: improve wording

Signed-off-by: Piotr Parczewski <piotr@stackhpc.com>
(cherry picked from commit 332804bad58c892d01d2d2da557e42104365ef8a)

doc/rados/operations: Improve health-checks.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 2aa82539ede8ade095ad0b22e973eac538354711)

rgw/notification: Fix Kafka persistent notification  breakage that are not retried.

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>
(cherry picked from commit b2fadb65ae6a9eb3eaade56e04b08f2b694b220c)

doc: Correct link to Prometheus docs

The link is to the `#http_sd_config` anchor in the prometheus config docs; that link only works without the trailing `/`.

This correction would ideally get backported to at least reef & squid.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
(cherry picked from commit 84a30ba6b94b34806faac8217ccaa299c9ee68d6)

Merge pull request #59550 from zdover23/wip-doc-2024-09-01-backport-59544-to-quincy

quincy: doc: update tests-integration-testing-teuthology-workflow.rst

doc: update tests-integration-testing-teuthology-workflow.rst

* add "Infrastructure" section.

* move "Naming the ceph-ci branch" section under
   "Getting binaries - Build Ceph". Also mention
   about centos9-only trick.

* in "Teuthology Archives", mention about developer
   playground machines and ceph log files.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>
(cherry picked from commit 9bfcb8e17db8c61e523e10856d12b237433d831a)

Merge pull request #59542 from zdover23/wip-doc-2024-08-31-backport-59528-to-quincy

quincy: doc/ceph-volume: add spillover fix procedure

doc/ceph-volume: add spillover fix procedure

Add a procedure that explains how, after an upgrade, to move bytes that
have spilled over to a relatively slow device back to the faster device.

This procedure was developed by Chris Dunlop on the [ceph-users] mailing
list, here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/POPUFSZGXR3P2RPYPJ4WJ4HGHZ3QESF6/

Eugen Block requested the addition of this procedure to the
documentation on 30 Aug 2024.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 98618aaa1c8b786c7d240a210b62cc737fdb048d)

Merge pull request #59359 from k0ste/wip-64380-quincy

quincy: rgw/auth/s3: validate x-amz-content-sha256 for empty payloads

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #59132 from k0ste/wip-67133-quincy

quincy: osd/ECTransaction: Remove incorrect asserts in generate_transactions

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #59283 from k0ste/wip-65041-quincy

quincy: librados: use CEPH_OSD_FLAG_FULL_FORCE for IoCtxImpl::remove

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #59272 from k0ste/wip-63064-quincy

quincy: test/cls_lock: expired lock before unlock and start check

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Sunil Angadi <Sunil.Angadi@ibm.com>

Merge pull request #59320 from k0ste/wip-58592-quincy

quincy: rgw/rgw-gap-list: refactoring and adding more error checking

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #59316 from k0ste/wip-57415-quincy

quincy: rgw: Use STANDARD storage class in objects appending operation when the

Merge pull request #59315 from k0ste/wip-55483-quincy

quincy: librgw: teach librgw about rgw_backend_store

Merge pull request #59314 from k0ste/wip-55289-quincy

quincy: rgw: disable RGWDataChangesLog::add_entry() when log_data is off

Merge pull request #59313 from k0ste/wip-54464-quincy

quincy: rgw: cmake configure error on fedora-37/rawhide

Merge pull request #59337 from k0ste/wip-62759-quincy

quincy: rgw: when there are a large number of multiparts, the unorder list result may miss objects

Merge pull request #59336 from k0ste/wip-59499-quincy

quincy: rgw/lua: fix CopyFrom crash

Merge pull request #59322 from k0ste/wip-58364-quincy

quincy: rgw/admin/notifications: support admin operations on topics with tenants

Merge pull request #59321 from k0ste/wip-58507-quincy

quincy: rgw/rgw-orphan-list: refactor and add more checks to the tool

Merge pull request #59319 from k0ste/wip-57421-quincy

quincy: rgw: avoid use-after-move in RGWDataSyncSingleEntryCR ctor

Merge pull request #59378 from k0ste/wip-64576-quincy

quincy: osd: always send returnvec-on-errors for client's retry

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #57614 from sajibreadd/wip-66147-quincy

quincy: osd: fix for segmentation fault on OSD fast shutdown

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57585 from NitzanMordhai/wip-63979-quincy

quincy: mgr/BaseMgrModule: Optimize CPython Call in Finish Function

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57199 from batrick/wip-65736-quincy

quincy: pybind/mgr: disable sqlite3/python autocommit

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55607 from jzhu116-bloomberg/wip-64447-quincy

quincy: rgw: do not copy olh attributes in versioning suspended bucket

Merge pull request #59325 from cbodley/wip-67618-quincy

quincy: qa: barbican: restrict python packages with upper-constraints

Merge pull request #54874 from trociny/wip-63776-quincy

quincy: [rgw][lc][rgw_lifecycle_work_time] adjust timing if the configured end time is less than the start time

Merge pull request #54873 from trociny/wip-56628-quincy

quincy: rgwlc: permit lifecycle to reduce data conditionally in archive zone

Merge pull request #56163 from yuvalif/wip-64886-quincy

quincy: rgw/kafka: set message timeout to 5 seconds

Merge pull request #55882 from kamoltat/wip-64673-quincy

quincy: qa/tasks/ceph_manager.py: Rewrite test_pool_min_size
Laura Flores <lflores@redhat.com>

Merge pull request #59432 from zdover23/wip-doc-2024-08-26-backport-59428-to-quincy

quincy: doc/cephadm: how to get exact size_spec from device

Merge pull request #59412 from adk3798/wip-67683-quincy

quincy: mgr/cephadm: add "original_weight" parameter to OSD class

Reviewed-by: John Mulligan <jmulligan@redhat.com>

doc/cephadm: how to get exact size_spec from device

Add instructions for retrieving the exact size of block devices.

Fixes: https://tracker.ceph.com/issues/66754
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit d00d1b52d50b5575d918c3be7b7a8249ef31f0a8)

Merge pull request #59426 from zdover23/wip-doc-2024-08-25-backport-59418-to-quincy

quincy: doc/glossary: add "object storage"

doc/glossary: add "object storage"

Add a (very basic) definition of object storage.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 43057b88538e868b817acb04d5e6c4e95b4c716e)

Merge pull request #59360 from k0ste/wip-67483-quincy

quincy: rgw: catches nobjects_begin() exceptions

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #59357 from k0ste/wip-64304-quincy

quincy: rgw: fix issue with concurrent versioned deletes leaving behind olh entries

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #59353 from k0ste/wip-63584-quincy

quincy: rgw/notifications: cleanup all coroutines after sending the notification

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #59350 from k0ste/wip-63424-quincy

quincy: rgwfile: fix lock_guard decl

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #59347 from k0ste/wip-63404-quincy

quincy: rgw/multisite: fix sync_error_trim command

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #59345 from k0ste/wip-59605-quincy

quincy: rgw/sts: changing identity to boost::none, when role policy

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #59344 from k0ste/wip-59608-quincy

quincy: rgw: add crypt attrs for iam policy to PostObj and Init/CompleteMultipart

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #59342 from k0ste/wip-61190-quincy

quincy: rgw/lua: add lib64 to the package search path

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #59340 from k0sReviewed-by: Yuval Lifshitz <ylifshit@redhat.com>te/wip-61521-quincy

quincy: test/rgw/notifications: fix kafka consumer shutdown issue

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #59382 from zdover23/wip-doc-2024-08-21-backport-59348-to-quincy

quincy: doc/rados: document unfound object cache-tiering scenario

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

mgr/cephadm: add "original_weight" parameter to OSD class

Fixes: https://tracker.ceph.com/issues/67329
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 04330f5df92994882efcd4879d5c37279138e97b)

Merge pull request #54795 from k0ste/wip-62414-quincy

quincy: rgw/auth: Fix the return code returned by AuthStrategy,

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #54792 from k0ste/wip-63625-quincy

quincy: rgw: SignatureDoesNotMatch for certain RGW Admin Ops endpoints w/v4 auth

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #54170 from yuvalif/wip-57888-quincy

quincy: rgw/amqp: store CA location string in connection object

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #54128 from BBoozmen/wip-63256-quincy

quincy: RGW/STS: when generating keys, take the trailing null character into account

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #54017 from k0ste/wip-57198-quincy

quincy: rgw: 'bucket check' deletes index of multipart meta when its pending_map is nonempty

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59330 from ljflores/wip-tracker-67562

quincy: qa/workunits/rados: enable crb and install generic package for c9

Merge pull request #57407 from k0ste/wip-62926-quincy

quincy: mon: stuck peering since warning is misleading

Merge pull request #55359 from ljflores/wip-revert-pr-55264

quincy: doc/rados: revert "doc/rados/operations: document `ceph balancer status detail`"

Merge pull request #59076 from tobias-urdin/quincy-keystone-admin-token

quincy: rgw: invalidate and retry keystone admin token

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57899 from cbodley/wip-59357-quincy

quincy: rgw/keystone: use secret key from EC2 for sigv4 streaming mode

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #57149 from k0ste/wip-57658-quincy

quincy: rgw: set requestPayment in slave zone

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #57133 from k0ste/wip-59376-quincy

quincy: rgw: handle old clients with transfer-encoding: chunked.

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #57132 from k0ste/wip-59144-quincy

quincy: Do not duplicate query-string in ops-log

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #57131 from k0ste/wip-58327-quincy

quincy: rgw: fix ListOpenIDConnectProviders XML format

Reviewed-by: Adam Emerson <aemerson@redhat.com>

doc/rados: Revert "doc/rados/operations: document `ceph balancer status detail`"

This reverts commit 7fff8e9a981c366390f61cf1eaeba6524fd6d365.

Signed-off-by: Laura Flores <lflores@ibm.com>

doc/rados: document unfound object cache-tiering scenario

Explain how to deal with "unfound objects" when restarting OSDs in a
cache-tiered environment.

Fixes: https://tracker.ceph.com/issues/44286
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f01d7a8d5b85170c034acb962b9833913853a1c5)

osd: always send returnvec-on-errors for client's retry

Currently there is a discrepancy in terms of the returnvec's
presence between MOSDOpReplys sent for original requests and
those on dups. The former always contain the returnvec if
an error happened, even if `allows_returnvec()` is `false`.

This commit extends the behavior on dups.

For RCA please see: https://tracker.ceph.com/issues/64192#note-9

Fixes: https://tracker.ceph.com/issues/64192
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 613575719191ce55bc8997577965f3a03e22453c)

rgw: RGWSI_RADOS::Pool::List::init() catches IoCtx::nobjects_begin() exceptions

Signed-off-by: lichaochao <lichaochao2_yewu@cmss.chinamobile.com>
(cherry picked from commit 63c7d2a44b421cb95bb62ec3db9b2ff84e72d059)

rgw:log_list_init() catches IoCtx::nobjects_begin() exceptions

Signed-off-by: lichaochao <lichaochao2_yewu@cmss.chinamobile.com>
(cherry picked from commit 14176f00f44d51b7ecbf9952a1fbbe056f7988eb)

rgw/auth/s3: validate x-amz-content-sha256 for empty payloads

when is_v4_payload_empty(), we return a null completer so never try to
validate the x-amz-content-sha256 for signed payloads. add this
checksum comparison to get_auth_data_v4() before we create the completer

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4bb49478fae09ead4646c1baada3bbc9a2555130)

rgw: fix issue with concurrent versioned deletes leaving behind olh entries

Fixes a scenario where BI OLH entries were left behind after racing
delete requests for the same versioned object instance.

Fixes: https://tracker.ceph.com/issues/64014
Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 66ac828f97c1b368e41c51be4e5699f3e92d47f9)

rgw: add reproducer for bug with concurrent versioned object deletes

Adds a test case to reproduce a scenario where concurrent versioned
object deletes can cause leftover OLH entries to be left behind.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit b65046e37f62f2c65b17ba6f3434a19d3d68c983)

rgw/notifications: cleanup all coroutines after sending the notification

this is fixing a regression from: 6b6592f50b6b81fa13a330bcb91273ba7f25c0c9

Fixes: https://tracker.ceph.com/issues/63580
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 63e14893cc7dc6a3aec92c0867ce5e3e67f44b29)

rgwfile: fix lock_guard decl

Reported by Leonid Usov <leonid.usov@ibm.com>.

Fixes: https://tracker.ceph.com/issues/63409
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit f637e53eb62f4f940918a793865db6629fc21275)

rgw/multisite: fix sync_error_trim command.
without --marker option, we trim everthing.
if --marker option is passed in, we trim upto that marker.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
(cherry picked from commit a0491a4f63885be9ea14a15c67cabd8903f19641)

rgw/sts: changing identity to boost::none, when role policy
is verified for putobj permissions, in case of renaming a
large file.

While renaming a large file, putobj is invoked as an intermediate
step, and role policy is verified for the source object if temp creds
are used. Since the role policy is attached to the identity (role)
itself and the role policy does not contain a Principal, there is no
need to verify the identity and hence boost::none is passed in place
of the identity.

fixes: https://tracker.ceph.com/issues/58628

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
(cherry picked from commit c2f5716e5196073abfc50917e5f687888f6dff42)

rgw/op: Init/CompleteMultipart add crypt attrs for iam policy

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4cc6e05a7a550b277c22ac43f2088fe7a0423716)

rgw/op: RGWPostObj adds crypt attrs for iam policy

Fixes: https://tracker.ceph.com/issues/58594
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c224bae02218b574f360cb718feb8f2a50f7ac7e)

rgw/op: factor out rgw_iam_add_crypt_attrs()

make the iam environment variables for server-side encryption available
to other ops

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 3b517a2679c62529950cd66136f9fb971ae612cc)

rgw/lua: add lib64 to the package search path

Fixes: https://tracker.ceph.com/issues/59733
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 001e0c004ebb871fe09e70ba66fddbc545705c34)

test/rgw/notifications: fix kafka consumer shutdown issue

Fixes: https://tracker.ceph.com/issues/61477
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit b00f92da9a6cf848705c0508c0166c12913731e1)

Revert "rgw: update last_added_entry when count == num_entries"

This reverts commit 7511f9f675ea4e43992605dc03109bc5f356a5e1.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 81d5bb74d7cd77033f9ab031fff99999e80f4a58)