test/common: keeping build happy

Since ceph_pthread_setname() has now been moved from a macro to a
function, we need to reference the common library to link it to.

Signed-off-by: Milind Changire <mchangir@redhat.com>

mds: add or update MDS thread names

To be consistent and sensical.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit a2a989aaa5e9780d98ec9c5cabc66ac2c3a387a4)
Signed-off-by: Milind Changire <mchangir@redhat.com>
Conflicts:
src/mds/MDLog.cc (no upkeep thread in reef)
src/mds/QuiesceAgent.h (not available in reef)
src/mds/QuiesceDbManager.cc (not available in reef)

log: cache recent threads up to a day

In general, a pthread implementation reuses thread ids routinely so the
circular_buffer is there to help us see what the thread name was historically
(capped at 4 entries).

However, to guard against this map from growing without bound, discard entries
that are more than a day old. This would happen if a thread logged an Entry and
has since disappeared for more than a day.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit bfe489ed19d599b67cacb0bdd4958f36661248f2)
Signed-off-by: Milind Changire <mchangir@redhat.com>

common: cache pthread names

This provides common ceph entrypoints for the pthread_[gs]name functions which
will also cache a thread_local copy. This also removes the pthread_t parameter
which precipitated the bug i50743.

Obviously, the overall goal here is to avoid system calls.

Conflicts:
src/ceph_nvmeof_monitor_client.cc (not available in reef)
src/common/ceph_timer.h
src/include/compat.h
src/mds/Beacon.cc
src/rgw/driver/rados/rgw_notify.cc
src/rgw/rgw_lua_background.cc (not available in reef)

See-also: https://tracker.ceph.com/issues/50743
Fixes: 0be8d01c9ddde0d7d24edd34dc75f6cfc861b5ba
Fixes: https://tracker.ceph.com/issues/68691
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit 7de5af59c2741b4203cc17f619ca53096c97b8ff)
Signed-off-by: Milind Changire <mchangir@redhat.com>

log: concatenate thread names and print once per thread

Fixes: 0be8d01c9ddde0d7d24edd34dc75f6cfc861b5ba
Fixes: https://tracker.ceph.com/issues/68691
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit 1f40d0c06c4f292d205bc6cc846161e0706a871b)
Signed-off-by: Milind Changire <mchangir@redhat.com>

log: thread name save/fetch infra

* pthread name is saved in a thread_local storage
* the thread_local name is copied into Entry object's ctor
* Log::dump_recent() reads the thread name from the Entry
  object's data member when dumping logs

Fixes: https://tracker.ceph.com/issues/50743
Signed-off-by: Milind Changire <mchangir@redhat.com>
(cherry picked from commit 0be8d01c9ddde0d7d24edd34dc75f6cfc861b5ba)

Merge pull request #60709 from zdover23/wip-doc-2024-11-12-backport-60704-to-reef

reef: doc/start: fix "are are" typo

doc/start: fix "are are" typo

Fix typo reading "They are are single-threaded".
s/are are/are/

Fixes: https://tracker.ceph.com/issues/68901
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 22010719d7edf2a6cf7de7b115ed797e7942ec47)

Merge pull request #60097 from cbodley/wip-68331-reef

reef: rgw/rados: don't rely on IoCtx::get_last_version() for async ops

Merge pull request #59351 from k0ste/wip-63423-reef

reef: rgwfile: fix lock_guard decl

Merge pull request #59346 from k0ste/wip-59607-reef

reef: rgw/sts: changing identity to boost::none, when role policy

Merge pull request #59341 from k0ste/wip-61436-reef

reef: rgw/archive: avoid duplicating objects when syncing from multiple zones

Merge pull request #59261 from k0ste/wip-62748-reef

reef: rgw: add versioning status during `radosgw-admin bucket stats`

Merge pull request #59122 from k0ste/wip-67100-reef

reef: [RGW] Fix the handling of HEAD requests that do not comply with RFC standards

Merge pull request #56400 from batrick/wip-62923-reef

reef: mon/MonmapMonitor: do not propose on error in prepare_update

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58340 from rishabh-d-dave/fixes-and-missing-backports

reef: qa/cephfs: a bug fix and few missing backport for caps_helper.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58492 from YiteGu/backport-reef

reef: client: check mds down status before getting mds_gid_t from mdsmap

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59620 from pponnuvel/wip-67740-reef

reef: client: Prevent race condition when printing Inode in ll_sync_inode

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58397 from dparmar18/wip-64740-reef

reef: client: do not proceed with I/O if filehandle is invalid

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #60344 from zdover23/wip-doc-2024-10-16-backport-60256-to-reef

reef: doc: update Key Idea in cephfs-mirroring.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #56690 from kotreshhr/wip-64940-reef

reef: qa: Add multifs root_squash testcase

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58111 from neesingh-rh/wip-65348-reef

reef: qa: fixing tests in test_cephfs_shell.TestShellOpts

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58726 from neesingh-rh/wip-67077-reef

reef: mds: fix session/client evict command.

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58744 from lxbsz/wip-58654

reef: common/TrackedOp: do not count the ops marked as nowarn

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58770 from lxbsz/wip-66778

reef: mds/client: return -ENODATA when xattr doesn't exist for removexattr

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59054 from vshankar/wip-cleanup-centos8

reef: qa/cephfs: remove dependency on centos8/rhel8 entirely

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #60680 from zdover23/wip-doc-2024-11-09-backport-60605-to-reef

reef: doc/cephadm: clean "Adv. OSD Service Specs"

doc/cephadm: clean "Adv. OSD Service Specs"

Clean the English grammar and the .RST in the section "Advanced OSD
Service Specifications" in the file doc/cephadm/services/osd.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f6b56f9ee5d9f0b54dcaf1832d7a977895eb9c94)

Merge pull request #60645 from zdover23/wip-doc-2024-11-07-backport-60643-to-reef

reef: doc/cephadm: link to "host pattern" matching sect

doc/cephadm: link to "host pattern" matching sect

Link to the "Placement by Pattern Matching" section in
doc/cephadm/services/index.rst from the "Advanced OSD Service
Specifications" section in doc/cephadm/services/osd.rst.

Fixes: https://tracker.ceph.com/issues/68812
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit cdfa299a2511ee35dfe0a857702af6d3ba257efd)

Merge pull request #60638 from zdover23/wip-doc-2024-11-06-backport-56599-to-reef

reef: doc/cephadm: Update operations.rst

doc/cephadm: Update operations.rst

Fixed typo 'no' to 'not' in sentence "If this test fails, cephadm will
no be able to manage services on that host."

Signed-off-by: rhkelson <119636079+rhkelson@users.noreply.github.com>
(cherry picked from commit 34bc35a73443fa56908395a83ea2dd404a8fc4a6)

Merge pull request #60418 from cbodley/wip-63024-reef

reef: qa/rgw/sts: keycloak task installs java manually

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #59104 from YiteGu/wip-53048-reef

reef: os/bluestore: add some slow count for bluestore

Merge pull request #59048 from YiteGu/wip-some-blk-discard-path

reef: a series of optimizations for kerneldevice discard

Merge pull request #59043 from NitzanMordhai/wip-67351-reef

reef: Test: osd-recovery-space.sh extends the wait time for "recovery toofull"

Merge pull request #59629 from sseshasa/wip-66422-reef

reef: osd/scrub: Change scrub cost to average object size

Merge pull request #56475 from amathuria/wip-65141-reef

reef: osd: Change PG Deletion cost for mClock

Merge pull request #60594 from zdover23/wip-doc-2024-11-02-backport-60584-to-reef

reef: doc: s/Whereas,/Although/

doc: s/Whereas,/Although/

Change the subordinating conjunction "Whereas" followed by a comma to
the less grammatically-incorrect "Although". I've been meaning to do
this since 22 Mar 2023.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ea1a470fb4809d312f854e72fc5f8081cbcb2c49)

Merge pull request #58802 from ljflores/wip-67129-reef

reef: osd: bump versions of decoders for upmap-primary

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #58741 from ifed01/wip-ifed-async-compact-reef

reef: os/bluestore: enable async manual compactions

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #58734 from pereman2/wip-67082-reef

reef: tools/objectstore: check for wrong coll open_collection

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58413 from ljflores/wip-66826-reef

reef: qa/suites/rados/thrash/workloads: remove cache tiering workload

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #58315 from cbodley/wip-66723

reef: build: Make boost_url a list

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #57588 from NitzanMordhai/wip-66137-reef

reef: osd: full-object read CRC mismatch due to 'truncate' modifying oi.size w/o clearing 'data_digest'

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #58283 from pponnuvel/wip-66699-reef

reef: mgr/nfs: Don't crash ceph-mgr if NFS clusters are unavailable.

Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #57615 from sajibreadd/wip-66148-reef

reef: osd: fix for segmentation fault on OSD fast shutdown

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #57456 from badone/wip-tracker-59196-lazy-omap-regex-segfault-reef

reef: test/lazy-omap-stats: Convert to boost::regex

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #57241 from sajibreadd/wip-65356-reef

reef: os/bluestore: expand BlueFS log if available space is insufficient

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #55774 from ifed01/wip-ifed-no-death-tests-reef

reef: test/store_test: get rid off assert_death.

Reviewed-by: Md Mahamudur Rahaman Sajib <mahamudur.sajib@croit.io>

Merge pull request #60538 from zdover23/wip-doc-2024-10-29-backport-60508-to-reef

reef: doc/rados: add blaum_roth coding guidance

Merge pull request #60526 from zdover23/wip-doc-2024-10-29-backport-60470-to-reef

reef: doc: explain the consequence of enabling mirroring through monitor co…

doc/rados: add blaum_roth coding guidance

Direct Ceph administrators using blaum_roth coding for erasure-coded
pools to change the default value of w=7 to a different value in order
to ensure that w+1 is prime.

This information was provided to the Ceph upstream by Benjamin Mare in
September of 2024.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 6648d94aed5ff5643f50fcd4f3f07d97b846e885)

doc: explain the consequence of enabling mirroring through monitor commands

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit ed3ce72fbc0b097131945cbb39caa0831401b94a)

Merge pull request #60501 from zdover23/wip-doc-2024-10-26-backport-60444-to-reef

reef: doc/rados: standardize markup of "clean"

Merge pull request #60504 from zdover23/wip-doc-2024-10-26-backport-60434-to-reef

reef: doc/glossary: add "PLP" to glossary

doc/glossary: add "PLP" to glossary

Add "PLP" to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 599bdbf0a240ee718ed6da4104141b0f23330fbe)

Merge pull request #60468 from zdover23/wip-doc-2024-10-24-backport-60465-to-reef

reef: doc/cephfs: edit "Disabling Volumes Plugin"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: add doc for disabling mgr/volumes plugin

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 373910de34fc9e1d98218aed0361cd9bf3badffa)
(cherry picked from commit 8868ecc6ce70603986c0c59a6cc0d3d695dbecaa)

Fix the handling of HEAD requests that do not comply with RFC standards

    According to RFC 9110, The HEAD method is identical to GET except
    that the server MUST NOT send content in the response.

    And RFC 9112 indicates that Transfer-Encoding can be omitted in HEAD
    Response.

    The following HEAD OP is using chunked encoding and send the final chunk

    HEAD /bucket?uploads            RGWListBucketMultiparts_ObjStore_S3
    HEAD /bucket/abc?uploadId       RGWListMultipart_ObjStore_S3
    HEAD /                          RGWListBuckets_ObjStore_S3

    Clients using persistent connections, such as nginx, will be unable to handle
    subsequent requests after processing such requests because the final chunk
    remains in the TCP byte stream, preventing the next HTTP response from
    being correctly parsed.

    Ref: https://www.rfc-editor.org/rfc/rfc9110.html#section-9.3.2
         https://www.rfc-editor.org/rfc/rfc9112#section-6.1-9

Signed-off-by: liubingrun <liubr1@chinatelecom.cn>
(cherry picked from commit e63101650d43a60ae21e6c682f77175690542869)

rgw/archive: avoid duplicating objects when syncing from multiple zones

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
(cherry picked from commit 74014cf8f7b1311ff3ea0a53b901e2cf3c7251c6)

rgw/sts: changing identity to boost::none, when role policy
is verified for putobj permissions, in case of renaming a
large file.

While renaming a large file, putobj is invoked as an intermediate
step, and role policy is verified for the source object if temp creds
are used. Since the role policy is attached to the identity (role)
itself and the role policy does not contain a Principal, there is no
need to verify the identity and hence boost::none is passed in place
of the identity.

fixes: https://tracker.ceph.com/issues/58628

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
(cherry picked from commit c2f5716e5196073abfc50917e5f687888f6dff42)

doc/rados: standardize markup of "clean"

Standardize the markup around the status "clean" in the documentation so
that readers don't mistakenly get the idea that inconsistent
presentation of the word "clean" implies a never-stated difference
between one instance and the other.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit c8b4155d2d27228917d4be1f11514504100c55c7)

Merge pull request #60497 from zdover23/wip-doc-2024-10-26-backport-58647-to-reef

reef: doc/cephfs: add doc for disabling mgr/volumes plugin

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: add doc for disabling mgr/volumes plugin

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 373910de34fc9e1d98218aed0361cd9bf3badffa)

rgw: add versioning status during `radosgw-admin bucket stats`

We received an RFE to including versioning informatio during the
bucket stats op. So in the produced JSON, under "versioning", it will
specify "off", "enabled", or "suspended".

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit ea2d7ff92aa23cdd5b7ede41fc6e7ce54d8c0e3b)

Merge pull request #60483 from zdover23/wip-doc-2024-10-25-backport-60480-to-reef

reef: doc/cephfs: use 'p' flag to set layouts or quotas

doc/cephfs: use 'p' flag to set layouts or quotas

Remind users that clients must use the 'p' flag when writing file
layouts.

This PR is made in response to
https://www.reddit.com/r/ceph/comments/1gbad51/comment/ltkgd5v/

Co-authored-by: Zac Dover <zac.dover@proton.me>
Signed-off-by: TruongSinh Tran-Nguyen <i@truongsinh.pro>
(cherry picked from commit 13c7c9d99881c7b4491ba1a693283c047161ea8a)

Merge pull request #60402 from smanjara/wip-reef-pr-56874

[reef] qa/multisite: stabilize multisite testing

Merge pull request #59326 from cbodley/wip-67619-reef

reef: qa: barbican: restrict python packages with upper-constraints

Merge pull request #59105 from YiteGu/wip-52185-reef

reef: osd/perf_counters: raise prio of before queue op perfcounter

Merge pull request #59103 from YiteGu/wip-53250-reef

reef: os/bluestore: add perfcount for bluestore/bluefs allocator

Merge pull request #58817 from Svelar/wip-67135-reef

reef: bluestore/bluestore_types: avoid heap-buffer-overflow in another way to keep code uniformity

Merge pull request #59415 from clwluvw/wip-67686-reef

reef: rgw: load copy source bucket attrs in putobj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58764 from smanjara/wip-67026-reef

reef: rgw/multisite: don't retain RGW_ATTR_OBJ_REPLICATION_TRACE attr on copy_object

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58058 from yuvalif/wip-66498-reef

reef: rgw: do not assert on thread name setting failures

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #59764 from yuvalif/wip-68046-reef

reef: rgw/kafka: refactor topic creation to avoid rd_kafka_topic_name()

Merge pull request #59669 from mkogan1/wip-67921-reef

reef: rgw: decrement qlen/qactive perf counters on error

Merge pull request #59440 from cbodley/wip-67713-reef

reef: rgw/http: finish_request() after logging errors

Merge pull request #59356 from k0ste/wip-64277-reef

reef: rgw: swift: tempurl fixes for ceph

Merge pull request #60126 from adamemerson/wip-40692-reef

reef: global: Call getnam_r with a 64KiB buffer on the heap

Merge pull request #60462 from zdover23/wip-doc-2024-10-24-backport-60460-to-reef

reef: doc/start: add vstart install guide

doc/start: add vstart install guide

Add "vstart Cluster Installation and Configuration Procedure" to the
Beginner's Guide.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit a42be085bd7b9159f4924853ac2b2d92c37884b0)

Merge pull request #60436 from zdover23/wip-doc-2024-10-23-backport-60433-to-reef

reef: doc/cephfs: rearrange subvolume group information

doc/cephfs: rearrange subvolume group information

Rearranging the documentation to align it with CephFS encapsulation design.

Signed-off-by: Indira Sawant <indira.sawant@ibm.com>
(cherry picked from commit f6cf7ae5281f5740ac78bfa5e2a22913353b8167)

qa/rgw/sts: keycloak task installs java manually

java had already been installed automatically before centos 9. add an
override to install the jdk-17 packages manually

Fixes: https://tracker.ceph.com/issues/62536
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 06f6a3e6e8ad65e75d640a128338ffa9dc246f4a)

Conflicts:
tasks/1-keycloak.yaml was not renamed from first.yaml

multisite/qa: resolve rebase conflicts

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
(cherry picked from commit 6753e27daab5135973fa45e5a45a4e8e5fbac101)
(cherry picked from commit dfb8f8544283a1dd20c921077231ff14f38aa78a)

Merge pull request #59271 from k0ste/wip-63065-reef

reef: test/cls_lock: expired lock before unlock and start check

Merge pull request #59501 from idryomov/wip-67698-reef

reef: rbd: "rbd bench" always writes the same byte

Merge pull request #59660 from idryomov/wip-67845-reef

reef: librbd/migration: prune snapshot extents in RawFormat::list_snaps()

Merge pull request #60170 from idryomov/wip-63184-reef

reef: librbd/crypto/LoadRequest: clone format for migration source image

Merge pull request #59772 from nbalacha/wip-67763-reef

reef: rbd-mirror: use correct ioctx for namespace

multisite/qa: disable sync error injection

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
(cherry picked from commit da788bc28dbf19bbb54fa91b8c3867063b711551)
(cherry picked from commit cccc43d530d3d4ae5f84c4bb01d3beba9edbd684)

qa/multisite: disable two-zonegroup and three-zone configs.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
(cherry picked from commit fb3186266ee52b352ebdf4ce394d3144a23db8f2)
(cherry picked from commit 7f0ae8e03636cf81811b295f62fdb4a5fddafbf2)

qa/multisite: disable sync policy, endpoint and data_sync_init tests until fixed.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
(cherry picked from commit 79d520698d4dc3f06535d3c8fc7ad119fd52ada1)
(cherry picked from commit 724b3d43af74ab3a23c5de58534dc6266a54d81c)

qa/rgw/multisite: add two-zone configuration

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 14204442ac2eaaaa3f40a76403da49f0f4ed92c1)
(cherry picked from commit 627dd7b86d0c59768625b508b614fa35d008708b)

qa/rgw/multisite: double cluster nodes and osd count

> WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand.

give each cluster 2 nodes instead of 1, and 6 osds instead of 3

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9a80cf4bf473c95c8d0b18fcdce88be20f74f973)
(cherry picked from commit a77c2d42277e44633643bcc1f8969f90724c49ef)

qa/rgw/multisite: reduce polling intervals from 20s to 5s

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit bc084c0ac64477abdbf54d96bc4981edf41e1343)
(cherry picked from commit 32fb2b923bc07b2228b8fe7d1c1f4bc2f3bf263b)

rgw/multisite: reconfigure checkpoint delay to 90s

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
(cherry picked from commit af7c4d83c7f5b61e907a90622c082a373f3995d9)