doc/glossary: add "flapping OSD"

Add an entry for "Flapping OSD" to the glossary.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 1ca89e6ca3d9e0042702b7ac3af9a3e596330c1d)

Merge pull request #60853 from zdover23/wip-doc-2024-11-27-backport-60846-to-reef

reef: doc/radosgw: Improve archive-sync-module.rst

doc/radosgw: Improve archive-sync-module.rst

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>
(cherry picked from commit 303ce155a34fdf2cde63b948d204aca339dd5682)

Merge pull request #58789 from batrick/wip-67121-reef

reef: qa/cephfs: ignore variant of MDS_UP_LESS_THAN_MAX

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58791 from batrick/wip-67154-reef

reef: qa: suppress __trans_list_add valgrind warning

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58805 from lxbsz/wip-67148

reef: cephfs: Fixed a bug in the readdir_cache_cb function that may have us…

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58809 from lxbsz/wip-67118

reef: qa: ignore container checkpoint/restore related selinux denials for c…

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58812 from lxbsz/wip-67114

reef: cephfs-shell: excute cmd 'rmdir_helper' reported error

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58842 from lxbsz/wip-67162

reef: mds: try to choose a new batch head in request_clientup()

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59071 from joscollin/wip-67386-reef

reef: cephfs_mirror: provide metrics for last successful snapshot sync

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59069 from joscollin/wip-67383-reef

reef: qa/suites: drop --show-reachable=yes from fs:valgrind tests

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #60659 from sseshasa/wip-68857-reef

reef: common,osd: Use last valid OSD IOPS value if measured IOPS is unrealistic

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #59264 from k0ste/wip-62781-reef

reef: os/bluestore: fix btree allocator

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #59399 from lxbsz/wip-67434

reef: qa/cephfs: fix TestRenameCommand and unmount the clinet before failin…

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59519 from batrick/wip-67750-reef

reef: mds: cephx path restriction incorrectly rejects snapshots of deleted directory

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59522 from batrick/wip-67807-reef

reef: qa: load all dirfrags before testing altname recovery

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59648 from mchangir/wip-67941-reef

reef: mgr/snap_schedule: correctly fetch mds_max_snaps_per_dir from mds

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59197 from neesingh-rh/wip-67301-reef

reef: pybind/ceph_argparse: Fix error message for ceph tell command

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59397 from lxbsz/wip-67586

reef: client: flush the caps release in filesystem sync

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #60728 from mchangir/wip-68651-reef

reef: log: save/fetch thread name infra

Merge pull request #60825 from zdover23/wip-doc-2024-11-25-backport-60813-to-reef

reef: doc/rados/troubleshooting: Improve log-and-debug.rst

doc/rados/troubleshooting: Improve log-and-debug.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 44c9519cc3eb013104c52014c302ba71ba806f8f)

qa: add test for ceph tell with unknown cephtype

Fixes: https://tracker.ceph.com/issues/59624
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>
(cherry picked from commit decf32e8234a00bd3af7291eac7845008d402717)

pybind/ceph_argparse: fixing error message for ceph tell command

Fixes: https://tracker.ceph.com/issues/59624
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>
(cherry picked from commit 2811bb0dfe3b00ba831e34eb7db50e528d8fbb13)

Merge pull request #59307 from batrick/wip-67477-reef

reef: mon/MonClient: handle ms_handle_fast_authentication return

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59828 from joscollin/wip-68107-reef

reef: qa: do the set/get attribute on the remote filesystem

Merge pull request #60410 from neesingh-rh/wip-68593-reef

reef: cephfs-shell: fixing cephfs-shell test failures

Merge pull request #60178 from joscollin/wip-68415-reef

reef: cephfs_mirror: 'ceph fs snapshot mirror ls' command

Merge pull request #60738 from zdover23/wip-doc-2024-11-15-backport-60703-to-reef

reef: doc/rados: correct "full ratio" note

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #60810 from zdover23/wip-doc-2024-11-24-backport-60804-to-reef-test-2

reef: doc/cephadm: Clarify "Deploying a new Cluster"

doc/cephadm: Clarify "Deploying a new Cluster"

Change the title of the section "Deploying a new Ceph cluster" to "Using
cephadm to Deploy a New Ceph Cluster". This is part of the initiative to
separate package-related documentation from container-based
documenation.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 8cf77d03a1f3ecb73931ddf96d13ef098aa61c1a)

Merge pull request #60789 from zdover23/wip-doc-2024-11-21-os-recommendations-reef

reef: doc/start: separate package and container support tables

doc/start: separate package and container support tables

Separate out container-related information from package-related information and put container-related support information in its own table.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60729 from YiteGu/wip-68935-reef

reef: mgr/dashboard: add cephfs rename REST API

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #60633 from rhcs-dashboard/wip-68845-reef

reef: mgr/dashboard: remove cherrypy_backports.py

doc/rados: correct "full ratio" note

Correct a note that directed users not to add an OSD after the cluster
has reached its "full ratio". The note now says "Do not let your cluster
reach its full ratio before adding an OSD."

Hat tip: Oskar Berggren

Fixes: https://tracker.ceph.com/issues/68900
Co-authored-by: Oskar Berggren <oskar.berggren@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f1a2637c79a15c26a769661dd72ca68d766b2f0d)

Merge pull request #60699 from zdover23/wip-doc-2024-11-12-backport-60529-to-reef

reef: doc/start: separate package chart from container chart

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #60672 from zdover23/wip-doc-2024-11-08-backport-56771-to-reef

reef: doc: Document which options are disabled by mClock.

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

test/common: keeping build happy

Since ceph_pthread_setname() has now been moved from a macro to a
function, we need to reference the common library to link it to.

Signed-off-by: Milind Changire <mchangir@redhat.com>

mds: add or update MDS thread names

To be consistent and sensical.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit a2a989aaa5e9780d98ec9c5cabc66ac2c3a387a4)
Signed-off-by: Milind Changire <mchangir@redhat.com>
Conflicts:
src/mds/MDLog.cc (no upkeep thread in reef)
src/mds/QuiesceAgent.h (not available in reef)
src/mds/QuiesceDbManager.cc (not available in reef)

log: cache recent threads up to a day

In general, a pthread implementation reuses thread ids routinely so the
circular_buffer is there to help us see what the thread name was historically
(capped at 4 entries).

However, to guard against this map from growing without bound, discard entries
that are more than a day old. This would happen if a thread logged an Entry and
has since disappeared for more than a day.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit bfe489ed19d599b67cacb0bdd4958f36661248f2)
Signed-off-by: Milind Changire <mchangir@redhat.com>

common: cache pthread names

This provides common ceph entrypoints for the pthread_[gs]name functions which
will also cache a thread_local copy. This also removes the pthread_t parameter
which precipitated the bug i50743.

Obviously, the overall goal here is to avoid system calls.

Conflicts:
src/ceph_nvmeof_monitor_client.cc (not available in reef)
src/common/ceph_timer.h
src/include/compat.h
src/mds/Beacon.cc
src/rgw/driver/rados/rgw_notify.cc
src/rgw/rgw_lua_background.cc (not available in reef)

See-also: https://tracker.ceph.com/issues/50743
Fixes: 0be8d01c9ddde0d7d24edd34dc75f6cfc861b5ba
Fixes: https://tracker.ceph.com/issues/68691
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit 7de5af59c2741b4203cc17f619ca53096c97b8ff)
Signed-off-by: Milind Changire <mchangir@redhat.com>

mgr/dashboard: add cephfs rename REST API

Introduce rename for the cephfs REST API controller, we can
rename the existing file or directory by it.

Fixes: https://tracker.ceph.com/issues/66797
Signed-off-by: Yite Gu <yitegu0@gmail.com>
(cherry picked from commit bac2689fdd0221a9b77833f20b0c512115f9d9e5)

log: concatenate thread names and print once per thread

Fixes: 0be8d01c9ddde0d7d24edd34dc75f6cfc861b5ba
Fixes: https://tracker.ceph.com/issues/68691
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit 1f40d0c06c4f292d205bc6cc846161e0706a871b)
Signed-off-by: Milind Changire <mchangir@redhat.com>

log: thread name save/fetch infra

* pthread name is saved in a thread_local storage
* the thread_local name is copied into Entry object's ctor
* Log::dump_recent() reads the thread name from the Entry
  object's data member when dumping logs

Fixes: https://tracker.ceph.com/issues/50743
Signed-off-by: Milind Changire <mchangir@redhat.com>
(cherry picked from commit 0be8d01c9ddde0d7d24edd34dc75f6cfc861b5ba)

Merge pull request #60709 from zdover23/wip-doc-2024-11-12-backport-60704-to-reef

reef: doc/start: fix "are are" typo

doc/start: fix "are are" typo

Fix typo reading "They are are single-threaded".
s/are are/are/

Fixes: https://tracker.ceph.com/issues/68901
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 22010719d7edf2a6cf7de7b115ed797e7942ec47)

Merge pull request #60097 from cbodley/wip-68331-reef

reef: rgw/rados: don't rely on IoCtx::get_last_version() for async ops

Merge pull request #59351 from k0ste/wip-63423-reef

reef: rgwfile: fix lock_guard decl

Merge pull request #59346 from k0ste/wip-59607-reef

reef: rgw/sts: changing identity to boost::none, when role policy

Merge pull request #59341 from k0ste/wip-61436-reef

reef: rgw/archive: avoid duplicating objects when syncing from multiple zones

Merge pull request #59261 from k0ste/wip-62748-reef

reef: rgw: add versioning status during `radosgw-admin bucket stats`

Merge pull request #59122 from k0ste/wip-67100-reef

reef: [RGW] Fix the handling of HEAD requests that do not comply with RFC standards

doc/start: separate package chart from container chart

Separate the packages-and-containers chart into two charts:
   (1) a chart that shows which OSes Ceph builds packages for
   (2) a chart that shows which OSes support Ceph's containers

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit e5ff1916e3d5e2cc3f47b4cea2762730dfd335c0)

Merge pull request #56400 from batrick/wip-62923-reef

reef: mon/MonmapMonitor: do not propose on error in prepare_update

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58340 from rishabh-d-dave/fixes-and-missing-backports

reef: qa/cephfs: a bug fix and few missing backport for caps_helper.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58492 from YiteGu/backport-reef

reef: client: check mds down status before getting mds_gid_t from mdsmap

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59620 from pponnuvel/wip-67740-reef

reef: client: Prevent race condition when printing Inode in ll_sync_inode

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58397 from dparmar18/wip-64740-reef

reef: client: do not proceed with I/O if filehandle is invalid

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #60344 from zdover23/wip-doc-2024-10-16-backport-60256-to-reef

reef: doc: update Key Idea in cephfs-mirroring.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #56690 from kotreshhr/wip-64940-reef

reef: qa: Add multifs root_squash testcase

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58111 from neesingh-rh/wip-65348-reef

reef: qa: fixing tests in test_cephfs_shell.TestShellOpts

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58726 from neesingh-rh/wip-67077-reef

reef: mds: fix session/client evict command.

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58744 from lxbsz/wip-58654

reef: common/TrackedOp: do not count the ops marked as nowarn

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58770 from lxbsz/wip-66778

reef: mds/client: return -ENODATA when xattr doesn't exist for removexattr

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59054 from vshankar/wip-cleanup-centos8

reef: qa/cephfs: remove dependency on centos8/rhel8 entirely

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #60680 from zdover23/wip-doc-2024-11-09-backport-60605-to-reef

reef: doc/cephadm: clean "Adv. OSD Service Specs"

doc/cephadm: clean "Adv. OSD Service Specs"

Clean the English grammar and the .RST in the section "Advanced OSD
Service Specifications" in the file doc/cephadm/services/osd.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f6b56f9ee5d9f0b54dcaf1832d7a977895eb9c94)

doc: Document which options are disabled by mClock.

Not only in the mClock docs, but also in the reference
of the options that are disabled.

Otherwise users are bound to miss it, and surprised
why their options are ignored or reset.

Signed-off-by: Niklas Hambüchen <mail@nh2.me>
(cherry picked from commit 790afd277730f24801a3b59c08d2dce0f2a9659f)

common,osd: Use last valid OSD IOPS value if measured IOPS is unrealistic

The OSD's IOPS capacity is used by the mClock scheduler to determine the
quantum of bandwidth allocation for the various operations on the OSD.
Prior to this commit, maybe_override_max_osd_capacity_for_qos() only
checked if the measured IOPS capacity exceeded the higher threshold defined
by 'osd_mclock_iops_capacity_threshold_[hdd|ssd]' and if so fallback to the
last valid or the default IOPS capacity as defined by
osd_mclock_max_capacity_iops_[hdd|ssd].

It's quite possible that the reported IOPS is unrealistically low. This
could be due to transient factors on the underlying device or it could
indicate bad health of the device. Either way, the safer option would be
to fallback to the last valid or the default IOPS setting for that OSD in
order to avoid cluster performance (slow or stalled ops) issues down the
line.

Therefore, to handle this case, the commit introduces additional config
options viz.,
 - osd_mclock_iops_capacity_low_threshold_hdd - set to 50 IOPS and
 - osd_mclock_iops_capacity_low_threshold_ssd - set to 1000 IOPS

If the measured IOPS capacity doesn't fall within the low and high
threshold range, the default or the last valid IOPS capacity is used.
The existing cluster log warning is suitably modified to convey the
reason.

Additionally, for a couple of valgrind related teuthology tests, the
cluster warning is added to the ignorelist since the reported IOPS can
be very low due to slowness.

Fixes: https://tracker.ceph.com/issues/67421
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
(cherry picked from commit da4b85c55a15f49b241f3fc44dda2263b42dc637)

Merge pull request #60645 from zdover23/wip-doc-2024-11-07-backport-60643-to-reef

reef: doc/cephadm: link to "host pattern" matching sect

doc/cephadm: link to "host pattern" matching sect

Link to the "Placement by Pattern Matching" section in
doc/cephadm/services/index.rst from the "Advanced OSD Service
Specifications" section in doc/cephadm/services/osd.rst.

Fixes: https://tracker.ceph.com/issues/68812
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit cdfa299a2511ee35dfe0a857702af6d3ba257efd)

Merge pull request #60638 from zdover23/wip-doc-2024-11-06-backport-56599-to-reef

reef: doc/cephadm: Update operations.rst

mgr/dashboard: remove cherrypy_backports.py

since its mostly used only for older cherrypy versions which we don't
support anymore in any of our recent upstream releases, we could remove
it completely

Fixes: https://tracker.ceph.com/issues/68802
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 7ae56f4f561999898a7411840f94e742981ca6bc)

 Conflicts:
src/pybind/mgr/prometheus/module.py
 - accepted incoming changes which removes the monkey-patch

doc/cephadm: Update operations.rst

Fixed typo 'no' to 'not' in sentence "If this test fails, cephadm will
no be able to manage services on that host."

Signed-off-by: rhkelson <119636079+rhkelson@users.noreply.github.com>
(cherry picked from commit 34bc35a73443fa56908395a83ea2dd404a8fc4a6)

Merge pull request #60418 from cbodley/wip-63024-reef

reef: qa/rgw/sts: keycloak task installs java manually

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #59104 from YiteGu/wip-53048-reef

reef: os/bluestore: add some slow count for bluestore

Merge pull request #59048 from YiteGu/wip-some-blk-discard-path

reef: a series of optimizations for kerneldevice discard

Merge pull request #59043 from NitzanMordhai/wip-67351-reef

reef: Test: osd-recovery-space.sh extends the wait time for "recovery toofull"

Merge pull request #59629 from sseshasa/wip-66422-reef

reef: osd/scrub: Change scrub cost to average object size

Merge pull request #56475 from amathuria/wip-65141-reef

reef: osd: Change PG Deletion cost for mClock

Merge pull request #60594 from zdover23/wip-doc-2024-11-02-backport-60584-to-reef

reef: doc: s/Whereas,/Although/

doc: s/Whereas,/Although/

Change the subordinating conjunction "Whereas" followed by a comma to
the less grammatically-incorrect "Although". I've been meaning to do
this since 22 Mar 2023.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ea1a470fb4809d312f854e72fc5f8081cbcb2c49)

Merge pull request #58802 from ljflores/wip-67129-reef

reef: osd: bump versions of decoders for upmap-primary

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #58741 from ifed01/wip-ifed-async-compact-reef

reef: os/bluestore: enable async manual compactions

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #58734 from pereman2/wip-67082-reef

reef: tools/objectstore: check for wrong coll open_collection

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58413 from ljflores/wip-66826-reef

reef: qa/suites/rados/thrash/workloads: remove cache tiering workload

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #58315 from cbodley/wip-66723

reef: build: Make boost_url a list

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #57588 from NitzanMordhai/wip-66137-reef

reef: osd: full-object read CRC mismatch due to 'truncate' modifying oi.size w/o clearing 'data_digest'

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #58283 from pponnuvel/wip-66699-reef

reef: mgr/nfs: Don't crash ceph-mgr if NFS clusters are unavailable.

Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #57615 from sajibreadd/wip-66148-reef

reef: osd: fix for segmentation fault on OSD fast shutdown

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #57456 from badone/wip-tracker-59196-lazy-omap-regex-segfault-reef

reef: test/lazy-omap-stats: Convert to boost::regex

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #57241 from sajibreadd/wip-65356-reef

reef: os/bluestore: expand BlueFS log if available space is insufficient

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #55774 from ifed01/wip-ifed-no-death-tests-reef

reef: test/store_test: get rid off assert_death.

Reviewed-by: Md Mahamudur Rahaman Sajib <mahamudur.sajib@croit.io>

Merge pull request #60538 from zdover23/wip-doc-2024-10-29-backport-60508-to-reef

reef: doc/rados: add blaum_roth coding guidance

Merge pull request #60526 from zdover23/wip-doc-2024-10-29-backport-60470-to-reef

reef: doc: explain the consequence of enabling mirroring through monitor co…

doc/rados: add blaum_roth coding guidance

Direct Ceph administrators using blaum_roth coding for erasure-coded
pools to change the default value of w=7 to a different value in order
to ensure that w+1 is prime.

This information was provided to the Ceph upstream by Benjamin Mare in
September of 2024.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 6648d94aed5ff5643f50fcd4f3f07d97b846e885)

doc: explain the consequence of enabling mirroring through monitor commands

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit ed3ce72fbc0b097131945cbb39caa0831401b94a)

Merge pull request #60501 from zdover23/wip-doc-2024-10-26-backport-60444-to-reef

reef: doc/rados: standardize markup of "clean"

Merge pull request #60504 from zdover23/wip-doc-2024-10-26-backport-60434-to-reef

reef: doc/glossary: add "PLP" to glossary