mgr/dashboard: replace individual cluster's capacity info with Usage bar in Multi-Cluster

fixes: https://tracker.ceph.com/issues/67193

Signed-off-by: Naman Munet <nmunet@redhat.com>

Merge pull request #58410 from bill-scales/erasure_coding_design_document

doc: doc/dev/osd_interals/erasure_coding/enhancements.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Sam Just <sjust@redhat.com>

Merge pull request #59177 from anthonyeleven/aad-qat-followup

doc/radosgw: improve qat-accel.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/radosgw: improve qat-accel.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge PR #58532 into main

* refs/pull/58532/head:
qa: improve and add commit ID to comment for context

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #55050 from rhcs-dashboard/nfs-monitoring

mgr/nfs: scrape nfs monitoring endpoint

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #54075 from aclamk/wip-aclamk-bs-nice-print

os/bluestore: Recompression, part 1. Nice debugs.

Merge pull request #59164 from toficly/patch-2

doc:update e-mail addresses governance

doc:update e-mail addresses governance

with the consent of my colleagues
- Dan van der Ster
- João Eduardo Luis
- Mark Nelson

changed their e-mail addresses to the current one

Signed-off-by: Tobias Fischer <tobias.fischer@clyso.com>
Signed-off-by: Tobias Fischer <138117964+toficly@users.noreply.github.com>

Merge pull request #59098 from athanatos/sjust/wip-67415-alienstore-mkfs-crash

crimson: clear AlienStore::coll_map in umount rather than in stop

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59077 from zdover23/wip-doc-2024-08-07-cephfs-cache-configuration-cache-pressure

doc/cephfs: add cache pressure information

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58951 from rhcs-dashboard/ceph-users-doc-fix

mgr/dashboard: fix ceph-users api

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58976 from athanatos/sjust/for-review/wip-nvmeofpaxosservice-cosmetic-fixes

mon: cosmetic fixes for NVMeofGwMon files

Reviewed-by: Alexander Indenbaum <aindenba@redhat.com>
Reviewed-by: Leonid Chernin <lechernin@gmail.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Aviv Caro <Aviv.Caro@ibm.com>

Merge pull request #57003 from linuxkidd/wip-pool-max-avail

PGMap: remove pool max_avail scale factor

Merge pull request #58573 from idryomov/wip-selinux-drop-ignore-agetty

qa: agetty entry in selinux allowlist is redundant now

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #58784 into main

* refs/pull/58784/head:
script/backport-create-issue: add debug messages for retry

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #58605 from adk3798/upgrade-suite-upgrade-in-progress-checks

qa/upgrade: fix checks to make sure upgrade is still in progress

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: fix ceph-users api doc

Fixes: https://tracker.ceph.com/issues/67224
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/nfs: scrape nfs monitoring endpoint

Fixes: https://tracker.ceph.com/issues/62558
Signed-off-by: avanthakkar <avanjohn@gmail.com>

doc/cephfs: add cache pressure information

Add information to doc/cephfs/cache-configuration.rst about how to deal
with a message that reads "clients failing to respond to cache
pressure". This procedure explains how to slow the growth of the
recall_caps value so that it does not exceed the
mds_recall_warning_threshold.

The information in this commit was developed by Eugen Block. See
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/5ROH5CWKKOEIQMVXOVRT5OO7CWK2HPM3/#J65DFUPP4BY57MICPANXKI7KAXSZ5Z5P
and https://www.spinics.net/lists/ceph-users/msg73188.html.

Fixes: https://tracker.ceph.com/issues/57115
Co-authored-by: Eugen Block <eblock@nde.ag>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59087 from kamoltat/wip-ksirivad-fix-stretch-device-class-docs

docs/rados/operations/stretch-mode: warn device class is not supported

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58736 from amathuria/wip-66922-amat

qa/rados/dashboard: Add PG_DEGRADED to ignorelist

Merge pull request #57037 from adk3798/cephadm-limit-shell-mounts

cephadm: limit mounts for shell and ceph-volume commands

Merge PR #58861 into main

* refs/pull/58861/head:
mds: only authpin on wrlock when not a locallock

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

PGMap: remove pool max_avail scale factor

The scaling of max_avail by the ratio of non-degraded to total objects
count results in the reported max_avail increasing proportionally to the
number of OSDs marked `down` but not `out`.  This is counter intuitive
since OSDs going `down` should never result in more space being
available.

Removing the scale factor allows max_avail to remain unchanged until the
OSDs are marked `out`.

Signed-off-by: Michael J. Kidd <linuxkidd@gmail.com>

Merge pull request #59091 from zdover23/wip-doc-2024-08-08-README-md-alerts

doc/README.md - add "tip" alert styling

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #59030 from ljflores/wip-fix-crontab-2

qa/crontab: force priority for main rados nightlies

Merge pull request #58583 from lxbsz/wip-66946

qa/cephfs: fix TestRenameCommand and unmount the clinet before failin…

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: doc/dev/osd_internals/erasure_coding/enhancements.rst

Erasure coding enhancements design document

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

doc/README.md - add "tip" alert styling

Add "tip" alert styling (what in Docbook XML is called "an admonition")
to information about Ninja in an ordered list (which is what markdown
has here instead of procedures).

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59049 from zdover23/wip-doc-2024-08-06-README-building-ceph

doc/README.md - allocate information correctly

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58913 from cyx1231st/wip-seastore-unlock-ool

crimson/os/seastore: move ool writes from collection lock to concurrent DeviceSubmission phase

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #58787 from xxhdx1985126/wip-67106

crimson/os/seastore: consider segment_header_t::modify_time as the segments' modify_time for no-tail OOL segments

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57911 from gardran/wip-gdran-mds-better-clease-handling

mds: some optimizations around client Capability and Lease tracking

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/.../alien_store: clear coll_map in umount rather than stop

As of a926a4a54ced, BlueStore::umount expects outstanding CollectionRef's
to already have been released.

Previously, 8389471b3 updated alien_store.cc to release coll_map in
stop() in in the alien tp.

Fixes: https://tracker.ceph.com/issues/67415
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #58983 from cyx1231st/wip-seastore-cleanup-cache

crimson/os/seastore/cache: cleanups and comments

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #58869 from xxhdx1985126/wip-seastore-lba-leaf-node-minor-bug-fix

crimson/os/seastore/lba_manager: correct the range end of lba mappings

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #59029 into main

* refs/pull/59029/head:
qa: simplify postmerge construction

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

crimson/osd/osd.cc: convert fmt::print to INFO in OSD::mkfs

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd.cc: convert OSD::mkfs to a coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #44366 from orozery/rbd-crypto-migration

librbd/crypto: fix issue when live-migrating from encrypted export

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58002 from nbalacha/wip-nbalacha-sorted-snaps

cls/rbd: add group_snap_list_order method to enable sorting snapshots in creation order

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

docs/rados/operations/stretch-mode: warn device class is not supported

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #59073 from anoopcs9/cephadm-docs-smb-spec

doc/cephadm/smb: Add missing user_sources option to service spec

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56233 from kamoltat/wip-ksirivad-fix-64802

RADOS: Generalize stretch mode pg temp handling to be usable without stretch mode
Samuel Just <sjust@redhat.com>

Merge pull request #57381 from kamoltat/wip-ksirivad-fix-65371

PeeringState.cc: Only populate want_acting when num_osds < bucket_max
Reviewed by: Samuel Just <sjust@redhat.com>

Merge pull request #58585 from anoopcs9/cephadm-update-smbd-version

cephadm/smb: Determine samba version within container

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52489 from ifed01/wip-ifed-alloc2

os/bluestore: introduce hybrid_btree2 allocator

os/bluestore: Fix debug after rebase

BufferSpace moved from Blob to Onode.
Moved relevant code from Blob::operator<< to Onode::operator<<.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add ability to partially print Onode

Now Onode can be printed in selected range.
It is useful in high-level dout modes that operate on a fragment of
entire Onode.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Move OBJECT_MAX_SIZE

From define in .cc to constexpr in .h

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Blob, Onode debug printout

Small improvement on debug output.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: small improvement for printers

- moved operator<< to BlueStore_debug file
- upcased Printer {} flags
- more reliable heap begin detection
- fixup after rebase

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add improved printer for Onode

Added nice replacement for dump_onode function.
Introduce printer class that allows to select parts of Onode that are to be printed.
It severly reduced amount of clutter in output.
Usage:
using P = Bluestore::printer;
dout << blob->print(P::ptr + P::sdisk + P::schk + P::buf + P::attrs);

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add printing buffers to Blob

Now printing Blob can include buffers.
There are 2 variants:
- 'buf' same as original in dump_onode
- 'sbuf' only fundamental params, no ptr etc.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add improved printer for Extent

Modify Extent similar to Blob, so that one can use improved Blob printing
when printing extents.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add improved printer for Blob

Introduce printer class that allows to select parts of Blob that are to be printed.
It severly reduced amount of clutter in output.
Usage:
using P = Bluestore::Blob::printer;
dout << blob->printer(P::ptr + P::sdisk + P::schk);

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

librbd/migration: make ImageDispatch handle encryption for non-native formats

With NativeFormat now being handled via dispatch, handling encryption
for non-native formats (i.e. mapping to raw image extents and performing
decryption/mapping back on completion) in the migration layer is really
straightforward.

Note that alignment doesn't need to be performed in the migration layer
because it happens on the destination image -- the "align and resubmit"
logic in C_UnalignedObjectReadRequest should kick in before the call to
read_parent().

Fixes: https://tracker.ceph.com/issues/53674
Co-authored-by: Or Ozeri <oro@il.ibm.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: don't make an extra copy of image_extents in C_ImageReadRequest ctor

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/workunits/rbd: perform cleanup in test_clone_encryption()

... so that RAW_DEV can be unmapped and future tests can reuse testimg
and other image names without bumping into watchers and older snapshots.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/workunits/rbd: no need to chmod in luks-encryption.sh

Most workunits expect the user to be a member of "disk" group, so we
can pretty much rely on that being the case at this point.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58907 from adam-lj/rbd-throttle-perf

librbd: Reduce use of atomics in librbd throttling

Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58993 from pereman2/unused-func

os/bluestore: Unused func

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #58693 from xxhdx1985126/wip-crimson-clear-temp-objects-startup

crimson/osd: clear ondisk temp objects on startup

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57562 from perezjosibm/wip-perezjos-cpu-map

script/: add cpu-map.sh to aid manual selection of CPU cores for threads for profiling

Merge pull request #55374 from aclamk/wip-aclamk-bs-multi-label

os/bluestore: Multiple bdev labels on main block device

Merge PR #59040 into main

* refs/pull/59040/head:
doc: documenting the feature that scrub clear the entries from damage table on repair

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

qa/cephfs: fix TestRenameCommand and unmount the clinet before failing fs

We should else bring and wait for MDS to be up since it is needed
for unmounting of CephFS in CephFSTestCase.tearDown() to be successful,
or just unmount the mountpoints before failing the filesystem.

Since the mountpoint won't be used in later tests so we just unmount
it.

Fixes: https://tracker.ceph.com/issues/66946
Signed-off-by: Xiubo Li <xiubli@redhat.com>

doc/cephadm/smb: Add missing user_sources option to service spec

Signed-off-by: Anoop C S <anoopcs@cryptolab.net>

crimson/os/seastore: report outstanding requests per phase

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: wait ool writes in DeviceSubmission phase

So that it is moved out of the collection lock.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #56782 from joscollin/wip-B65171-support-replication-start-end-notifications

cephfs_mirror: provide metrics for last successful snapshot sync

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57458 from lxbsz/wip-session-evict

qa/cephfs: add test_session_evict_non_blocklisted test case

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57987 from joscollin/wip-B64752-valgrind-report-leaks

qa/suites: drop --show-reachable=yes from fs:valgrind tests

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58097 from gardran/wip-gardran-kill-count

client: eliminate duplicate lookups wherever possible

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: documenting the feature that scrub clear the entries from damage table on repair

Fixes: https://tracker.ceph.com/issues/67337
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

doc/README.md - allocate information correctly

Add information in the "Building Ceph" procedure to the steps where it
belongs. Add information about "do_cmake.sh" to the step about
"do_cmake.sh". Add information about Ninja to the step about Ninja.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59060 from qn2060/patch-3

doc/dev/peering: Change acting set num

Merge PR #52939 into main

* refs/pull/52939/head:
mon/MonClient: handle ms_handle_fast_authentication return

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #58437 into main

* refs/pull/58437/head:
mds: remove the misleading and redundant logs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #58582 into main

* refs/pull/58582/head:
test: add a test for `fs set down` on online cluster
mon: fix `fs set down` to adjust max_mds only when cluster is not down

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58511 from pereman2/ch-umount

os/bluestore: assert CollectionRef count is 1 on umount

Merge pull request #57572 from pereman2/refactors

os/bluestore: Improve Readability of Extent Map Sharding

Merge pull request #59011 from xxhdx1985126/wip-67327

crimson/osd: send empty transactions to backfill targets that haven't backfilled the objects yet

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58708 from xxhdx1985126/wip-67068

crimson/osd/osd_operation/client_request: requeue client requests from a temporary queue other than ClientRequest::Orderer::list

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58148 from xxhdx1985126/wip-65696

crimson/osd/osd_operations: hang requests if the objects are unfound

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59051 from dang/wip-dang-67321

RGW - Zipper: pull in upstream fix for zpp_bits

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/dev/peering: Change acting set num

This commit updates the peering doc page with the changed [2,1,2] to [3,1,2]

Signed-off-by: qn2060 <qn2060@users.noreply.github.com>

Merge pull request #58971 from cbodley/wip-rm-boost-redis-again-again

submodule: remove the boost_redis submodule again (again)

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #58037 from Suyashd999/refractor-PrimaryLogPG

osd/PrimaryLogPG.cc bl length would be stored before moving

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

RGW - Zipper: pull in upstream fix for zpp_bits

zpp_bits included code that required a newere compiler.  Pull in the
upstream fix from:

https://github.com/eyalz800/zpp_bits/pull/170

Fixes: https://tracker.ceph.com/issues/67321
Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

Merge pull request #59024 from zdover23/wip-doc-2024-08-05-README-building-ceph

doc/README.md - add ordered list

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58282 from NitzanMordhai/wip-nitzan-daemonwatchdog-should-terminate-thrasher-when-bark

qa/tasks: watchdog should terminate thrasher

Merge pull request #58664 from aclamk/wip-aclamk-qa-less-bluestore-debug

qa/suites/rados: Reduced BlueStore log levels

script/: add cpu-map.sh to aid manual selection of CPU cores for threads for profiling

Signed-off-by: Jose J Palacios-Perez <perezjos@uk.ibm.com>

thrashers: standardize stop and join method names

Thrashers that do not inherit from ThrasherGreenlet previously used a
method called do_join, which combined stop and join functionality. To
ensure consistency and clarity, we want all thrashers to use separate
stop, join, and stop_and_join methods.

This commit renames methods and implements missing stop and stop_and_join
methods in thrashers that did not inherit from ThrasherGreenlet.

Fixes: https://tracker.ceph.com/issues/66698
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

qa/tasks: watchdog terminate thrasher

If a thrasher exception occurs, the do_dump_ops thread will continue
looping until the Teuthology timeout is reached.
The watchdog should terminate the thrasher to free up resources.

Fixes: https://tracker.ceph.com/issues/66698
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge PR #58500 into main

* refs/pull/58500/head:
ptl-tool: Fix token file name and variable names

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58963 from rhcs-dashboard/fix-67009-main

mgr/dashboard: Dashboard not showing Object/Overview correctly.

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #58746 from rhcs-dashboard/fix-67069-main

mgr/dashboard: make multi-cluster context switcher scrollable if we have many entries in it

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>