os/bluestore: Fix debug after rebase

BufferSpace moved from Blob to Onode.
Moved relevant code from Blob::operator<< to Onode::operator<<.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add ability to partially print Onode

Now Onode can be printed in selected range.
It is useful in high-level dout modes that operate on a fragment of
entire Onode.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Move OBJECT_MAX_SIZE

From define in .cc to constexpr in .h

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Blob, Onode debug printout

Small improvement on debug output.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: small improvement for printers

- moved operator<< to BlueStore_debug file
- upcased Printer {} flags
- more reliable heap begin detection
- fixup after rebase

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add improved printer for Onode

Added nice replacement for dump_onode function.
Introduce printer class that allows to select parts of Onode that are to be printed.
It severly reduced amount of clutter in output.
Usage:
using P = Bluestore::printer;
dout << blob->print(P::ptr + P::sdisk + P::schk + P::buf + P::attrs);

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add printing buffers to Blob

Now printing Blob can include buffers.
There are 2 variants:
- 'buf' same as original in dump_onode
- 'sbuf' only fundamental params, no ptr etc.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add improved printer for Extent

Modify Extent similar to Blob, so that one can use improved Blob printing
when printing extents.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add improved printer for Blob

Introduce printer class that allows to select parts of Blob that are to be printed.
It severly reduced amount of clutter in output.
Usage:
using P = Bluestore::Blob::printer;
dout << blob->printer(P::ptr + P::sdisk + P::schk);

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #58907 from adam-lj/rbd-throttle-perf

librbd: Reduce use of atomics in librbd throttling

Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58993 from pereman2/unused-func

os/bluestore: Unused func

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #58693 from xxhdx1985126/wip-crimson-clear-temp-objects-startup

crimson/osd: clear ondisk temp objects on startup

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57562 from perezjosibm/wip-perezjos-cpu-map

script/: add cpu-map.sh to aid manual selection of CPU cores for threads for profiling

Merge pull request #55374 from aclamk/wip-aclamk-bs-multi-label

os/bluestore: Multiple bdev labels on main block device

Merge PR #59040 into main

* refs/pull/59040/head:
doc: documenting the feature that scrub clear the entries from damage table on repair

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #56782 from joscollin/wip-B65171-support-replication-start-end-notifications

cephfs_mirror: provide metrics for last successful snapshot sync

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57458 from lxbsz/wip-session-evict

qa/cephfs: add test_session_evict_non_blocklisted test case

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57987 from joscollin/wip-B64752-valgrind-report-leaks

qa/suites: drop --show-reachable=yes from fs:valgrind tests

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58097 from gardran/wip-gardran-kill-count

client: eliminate duplicate lookups wherever possible

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: documenting the feature that scrub clear the entries from damage table on repair

Fixes: https://tracker.ceph.com/issues/67337
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #59060 from qn2060/patch-3

doc/dev/peering: Change acting set num

Merge PR #52939 into main

* refs/pull/52939/head:
mon/MonClient: handle ms_handle_fast_authentication return

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #58437 into main

* refs/pull/58437/head:
mds: remove the misleading and redundant logs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #58582 into main

* refs/pull/58582/head:
test: add a test for `fs set down` on online cluster
mon: fix `fs set down` to adjust max_mds only when cluster is not down

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58511 from pereman2/ch-umount

os/bluestore: assert CollectionRef count is 1 on umount

Merge pull request #57572 from pereman2/refactors

os/bluestore: Improve Readability of Extent Map Sharding

Merge pull request #59011 from xxhdx1985126/wip-67327

crimson/osd: send empty transactions to backfill targets that haven't backfilled the objects yet

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58708 from xxhdx1985126/wip-67068

crimson/osd/osd_operation/client_request: requeue client requests from a temporary queue other than ClientRequest::Orderer::list

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58148 from xxhdx1985126/wip-65696

crimson/osd/osd_operations: hang requests if the objects are unfound

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59051 from dang/wip-dang-67321

RGW - Zipper: pull in upstream fix for zpp_bits

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/dev/peering: Change acting set num

This commit updates the peering doc page with the changed [2,1,2] to [3,1,2]

Signed-off-by: qn2060 <qn2060@users.noreply.github.com>

Merge pull request #58971 from cbodley/wip-rm-boost-redis-again-again

submodule: remove the boost_redis submodule again (again)

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #58037 from Suyashd999/refractor-PrimaryLogPG

osd/PrimaryLogPG.cc bl length would be stored before moving

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

RGW - Zipper: pull in upstream fix for zpp_bits

zpp_bits included code that required a newere compiler.  Pull in the
upstream fix from:

https://github.com/eyalz800/zpp_bits/pull/170

Fixes: https://tracker.ceph.com/issues/67321
Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

Merge pull request #59024 from zdover23/wip-doc-2024-08-05-README-building-ceph

doc/README.md - add ordered list

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58282 from NitzanMordhai/wip-nitzan-daemonwatchdog-should-terminate-thrasher-when-bark

qa/tasks: watchdog should terminate thrasher

Merge pull request #58664 from aclamk/wip-aclamk-qa-less-bluestore-debug

qa/suites/rados: Reduced BlueStore log levels

script/: add cpu-map.sh to aid manual selection of CPU cores for threads for profiling

Signed-off-by: Jose J Palacios-Perez <perezjos@uk.ibm.com>

thrashers: standardize stop and join method names

Thrashers that do not inherit from ThrasherGreenlet previously used a
method called do_join, which combined stop and join functionality. To
ensure consistency and clarity, we want all thrashers to use separate
stop, join, and stop_and_join methods.

This commit renames methods and implements missing stop and stop_and_join
methods in thrashers that did not inherit from ThrasherGreenlet.

Fixes: https://tracker.ceph.com/issues/66698
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

qa/tasks: watchdog terminate thrasher

If a thrasher exception occurs, the do_dump_ops thread will continue
looping until the Teuthology timeout is reached.
The watchdog should terminate the thrasher to free up resources.

Fixes: https://tracker.ceph.com/issues/66698
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge PR #58500 into main

* refs/pull/58500/head:
ptl-tool: Fix token file name and variable names

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58963 from rhcs-dashboard/fix-67009-main

mgr/dashboard: Dashboard not showing Object/Overview correctly.

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #58746 from rhcs-dashboard/fix-67069-main

mgr/dashboard: make multi-cluster context switcher scrollable if we have many entries in it

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #57905 from mcpreston/docs-dashboard-typo

doc/mgr/dashboard: fix TLS typo

Merge pull request #58767 from kshtsk/wip-python-invalid-escape-sequence

mgr: fix invalid escape sequence

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58373 from ceph/wip-tasks-ceph-mkfs-has-no-f

qa: ceph.py mkfs.ext4 has no -f

os/bluestore: BluestoreRepairTest ch cleanup

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

doc/README.md - add ordered list

Add ordered-list formatting to the "Building Ceph" section of README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58891 from zdover23/wip-doc-2024-07-28-cephfs-file-layouts-layout-fields

doc/cephfs: edit "Layout Fields" text

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58115 from ronen-fr/wip-rf-memmodel

common: MemoryModel: performance improvements and API changes

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

librbd: Reduce use of atomics in librbd throttling

Signed-off-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>

Merge pull request #57786 from Svelar/asan_cli_test

crush: avoid out-of-bound access and simplify enlarging buckets

Merge pull request #58718 from wanglinke521/wlk-kv-stats

kv/RocksDBStore: add kv_stats function to print all CF info

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #58470 from mohit84/issue_66867

mon: Remove any pg_upmap_primary mapping during remove a pool

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Josh Salomon <josh.salomon@gmail.com>

Merge pull request #58302 from jianwei1216/feat_add_mclock_perfcounter_main

osd/scheduler: add mclock queue length perfcounter

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

crimson/osd: send empty transactions to backfill targets that haven't
backfilled the objects yet

Fixes: https://tracker.ceph.com/issues/67327
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #59006 from zdover23/wip-doc-2024-08-03-glossary-ceph-ansible

doc/glossary: add "ceph-ansible"

doc/glossary: add "ceph-ansible"

Add a "ceph-ansible" entry to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58969 from gbregman/main

mgr/cephadm: Add SPDK log level to nvmeof configuration file

Merge pull request #58402 from rkachach/fix_issue_mtls_support

mgr/cephadm: adding mTLS for ceph mgmt-gateway and backend services communication

Reviewed-by: Adam King <adking@redhat.com>

mgr/cephadm: add SPDK log level to nvmeof configuration
Fixes https://tracker.ceph.com/issues/67258

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

python-common/ceph/deployment: add SPDK log level to nvmeof configuration
Fixes https://tracker.ceph.com/issues/67258

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #58883 from idryomov/wip-backport-create-resourceattrerror

script/backport-create-issue: handle ResourceAttrError when getting CF_TAGS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58882 from idryomov/wip-native-format-dispatch

librbd/migration: don't instantiate NativeFormat, handle it via dispatch

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: N Balachandran <nibalach@redhat.com>

Merge pull request #58862 from cbodley/wip-rgw-posix-bucket-cache-lock

rgw/posix: name the lock_guard in BucketCacheEntry::reclaim()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

os/bluestore: remove unused functions

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #58765 from yuvalif/wip-yuval-64184

rgw/async/notifications: use common async waiter in pubsub push

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58672 from YiteGu/code-cleanup-osdmon

mon/osdmonitor: cleanup the code for preprocess_mark_me_dead

Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #58635 from badone/wip-tracker-50371-rados_api_test-timeout-failures

qa: Restrict rados api tests to large clusters and increase timeout

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #58631 from cbodley/wip-66336

msg: insert PriorityDispatchers in sorted position

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57863 from NitzanMordhai/wip-nitzan-thrash-erasure-code-crush-4-nodes-8-6-overrides

suites/ec-rados-plugin=jerasure-k=8-m=6-crush: roles set with overrides

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #58747 from AliMasarweh/wip-alimasa-bn-test-fix-http

RGW|BN: fix lifecycle test issue

Reviewed-by: yuvalif<ylifshit@redhat.com>

mgr/cephadm: fixing Grafana domain handling

it seems that with Grafana 10.4.0 the domain parameter is taken into
account while building the final url (earlier versions didn't seem to
behave the same way). This change sets the domain to the hostname where
Grafana daemon is running instead of '*.lab'. serve_from_sub_path is
removed as it's no needed and when add it causes some undesirable
redirections that could break monitoring HA.

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/dashboard: Dashboard not showing Object/Overview correctly.

Add undefined checks wherever necessary to the updateChartData method in
dashboard-area-chart component

Fixes: https://tracker.ceph.com/issues/67009
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge PR #51332 into main

* refs/pull/51332/head:
qa: add test for ceph tell with unknown cephtype
pybind/ceph_argparse: fixing error message for ceph tell command

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: make multi-cluster context switcher scrollable if we have many entries in it

Fixes: https://tracker.ceph.com/issues/67069
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

doc/cephfs: edit "Layout Fields" text

Improve the English in the "Layout Fields" section of
doc/cephfs/file-layouts.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58899 from zdover23/wip-doc-2024-07-29-glossary-prometheus

doc/glossary: add "Prometheus"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58972 from anthonyeleven/glacier-melting

doc: Improve doc/radosgw/placement.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc: Improve doc/radosgw/placement.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

submodule: remove the boost_redis submodule again (again)

5843c6b04bacf9a7c981fca5d874ab3400f855db accidentally added back to the
boost_redis submodule that was last removed in 924f2e9f87f0ee940eff363c26a714a309639d26

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/cephadm: add SSL support to ceph-exporter

This commit adds SSL support to the ceph-exporter deployment
made by cephadm. When `secure_monitoring_stack` is set to `True`,
the `ceph-exporter` container is restarted with SSL enabled.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding mTLS support

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #56792 from cbodley/wip-qa-rgw-notifications-valgrind

qa/rgw: enable valgrind in rgw/notifications suite

Merge pull request #54671 from baum/ceph-nvmeof-mon

mon: add NVMe-oF gateway monitor and HA

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #58931 from ronen-fr/wip-rf-repair-decode

qa/standalone/scrub: fix the searched-for text for snaps decode errors

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #56976 from YiteGu/add-watcher-timeout-discover

osd: add watch ping timeout count in osd

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #55130 from ifed01/wip-ifed-better-shared-blob-repair

test/store_test: get rid off explicit offset specifications in shared

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #52068 from YiteGu/fix-trackedop-perfcounter

common/TrackedOp: rename and raise prio of slow op perfcounter

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #58349 from dang/wip-dang-posix-driver

RGW - Add Multipart and Versioned POSIX

Merge PR #58878 into main

* refs/pull/58878/head:
client: Fix leading / issue with mds_check_access
qa: Add mds caps test for testing fs read and a path rw

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

nvmeof gw monitor: disable by default

Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

qa/suites/rbd/nvmeof: add multi-subsystem setup and thrash test

1. qa/tasks/nvmeof.py:
    1.1. create multiple rbd images for all subsystems
    1.2. add NvmeofThrasher and ThrashTest
2. qa/tasks/mon_thrash.py: add 'switch_thrashers' option
3. nvmeof_setup_subsystem.sh: create multiple subsystems and enable HA
4. Restructure qa/suites/rbd/nvmeof: Create two sub-suites
   - "basic" (nvmeof_initiator job)
   - "thrash" (new: nvmeof_mon_thrash and nvmeof_thrash jobs)

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

mgr/cephadm: ceph nvmeof monitor support

Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

mon: add NVMe-oF gateway monitor and HA doc

Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

mon: add NVMe-oF gateway monitor and HA

- gateway submodule

Fixes: https://tracker.ceph.com/issues/64777
This PR adds high availability support for the nvmeof Ceph service. High availability means that even in the case that a certain GW is down, there will be another available path for the initiator to be able to continue the IO through another GW. High availability is achieved by running nvmeof service consisting of at least 2 nvmeof GWs in the Ceph cluster. Every GW will be seen by the host (initiator) as a separate path to the nvme namespaces (volumes).

The implementation consists of the following main modules:

- NVMeofGWMon - a PaxosService. It is a monitor that tracks the status of the nvmeof running services, and take actions in case that services fail, and in case services restored.
- NVMeofGwMonitorClient – It is an agent that is running as a part of each nvmeof GW. It is sending beacons to the monitor to signal that the GW is alive. As a part of the beacon, the client also sends information about the service. This information is used by the monitor to take decisions and perform some operations.
- MNVMeofGwBeacon – It is a structure used by the client and the monitor to send/recv the beacons.
- MNVMeofGwMap – The map is tracking the nvmeof GWs status. It also defines what should be the new role of every GW. So in the events of GWs go down or GWs restored, the map will reflect the new role of each GW resulted by these events. The map is distributed to the NVMeofGwMonitorClient on each GW, and it knows to update the GW with the required changes.

It is also adding 3 new mon commands:
- nvme-gw create
- nvme-gw delete
- nvme-gw show

The commands are used by the ceph adm to update the monitor that a new GW is deployed. The monitor will update the map accordingly and will start tracking this GW until it is deleted.

Signed-off-by: Leonid Chernin <lechernin@gmail.com>
Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

RGW|BN: fix lifecycle test and http server issue

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>

Merge pull request #58897 from Matan-B/wip-matanb-crimson-string-format

crimson/common/tri_mutex: avoid hobject_t formatting

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #58424 from guits/tracker_61597

ceph-volume: do not convert LVs's symlink to real path

mgr/cephadm: introducing new cmd to generate self-signed certs

this new Cephadm cmd introduces the ability to generate self-signed
certificates for external modules, signed by Cephadm as the root CA.
This feature is essential for implementing mTLS. Previously, if the
user did not provide a certificate and key, the dashboard would
generate its own. With this update, the dashboard now calls Cephadm
to generate self-signed certificates, enabling secure mTLS
communication with other backend applications. Prometheus module
also makes use of this new functionality to generate self-signed
certificates.

Signed-off-by: Redouane Kachach <rkachach@ibm.com>