doc/rgw/notification: persistent notification queue full behavior

Fixes: https://tracker.ceph.com/issues/50610
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #59168 from zdover23/wip-doc-2024-08-12-cephfs-cache-configuration

doc/cephfs: improve cache-configuration.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #59186 from rhcs-dashboard/replace-cluster-capacity-with-usage-bar

mgr/dashboard: replace individual cluster's capacity info with Usage bar in Multi-Cluster

Merge PR #59025 into main

* refs/pull/59025/head:
tools/rados: Fix extra NL in getxattr

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Gabriel Benhanokh <gbenhano@redhat.com>

Merge pull request #58485 from ivoalmeida/carbon-datatable

mgr/dashboard: replace ngx-datatable by carbon datatable

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #58140 from guits/cv-tpm2-support

ceph-volume: add TPM2 token enrollment support for encrypted OSDs

Merge pull request #58956 from ThomasLamprecht/ceph-volume-debian-dependency

debian pkg: record python3-packaging dependency for ceph-volume

mgr/dashboard: replace ngx-datatable by carbon

Fixes: https://tracker.ceph.com/issues/66965
* replaced ngx-datatable by carbon datatable
* created carbon themes for content and tables
* redesigned table actions to render as kebab menu options per data row
* keep only primary actions on datatable toolbar
* implemented carbon batch actions

Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge pull request #59112 from idryomov/wip-67353

qa: adjust expected io_opt in krbd_discard_granularity.t

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #54504 from aclamk/wip-aclamk-bs-refactor-write-path

os/bluestore: Recompression, part 2. New write path.

Merge pull request #59001 from cbodley/wip-rgw-async-inline-gc

rgw/rados: delete_objs_inline() uses rgw_rados_operate()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mgr/dashboard: replace individual cluster's capacity info with Usage bar in Multi-Cluster

fixes: https://tracker.ceph.com/issues/67193

Signed-off-by: Naman Munet <nmunet@redhat.com>

Merge pull request #58410 from bill-scales/erasure_coding_design_document

doc: doc/dev/osd_interals/erasure_coding/enhancements.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Sam Just <sjust@redhat.com>

Merge pull request #59177 from anthonyeleven/aad-qat-followup

doc/radosgw: improve qat-accel.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/radosgw: improve qat-accel.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge PR #58532 into main

* refs/pull/58532/head:
qa: improve and add commit ID to comment for context

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephfs: improve cache-configuration.rst

Improve the text in the section about dealing with cache-pressure alerts
that was added in https://github.com/ceph/ceph/pull/59077. The changes
in this commit were suggested by Anthony D'Atri.

Co-authored-by: Patrick Donnelly <pdonnelly@redhat.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55050 from rhcs-dashboard/nfs-monitoring

mgr/nfs: scrape nfs monitoring endpoint

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #54075 from aclamk/wip-aclamk-bs-nice-print

os/bluestore: Recompression, part 1. Nice debugs.

Merge pull request #59164 from toficly/patch-2

doc:update e-mail addresses governance

doc:update e-mail addresses governance

with the consent of my colleagues
- Dan van der Ster
- João Eduardo Luis
- Mark Nelson

changed their e-mail addresses to the current one

Signed-off-by: Tobias Fischer <tobias.fischer@clyso.com>
Signed-off-by: Tobias Fischer <138117964+toficly@users.noreply.github.com>

ceph-volume: refactor device path handling for LVM lookups

This consolidates the conditional checks for device paths to
reduce redundancy and improve readability and adds logic to
handle both '/dev/mapper' and '/dev/dm-' paths uniformly by
introducing a utility function `get_lvm_mapper_path_from_dm()`.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: add TPM2 token enrollment support for encrypted OSDs

This adds the required changes to ceph-volume and cephadm in order to support
deploying tpm2 token enrolled encrypted OSDs.

Adding `--with-tpm` when deploying with `--dmcrypt` will enroll a tpm2
token to the corresponding LUKS2 devices.

Example of a osd service spec:

```
service_type: osd
service_id: tpm2_osds
placement:
  host_pattern: '*'
spec:
  data_devices:
    paths:
      - /dev/sdb
  encrypted: true
  tpm2: true
```

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #59098 from athanatos/sjust/wip-67415-alienstore-mkfs-crash

crimson: clear AlienStore::coll_map in umount rather than in stop

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59077 from zdover23/wip-doc-2024-08-07-cephfs-cache-configuration-cache-pressure

doc/cephfs: add cache pressure information

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58951 from rhcs-dashboard/ceph-users-doc-fix

mgr/dashboard: fix ceph-users api

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58976 from athanatos/sjust/for-review/wip-nvmeofpaxosservice-cosmetic-fixes

mon: cosmetic fixes for NVMeofGwMon files

Reviewed-by: Alexander Indenbaum <aindenba@redhat.com>
Reviewed-by: Leonid Chernin <lechernin@gmail.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Aviv Caro <Aviv.Caro@ibm.com>

Merge pull request #57003 from linuxkidd/wip-pool-max-avail

PGMap: remove pool max_avail scale factor

Merge pull request #58573 from idryomov/wip-selinux-drop-ignore-agetty

qa: agetty entry in selinux allowlist is redundant now

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #58784 into main

* refs/pull/58784/head:
script/backport-create-issue: add debug messages for retry

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

qa: cover a custom object size in krbd_discard_granularity.t

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: adjust expected io_opt in krbd_discard_granularity.t

With linux.git commit a00d4bfce7c6 ("rbd: increase io_opt again"),
io_opt is set to object set size.

Fixes: https://tracker.ceph.com/issues/67353
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58605 from adk3798/upgrade-suite-upgrade-in-progress-checks

qa/upgrade: fix checks to make sure upgrade is still in progress

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: fix ceph-users api doc

Fixes: https://tracker.ceph.com/issues/67224
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/nfs: scrape nfs monitoring endpoint

Fixes: https://tracker.ceph.com/issues/62558
Signed-off-by: avanthakkar <avanjohn@gmail.com>

doc/cephfs: add cache pressure information

Add information to doc/cephfs/cache-configuration.rst about how to deal
with a message that reads "clients failing to respond to cache
pressure". This procedure explains how to slow the growth of the
recall_caps value so that it does not exceed the
mds_recall_warning_threshold.

The information in this commit was developed by Eugen Block. See
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/5ROH5CWKKOEIQMVXOVRT5OO7CWK2HPM3/#J65DFUPP4BY57MICPANXKI7KAXSZ5Z5P
and https://www.spinics.net/lists/ceph-users/msg73188.html.

Fixes: https://tracker.ceph.com/issues/57115
Co-authored-by: Eugen Block <eblock@nde.ag>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59087 from kamoltat/wip-ksirivad-fix-stretch-device-class-docs

docs/rados/operations/stretch-mode: warn device class is not supported

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58736 from amathuria/wip-66922-amat

qa/rados/dashboard: Add PG_DEGRADED to ignorelist

Merge pull request #57037 from adk3798/cephadm-limit-shell-mounts

cephadm: limit mounts for shell and ceph-volume commands

Merge PR #58861 into main

* refs/pull/58861/head:
mds: only authpin on wrlock when not a locallock

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

PGMap: remove pool max_avail scale factor

The scaling of max_avail by the ratio of non-degraded to total objects
count results in the reported max_avail increasing proportionally to the
number of OSDs marked `down` but not `out`.  This is counter intuitive
since OSDs going `down` should never result in more space being
available.

Removing the scale factor allows max_avail to remain unchanged until the
OSDs are marked `out`.

Signed-off-by: Michael J. Kidd <linuxkidd@gmail.com>

Merge pull request #59091 from zdover23/wip-doc-2024-08-08-README-md-alerts

doc/README.md - add "tip" alert styling

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #59030 from ljflores/wip-fix-crontab-2

qa/crontab: force priority for main rados nightlies

Merge pull request #58583 from lxbsz/wip-66946

qa/cephfs: fix TestRenameCommand and unmount the clinet before failin…

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: doc/dev/osd_internals/erasure_coding/enhancements.rst

Erasure coding enhancements design document

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

doc/README.md - add "tip" alert styling

Add "tip" alert styling (what in Docbook XML is called "an admonition")
to information about Ninja in an ordered list (which is what markdown
has here instead of procedures).

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59049 from zdover23/wip-doc-2024-08-06-README-building-ceph

doc/README.md - allocate information correctly

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58913 from cyx1231st/wip-seastore-unlock-ool

crimson/os/seastore: move ool writes from collection lock to concurrent DeviceSubmission phase

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #58787 from xxhdx1985126/wip-67106

crimson/os/seastore: consider segment_header_t::modify_time as the segments' modify_time for no-tail OOL segments

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57911 from gardran/wip-gdran-mds-better-clease-handling

mds: some optimizations around client Capability and Lease tracking

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/.../alien_store: clear coll_map in umount rather than stop

As of a926a4a54ced, BlueStore::umount expects outstanding CollectionRef's
to already have been released.

Previously, 8389471b3 updated alien_store.cc to release coll_map in
stop() in in the alien tp.

Fixes: https://tracker.ceph.com/issues/67415
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #58983 from cyx1231st/wip-seastore-cleanup-cache

crimson/os/seastore/cache: cleanups and comments

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #58869 from xxhdx1985126/wip-seastore-lba-leaf-node-minor-bug-fix

crimson/os/seastore/lba_manager: correct the range end of lba mappings

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #59029 into main

* refs/pull/59029/head:
qa: simplify postmerge construction

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

crimson/osd/osd.cc: convert fmt::print to INFO in OSD::mkfs

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd.cc: convert OSD::mkfs to a coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #44366 from orozery/rbd-crypto-migration

librbd/crypto: fix issue when live-migrating from encrypted export

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58002 from nbalacha/wip-nbalacha-sorted-snaps

cls/rbd: add group_snap_list_order method to enable sorting snapshots in creation order

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

docs/rados/operations/stretch-mode: warn device class is not supported

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #59073 from anoopcs9/cephadm-docs-smb-spec

doc/cephadm/smb: Add missing user_sources option to service spec

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56233 from kamoltat/wip-ksirivad-fix-64802

RADOS: Generalize stretch mode pg temp handling to be usable without stretch mode
Samuel Just <sjust@redhat.com>

Merge pull request #57381 from kamoltat/wip-ksirivad-fix-65371

PeeringState.cc: Only populate want_acting when num_osds < bucket_max
Reviewed by: Samuel Just <sjust@redhat.com>

Merge pull request #58585 from anoopcs9/cephadm-update-smbd-version

cephadm/smb: Determine samba version within container

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52489 from ifed01/wip-ifed-alloc2

os/bluestore: introduce hybrid_btree2 allocator

os/bluestore: Write_v2 changes

4) remove Writer::shared_changed and use txc::shared_blobs directly

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Write_v2 changes

1) moved stats and blobs update to Writer::do_write
2) preallocate space in Writer:_split_data
3) fixed Writer::_write_expand_l that could check one extent too much

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

tests/bluestore_types: Fixed data generation bluestore_blob_t::release_extents

The #1 and #2 elements could form a continuous sequence but still not
joined:
Expected equality of these values:
  result
    Which is: { 0x7b138000~48000, 0x883b0000~48000, 0xf0c10000~10000, 0x727b8000~38000 }
  mid
    Which is: { 0x7b138000~30000, 0x7b168000~18000, 0x883b0000~48000, 0xf0c10000~10000, 0x727b8000~38000 }

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

qa/bluestore: Add write_v1/v2 selection

Add framework for various random options for debug bluestore.
Use framework to select:
- write_v1
- write_v2
- write_v1 / write_v2 selected at random

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add conf.bluestore_write_v2_random

Added conf.bluestore_write_v2_random. This is useful only for testing.
If set, it overrides value of bluestore_write_v2 with a random
true/false selection.
It is useful for v1 / v2 compatibility testing.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add compression fallback

For write_v2 create fallback to write_v1 if compression is selected.
This is temporary until compression dedicated to benefit from v2 is
merged.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Writer, fix find_mutable_blob

1) Algorithm assumed that blob->blob_start() is aligned to csum size.
It is true for blobs created by write_v2, but write_v1 can generate
blob like: begin = 0x9000, size = 0x6000, csum = 0x2000.
2) Blobs with unused were selected even if those need to be expanded.
This is illegal since we cannot expand unused.

Fixed blob selection algorithm.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Writer, improved calculation of need_size

More diligent calcualtion algorithm of need_size.
Takes into account front and back alignment.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Writer, fix for clang

Clang fails at _construct_at().

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

crimson/alienstore: New write path & debug

Added missing files to alienstore CMake list.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add Writer::_crop_allocs_to_io

Usually the data we put to disk is AU aligned.
In weird cases like AU=16K we put less data than we allocated.
_crop_allocs_to_io trims allocated extents into disk block extents
to reflect real IO.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix after rebase

BufferSpace is not with Onode, not Blob.
Modify code to adapt to this change.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Review fixes to Writer

Plus making const& variables in `for`.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add perf counters to Writer

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Writer, fix _try_reuse_allocated_l/r

It was not checked if necessary location is still within blob range.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Cleanup on bluestore_blob_t::release_extents

Result of ongoing review.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Writer, rename _deferred_decision -> _defer_or_allocate

A review proposal.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Review fixes of Writer

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix _try_reuse_allocated_r in Writer.cc

The search must go over entire range where we can have aligned mutable blobs.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Modify _write_expand_l/r

We need to scan all extents that are under proposed read / padding.
It was broken and we could clear out something defined.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add bluestore_write_v2 conf variable

Add new conf variable.
bluestore_write_v2 = true : use new _do_write_v2() function
bluestore_write_v2 = false : use legacy _do_write() function
This variable is read only at start time.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Added new write path with BlueStore

Added _do_write_v2 function to BlueStore.
Function is not integrated yet.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add "write_lat" perf counter

New "write_lat" tracks wallclock time spent in execution of BlueStore::_write().

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add unit test for BlueStore::Writer

Extensive tests for BlueStore::Writer.
Some extra debug hooks for BlueStore.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Introducing BlueStore::Writer

BlueStore::Writer is a toolkit to give more options to control write.
It gives more control over compression process, letting user of the class
manually split incoming data to blobs.
Now for large writes all but last blob can be fully filled with data.

There is now a single place that decides on deferred/direct.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Functions to manipulate Blob

Set of various simple functions to simplify code.
No special logic here.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add csum_type to WriteContext

It is more organized this way.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

tests/bluestore_types: Fix % 0 in tests

It was possible to get x % 0. It generated exception.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Unit tests for punch_hole_2

Comprehensive tests for punch_hole_2.
New formulation of punch_hole_2 makes it very easy to
create patterns and inspect results.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Refactor of write path. New punch_hole_2 function.

Introducing new logic of Onode processing during write.
New punch_hole_2 function empties range, but keeps track of elements:
- allocations that are no longer used
- blobs that are now empty
- shared blobs that got modified
- statfs changes to apply later

This change allows to reuse allocation for deferred freely, which means
that we can use allocations in deferred mode in other blob then they come from.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add ExtentMap::split_at and ExtentMap::maybe_split_at

Created dedicated mutator of ExtentMap that is useful when
a logical extent must be split.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add bluestore_blob_use_tracker_t::put_simple

New version of put().
It is simpler and faster, but does not allow for
overprovisioning of used AUs.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

include/intarith: add p2remain

p2remain gives remaining data in a block.
It simialar to p2nphase, but for 0 offset returns full size.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: New variant of bluestore_blob_t::release_extents

Created new variant of bluestore_blob_t::release_extents function.
Now the function takes range [offset~length] as an argument,
a simplification that allows it to have much better performance.

Created comprehensive unit test, tests 40k random blobs.
The unit test does not test for a potential case of having
bluestore_blob_t.extents that are not allocation unit aligned.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix debug after rebase

BufferSpace moved from Blob to Onode.
Moved relevant code from Blob::operator<< to Onode::operator<<.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add ability to partially print Onode

Now Onode can be printed in selected range.
It is useful in high-level dout modes that operate on a fragment of
entire Onode.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>