os/bluestore: Write_v2 changes

4) remove Writer::shared_changed and use txc::shared_blobs directly

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Write_v2 changes

1) moved stats and blobs update to Writer::do_write
2) preallocate space in Writer:_split_data
3) fixed Writer::_write_expand_l that could check one extent too much

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

tests/bluestore_types: Fixed data generation bluestore_blob_t::release_extents

The #1 and #2 elements could form a continuous sequence but still not
joined:
Expected equality of these values:
  result
    Which is: { 0x7b138000~48000, 0x883b0000~48000, 0xf0c10000~10000, 0x727b8000~38000 }
  mid
    Which is: { 0x7b138000~30000, 0x7b168000~18000, 0x883b0000~48000, 0xf0c10000~10000, 0x727b8000~38000 }

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

qa/bluestore: Add write_v1/v2 selection

Add framework for various random options for debug bluestore.
Use framework to select:
- write_v1
- write_v2
- write_v1 / write_v2 selected at random

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add conf.bluestore_write_v2_random

Added conf.bluestore_write_v2_random. This is useful only for testing.
If set, it overrides value of bluestore_write_v2 with a random
true/false selection.
It is useful for v1 / v2 compatibility testing.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add compression fallback

For write_v2 create fallback to write_v1 if compression is selected.
This is temporary until compression dedicated to benefit from v2 is
merged.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Writer, fix find_mutable_blob

1) Algorithm assumed that blob->blob_start() is aligned to csum size.
It is true for blobs created by write_v2, but write_v1 can generate
blob like: begin = 0x9000, size = 0x6000, csum = 0x2000.
2) Blobs with unused were selected even if those need to be expanded.
This is illegal since we cannot expand unused.

Fixed blob selection algorithm.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Writer, improved calculation of need_size

More diligent calcualtion algorithm of need_size.
Takes into account front and back alignment.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Writer, fix for clang

Clang fails at _construct_at().

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

crimson/alienstore: New write path & debug

Added missing files to alienstore CMake list.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add Writer::_crop_allocs_to_io

Usually the data we put to disk is AU aligned.
In weird cases like AU=16K we put less data than we allocated.
_crop_allocs_to_io trims allocated extents into disk block extents
to reflect real IO.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix after rebase

BufferSpace is not with Onode, not Blob.
Modify code to adapt to this change.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Review fixes to Writer

Plus making const& variables in `for`.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add perf counters to Writer

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Writer, fix _try_reuse_allocated_l/r

It was not checked if necessary location is still within blob range.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Cleanup on bluestore_blob_t::release_extents

Result of ongoing review.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Writer, rename _deferred_decision -> _defer_or_allocate

A review proposal.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Review fixes of Writer

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix _try_reuse_allocated_r in Writer.cc

The search must go over entire range where we can have aligned mutable blobs.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Modify _write_expand_l/r

We need to scan all extents that are under proposed read / padding.
It was broken and we could clear out something defined.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add bluestore_write_v2 conf variable

Add new conf variable.
bluestore_write_v2 = true : use new _do_write_v2() function
bluestore_write_v2 = false : use legacy _do_write() function
This variable is read only at start time.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Added new write path with BlueStore

Added _do_write_v2 function to BlueStore.
Function is not integrated yet.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add "write_lat" perf counter

New "write_lat" tracks wallclock time spent in execution of BlueStore::_write().

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add unit test for BlueStore::Writer

Extensive tests for BlueStore::Writer.
Some extra debug hooks for BlueStore.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Introducing BlueStore::Writer

BlueStore::Writer is a toolkit to give more options to control write.
It gives more control over compression process, letting user of the class
manually split incoming data to blobs.
Now for large writes all but last blob can be fully filled with data.

There is now a single place that decides on deferred/direct.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Functions to manipulate Blob

Set of various simple functions to simplify code.
No special logic here.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add csum_type to WriteContext

It is more organized this way.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

tests/bluestore_types: Fix % 0 in tests

It was possible to get x % 0. It generated exception.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Unit tests for punch_hole_2

Comprehensive tests for punch_hole_2.
New formulation of punch_hole_2 makes it very easy to
create patterns and inspect results.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Refactor of write path. New punch_hole_2 function.

Introducing new logic of Onode processing during write.
New punch_hole_2 function empties range, but keeps track of elements:
- allocations that are no longer used
- blobs that are now empty
- shared blobs that got modified
- statfs changes to apply later

This change allows to reuse allocation for deferred freely, which means
that we can use allocations in deferred mode in other blob then they come from.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add ExtentMap::split_at and ExtentMap::maybe_split_at

Created dedicated mutator of ExtentMap that is useful when
a logical extent must be split.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add bluestore_blob_use_tracker_t::put_simple

New version of put().
It is simpler and faster, but does not allow for
overprovisioning of used AUs.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

include/intarith: add p2remain

p2remain gives remaining data in a block.
It simialar to p2nphase, but for 0 offset returns full size.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: New variant of bluestore_blob_t::release_extents

Created new variant of bluestore_blob_t::release_extents function.
Now the function takes range [offset~length] as an argument,
a simplification that allows it to have much better performance.

Created comprehensive unit test, tests 40k random blobs.
The unit test does not test for a potential case of having
bluestore_blob_t.extents that are not allocation unit aligned.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix debug after rebase

BufferSpace moved from Blob to Onode.
Moved relevant code from Blob::operator<< to Onode::operator<<.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add ability to partially print Onode

Now Onode can be printed in selected range.
It is useful in high-level dout modes that operate on a fragment of
entire Onode.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Move OBJECT_MAX_SIZE

From define in .cc to constexpr in .h

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Blob, Onode debug printout

Small improvement on debug output.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: small improvement for printers

- moved operator<< to BlueStore_debug file
- upcased Printer {} flags
- more reliable heap begin detection
- fixup after rebase

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add improved printer for Onode

Added nice replacement for dump_onode function.
Introduce printer class that allows to select parts of Onode that are to be printed.
It severly reduced amount of clutter in output.
Usage:
using P = Bluestore::printer;
dout << blob->print(P::ptr + P::sdisk + P::schk + P::buf + P::attrs);

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add printing buffers to Blob

Now printing Blob can include buffers.
There are 2 variants:
- 'buf' same as original in dump_onode
- 'sbuf' only fundamental params, no ptr etc.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add improved printer for Extent

Modify Extent similar to Blob, so that one can use improved Blob printing
when printing extents.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add improved printer for Blob

Introduce printer class that allows to select parts of Blob that are to be printed.
It severly reduced amount of clutter in output.
Usage:
using P = Bluestore::Blob::printer;
dout << blob->printer(P::ptr + P::sdisk + P::schk);

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #58907 from adam-lj/rbd-throttle-perf

librbd: Reduce use of atomics in librbd throttling

Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58993 from pereman2/unused-func

os/bluestore: Unused func

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #58693 from xxhdx1985126/wip-crimson-clear-temp-objects-startup

crimson/osd: clear ondisk temp objects on startup

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57562 from perezjosibm/wip-perezjos-cpu-map

script/: add cpu-map.sh to aid manual selection of CPU cores for threads for profiling

Merge pull request #55374 from aclamk/wip-aclamk-bs-multi-label

os/bluestore: Multiple bdev labels on main block device

Merge PR #59040 into main

* refs/pull/59040/head:
doc: documenting the feature that scrub clear the entries from damage table on repair

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #56782 from joscollin/wip-B65171-support-replication-start-end-notifications

cephfs_mirror: provide metrics for last successful snapshot sync

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57458 from lxbsz/wip-session-evict

qa/cephfs: add test_session_evict_non_blocklisted test case

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57987 from joscollin/wip-B64752-valgrind-report-leaks

qa/suites: drop --show-reachable=yes from fs:valgrind tests

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58097 from gardran/wip-gardran-kill-count

client: eliminate duplicate lookups wherever possible

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: documenting the feature that scrub clear the entries from damage table on repair

Fixes: https://tracker.ceph.com/issues/67337
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #59060 from qn2060/patch-3

doc/dev/peering: Change acting set num

Merge PR #52939 into main

* refs/pull/52939/head:
mon/MonClient: handle ms_handle_fast_authentication return

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #58437 into main

* refs/pull/58437/head:
mds: remove the misleading and redundant logs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #58582 into main

* refs/pull/58582/head:
test: add a test for `fs set down` on online cluster
mon: fix `fs set down` to adjust max_mds only when cluster is not down

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58511 from pereman2/ch-umount

os/bluestore: assert CollectionRef count is 1 on umount

Merge pull request #57572 from pereman2/refactors

os/bluestore: Improve Readability of Extent Map Sharding

Merge pull request #59011 from xxhdx1985126/wip-67327

crimson/osd: send empty transactions to backfill targets that haven't backfilled the objects yet

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58708 from xxhdx1985126/wip-67068

crimson/osd/osd_operation/client_request: requeue client requests from a temporary queue other than ClientRequest::Orderer::list

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58148 from xxhdx1985126/wip-65696

crimson/osd/osd_operations: hang requests if the objects are unfound

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59051 from dang/wip-dang-67321

RGW - Zipper: pull in upstream fix for zpp_bits

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/dev/peering: Change acting set num

This commit updates the peering doc page with the changed [2,1,2] to [3,1,2]

Signed-off-by: qn2060 <qn2060@users.noreply.github.com>

Merge pull request #58971 from cbodley/wip-rm-boost-redis-again-again

submodule: remove the boost_redis submodule again (again)

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #58037 from Suyashd999/refractor-PrimaryLogPG

osd/PrimaryLogPG.cc bl length would be stored before moving

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

RGW - Zipper: pull in upstream fix for zpp_bits

zpp_bits included code that required a newere compiler.  Pull in the
upstream fix from:

https://github.com/eyalz800/zpp_bits/pull/170

Fixes: https://tracker.ceph.com/issues/67321
Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

Merge pull request #59024 from zdover23/wip-doc-2024-08-05-README-building-ceph

doc/README.md - add ordered list

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58282 from NitzanMordhai/wip-nitzan-daemonwatchdog-should-terminate-thrasher-when-bark

qa/tasks: watchdog should terminate thrasher

Merge pull request #58664 from aclamk/wip-aclamk-qa-less-bluestore-debug

qa/suites/rados: Reduced BlueStore log levels

script/: add cpu-map.sh to aid manual selection of CPU cores for threads for profiling

Signed-off-by: Jose J Palacios-Perez <perezjos@uk.ibm.com>

thrashers: standardize stop and join method names

Thrashers that do not inherit from ThrasherGreenlet previously used a
method called do_join, which combined stop and join functionality. To
ensure consistency and clarity, we want all thrashers to use separate
stop, join, and stop_and_join methods.

This commit renames methods and implements missing stop and stop_and_join
methods in thrashers that did not inherit from ThrasherGreenlet.

Fixes: https://tracker.ceph.com/issues/66698
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

qa/tasks: watchdog terminate thrasher

If a thrasher exception occurs, the do_dump_ops thread will continue
looping until the Teuthology timeout is reached.
The watchdog should terminate the thrasher to free up resources.

Fixes: https://tracker.ceph.com/issues/66698
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge PR #58500 into main

* refs/pull/58500/head:
ptl-tool: Fix token file name and variable names

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58963 from rhcs-dashboard/fix-67009-main

mgr/dashboard: Dashboard not showing Object/Overview correctly.

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #58746 from rhcs-dashboard/fix-67069-main

mgr/dashboard: make multi-cluster context switcher scrollable if we have many entries in it

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #57905 from mcpreston/docs-dashboard-typo

doc/mgr/dashboard: fix TLS typo

Merge pull request #58767 from kshtsk/wip-python-invalid-escape-sequence

mgr: fix invalid escape sequence

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58373 from ceph/wip-tasks-ceph-mkfs-has-no-f

qa: ceph.py mkfs.ext4 has no -f

os/bluestore: BluestoreRepairTest ch cleanup

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

doc/README.md - add ordered list

Add ordered-list formatting to the "Building Ceph" section of README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58891 from zdover23/wip-doc-2024-07-28-cephfs-file-layouts-layout-fields

doc/cephfs: edit "Layout Fields" text

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58115 from ronen-fr/wip-rf-memmodel

common: MemoryModel: performance improvements and API changes

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

librbd: Reduce use of atomics in librbd throttling

Signed-off-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>

Merge pull request #57786 from Svelar/asan_cli_test

crush: avoid out-of-bound access and simplify enlarging buckets

Merge pull request #58718 from wanglinke521/wlk-kv-stats

kv/RocksDBStore: add kv_stats function to print all CF info

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #58470 from mohit84/issue_66867

mon: Remove any pg_upmap_primary mapping during remove a pool

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Josh Salomon <josh.salomon@gmail.com>

Merge pull request #58302 from jianwei1216/feat_add_mclock_perfcounter_main

osd/scheduler: add mclock queue length perfcounter

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

crimson/osd: send empty transactions to backfill targets that haven't
backfilled the objects yet

Fixes: https://tracker.ceph.com/issues/67327
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #59006 from zdover23/wip-doc-2024-08-03-glossary-ceph-ansible

doc/glossary: add "ceph-ansible"

doc/glossary: add "ceph-ansible"

Add a "ceph-ansible" entry to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58969 from gbregman/main

mgr/cephadm: Add SPDK log level to nvmeof configuration file

Merge pull request #58402 from rkachach/fix_issue_mtls_support

mgr/cephadm: adding mTLS for ceph mgmt-gateway and backend services communication

Reviewed-by: Adam King <adking@redhat.com>

mgr/cephadm: add SPDK log level to nvmeof configuration
Fixes https://tracker.ceph.com/issues/67258

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

python-common/ceph/deployment: add SPDK log level to nvmeof configuration
Fixes https://tracker.ceph.com/issues/67258

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #58883 from idryomov/wip-backport-create-resourceattrerror

script/backport-create-issue: handle ResourceAttrError when getting CF_TAGS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58882 from idryomov/wip-native-format-dispatch

librbd/migration: don't instantiate NativeFormat, handle it via dispatch

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: N Balachandran <nibalach@redhat.com>

Merge pull request #58862 from cbodley/wip-rgw-posix-bucket-cache-lock

rgw/posix: name the lock_guard in BucketCacheEntry::reclaim()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

os/bluestore: remove unused functions

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>