Merge pull request #59049 from zdover23/wip-doc-2024-08-06-README-building-ceph

doc/README.md - allocate information correctly

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58913 from cyx1231st/wip-seastore-unlock-ool

crimson/os/seastore: move ool writes from collection lock to concurrent DeviceSubmission phase

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #58787 from xxhdx1985126/wip-67106

crimson/os/seastore: consider segment_header_t::modify_time as the segments' modify_time for no-tail OOL segments

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57911 from gardran/wip-gdran-mds-better-clease-handling

mds: some optimizations around client Capability and Lease tracking

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58983 from cyx1231st/wip-seastore-cleanup-cache

crimson/os/seastore/cache: cleanups and comments

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #58869 from xxhdx1985126/wip-seastore-lba-leaf-node-minor-bug-fix

crimson/os/seastore/lba_manager: correct the range end of lba mappings

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #59029 into main

* refs/pull/59029/head:
qa: simplify postmerge construction

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #44366 from orozery/rbd-crypto-migration

librbd/crypto: fix issue when live-migrating from encrypted export

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58002 from nbalacha/wip-nbalacha-sorted-snaps

cls/rbd: add group_snap_list_order method to enable sorting snapshots in creation order

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #59073 from anoopcs9/cephadm-docs-smb-spec

doc/cephadm/smb: Add missing user_sources option to service spec

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56233 from kamoltat/wip-ksirivad-fix-64802

RADOS: Generalize stretch mode pg temp handling to be usable without stretch mode
Samuel Just <sjust@redhat.com>

Merge pull request #57381 from kamoltat/wip-ksirivad-fix-65371

PeeringState.cc: Only populate want_acting when num_osds < bucket_max
Reviewed by: Samuel Just <sjust@redhat.com>

Merge pull request #58585 from anoopcs9/cephadm-update-smbd-version

cephadm/smb: Determine samba version within container

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52489 from ifed01/wip-ifed-alloc2

os/bluestore: introduce hybrid_btree2 allocator

librbd/migration: make ImageDispatch handle encryption for non-native formats

With NativeFormat now being handled via dispatch, handling encryption
for non-native formats (i.e. mapping to raw image extents and performing
decryption/mapping back on completion) in the migration layer is really
straightforward.

Note that alignment doesn't need to be performed in the migration layer
because it happens on the destination image -- the "align and resubmit"
logic in C_UnalignedObjectReadRequest should kick in before the call to
read_parent().

Fixes: https://tracker.ceph.com/issues/53674
Co-authored-by: Or Ozeri <oro@il.ibm.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: don't make an extra copy of image_extents in C_ImageReadRequest ctor

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/workunits/rbd: perform cleanup in test_clone_encryption()

... so that RAW_DEV can be unmapped and future tests can reuse testimg
and other image names without bumping into watchers and older snapshots.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/workunits/rbd: no need to chmod in luks-encryption.sh

Most workunits expect the user to be a member of "disk" group, so we
can pretty much rely on that being the case at this point.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58907 from adam-lj/rbd-throttle-perf

librbd: Reduce use of atomics in librbd throttling

Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58993 from pereman2/unused-func

os/bluestore: Unused func

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #58693 from xxhdx1985126/wip-crimson-clear-temp-objects-startup

crimson/osd: clear ondisk temp objects on startup

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57562 from perezjosibm/wip-perezjos-cpu-map

script/: add cpu-map.sh to aid manual selection of CPU cores for threads for profiling

Merge pull request #55374 from aclamk/wip-aclamk-bs-multi-label

os/bluestore: Multiple bdev labels on main block device

Merge PR #59040 into main

* refs/pull/59040/head:
doc: documenting the feature that scrub clear the entries from damage table on repair

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc/cephadm/smb: Add missing user_sources option to service spec

Signed-off-by: Anoop C S <anoopcs@cryptolab.net>

crimson/os/seastore: report outstanding requests per phase

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: wait ool writes in DeviceSubmission phase

So that it is moved out of the collection lock.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #56782 from joscollin/wip-B65171-support-replication-start-end-notifications

cephfs_mirror: provide metrics for last successful snapshot sync

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57458 from lxbsz/wip-session-evict

qa/cephfs: add test_session_evict_non_blocklisted test case

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57987 from joscollin/wip-B64752-valgrind-report-leaks

qa/suites: drop --show-reachable=yes from fs:valgrind tests

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58097 from gardran/wip-gardran-kill-count

client: eliminate duplicate lookups wherever possible

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: documenting the feature that scrub clear the entries from damage table on repair

Fixes: https://tracker.ceph.com/issues/67337
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

doc/README.md - allocate information correctly

Add information in the "Building Ceph" procedure to the steps where it
belongs. Add information about "do_cmake.sh" to the step about
"do_cmake.sh". Add information about Ninja to the step about Ninja.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59060 from qn2060/patch-3

doc/dev/peering: Change acting set num

Merge PR #52939 into main

* refs/pull/52939/head:
mon/MonClient: handle ms_handle_fast_authentication return

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #58437 into main

* refs/pull/58437/head:
mds: remove the misleading and redundant logs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #58582 into main

* refs/pull/58582/head:
test: add a test for `fs set down` on online cluster
mon: fix `fs set down` to adjust max_mds only when cluster is not down

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58511 from pereman2/ch-umount

os/bluestore: assert CollectionRef count is 1 on umount

Merge pull request #57572 from pereman2/refactors

os/bluestore: Improve Readability of Extent Map Sharding

Merge pull request #59011 from xxhdx1985126/wip-67327

crimson/osd: send empty transactions to backfill targets that haven't backfilled the objects yet

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58708 from xxhdx1985126/wip-67068

crimson/osd/osd_operation/client_request: requeue client requests from a temporary queue other than ClientRequest::Orderer::list

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58148 from xxhdx1985126/wip-65696

crimson/osd/osd_operations: hang requests if the objects are unfound

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59051 from dang/wip-dang-67321

RGW - Zipper: pull in upstream fix for zpp_bits

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/dev/peering: Change acting set num

This commit updates the peering doc page with the changed [2,1,2] to [3,1,2]

Signed-off-by: qn2060 <qn2060@users.noreply.github.com>

Merge pull request #58971 from cbodley/wip-rm-boost-redis-again-again

submodule: remove the boost_redis submodule again (again)

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #58037 from Suyashd999/refractor-PrimaryLogPG

osd/PrimaryLogPG.cc bl length would be stored before moving

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

RGW - Zipper: pull in upstream fix for zpp_bits

zpp_bits included code that required a newere compiler.  Pull in the
upstream fix from:

https://github.com/eyalz800/zpp_bits/pull/170

Fixes: https://tracker.ceph.com/issues/67321
Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

Merge pull request #59024 from zdover23/wip-doc-2024-08-05-README-building-ceph

doc/README.md - add ordered list

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58282 from NitzanMordhai/wip-nitzan-daemonwatchdog-should-terminate-thrasher-when-bark

qa/tasks: watchdog should terminate thrasher

Merge pull request #58664 from aclamk/wip-aclamk-qa-less-bluestore-debug

qa/suites/rados: Reduced BlueStore log levels

script/: add cpu-map.sh to aid manual selection of CPU cores for threads for profiling

Signed-off-by: Jose J Palacios-Perez <perezjos@uk.ibm.com>

thrashers: standardize stop and join method names

Thrashers that do not inherit from ThrasherGreenlet previously used a
method called do_join, which combined stop and join functionality. To
ensure consistency and clarity, we want all thrashers to use separate
stop, join, and stop_and_join methods.

This commit renames methods and implements missing stop and stop_and_join
methods in thrashers that did not inherit from ThrasherGreenlet.

Fixes: https://tracker.ceph.com/issues/66698
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

qa/tasks: watchdog terminate thrasher

If a thrasher exception occurs, the do_dump_ops thread will continue
looping until the Teuthology timeout is reached.
The watchdog should terminate the thrasher to free up resources.

Fixes: https://tracker.ceph.com/issues/66698
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge PR #58500 into main

* refs/pull/58500/head:
ptl-tool: Fix token file name and variable names

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58963 from rhcs-dashboard/fix-67009-main

mgr/dashboard: Dashboard not showing Object/Overview correctly.

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #58746 from rhcs-dashboard/fix-67069-main

mgr/dashboard: make multi-cluster context switcher scrollable if we have many entries in it

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

cls/rbd: add functions to get group snap orders

Added functions to get the group snap order keys.

Signed-off-by: N Balachandran <nibalach@redhat.com>

crimson/os/seastore/transaction: cleanup, drop unused getters

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction: rename ool_block_lists

Drop the inaccurate written_ prefix because they may be still writting
when added to the lists.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/epm: RandomBlockOolWriter to update extents upon submitting writes

In order to move the ool write futures out of the collection lock, which
will be after the transaction prepare phase.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

qa: simplify postmerge construction

and avoid errors when "clusternodes" is not defined.

Fixes: https://tracker.ceph.com/issues/67352
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57905 from mcpreston/docs-dashboard-typo

doc/mgr/dashboard: fix TLS typo

Merge pull request #58767 from kshtsk/wip-python-invalid-escape-sequence

mgr: fix invalid escape sequence

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58373 from ceph/wip-tasks-ceph-mkfs-has-no-f

qa: ceph.py mkfs.ext4 has no -f

os/bluestore: BluestoreRepairTest ch cleanup

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

cls/rbd: save max group snap order

Save the last used group snap order value to speed up
snapshot creation.

Signed-off-by: N Balachandran <nibalach@redhat.com>

cls/rbd: save group snapshot creation order in a new key

In order to be able to list group snapshots in creation order, a
new snapshot order key is stored on snapshot creation.

[nbalacha: removed the group_snap_unsorted function,
reverted the group_snap_list function changes]

Signed-off-by: Mykola Golub <mgolub@suse.com>
Signed-off-by: N Balachandran <nibalach@redhat.com>

doc/README.md - add ordered list

Add ordered-list formatting to the "Building Ceph" section of README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58891 from zdover23/wip-doc-2024-07-28-cephfs-file-layouts-layout-fields

doc/cephfs: edit "Layout Fields" text

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

PeeringState.cc: Only populate want_acting when num_osds < bucket_max

Currently, in `calc_replicated_acting_stretch` we would
call `pop_ancester` before checking whether the next
OSD we populate in the want_acting set would have
the ancestor count that is >= bucket_max.

Fixes: https://tracker.ceph.com/issues/65371
Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #58115 from ronen-fr/wip-rf-memmodel

common: MemoryModel: performance improvements and API changes

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

librbd: Reduce use of atomics in librbd throttling

Signed-off-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>

crimson/os/seastore/cache: introduce maybe_add_to_read_set()

And only touch extent if it is absent in transaction.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: verify pending extents in get_extent_viewable_by_trans()

Distinguish specific cases carefully.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57786 from Svelar/asan_cli_test

crush: avoid out-of-bound access and simplify enlarging buckets

crimson/os/seastore: consider segment_header_t::modify_time as the
segments' modify_time for no-tail OOL segments

OOL segments don't contain record headers anymore, we use
segment_header_t::modify_time as the approximation of segment
modify_time

Fixes: https://tracker.ceph.com/issues/67106
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/epm: SegmentedOolWriter to update ool paddr upon submitting writes

In order to move the ool write futures out of the collection lock, which
will be after the transaction prepare phase.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/record_submitter: return record-base immediately upon submit

For out-of-line write only.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/record_submitter: cleanup around write_base

No longer need to pass write_base through the write promise.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/record_submitter: minor cleanups

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/record_submitter: refactor to make write base available

Make sure the write base is available upon RecordSubmitter::submit().

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/record_submitter: refactor JournalAllocator::write()

Decouple get_written_to() so it can be queried upon writes.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #58718 from wanglinke521/wlk-kv-stats

kv/RocksDBStore: add kv_stats function to print all CF info

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #58470 from mohit84/issue_66867

mon: Remove any pg_upmap_primary mapping during remove a pool

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Josh Salomon <josh.salomon@gmail.com>

Merge pull request #58302 from jianwei1216/feat_add_mclock_perfcounter_main

osd/scheduler: add mclock queue length perfcounter

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

crimson/osd: send empty transactions to backfill targets that haven't
backfilled the objects yet

Fixes: https://tracker.ceph.com/issues/67327
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #59006 from zdover23/wip-doc-2024-08-03-glossary-ceph-ansible

doc/glossary: add "ceph-ansible"

doc/glossary: add "ceph-ansible"

Add a "ceph-ansible" entry to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58969 from gbregman/main

mgr/cephadm: Add SPDK log level to nvmeof configuration file

Merge pull request #58402 from rkachach/fix_issue_mtls_support

mgr/cephadm: adding mTLS for ceph mgmt-gateway and backend services communication

Reviewed-by: Adam King <adking@redhat.com>

mgr/cephadm: add SPDK log level to nvmeof configuration
Fixes https://tracker.ceph.com/issues/67258

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

python-common/ceph/deployment: add SPDK log level to nvmeof configuration
Fixes https://tracker.ceph.com/issues/67258

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #58883 from idryomov/wip-backport-create-resourceattrerror

script/backport-create-issue: handle ResourceAttrError when getting CF_TAGS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58882 from idryomov/wip-native-format-dispatch

librbd/migration: don't instantiate NativeFormat, handle it via dispatch

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: N Balachandran <nibalach@redhat.com>

Merge pull request #58862 from cbodley/wip-rgw-posix-bucket-cache-lock

rgw/posix: name the lock_guard in BucketCacheEntry::reclaim()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

os/bluestore: remove unused functions

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #58765 from yuvalif/wip-yuval-64184

rgw/async/notifications: use common async waiter in pubsub push

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58672 from YiteGu/code-cleanup-osdmon

mon/osdmonitor: cleanup the code for preprocess_mark_me_dead

Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #58635 from badone/wip-tracker-50371-rados_api_test-timeout-failures

qa: Restrict rados api tests to large clusters and increase timeout

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #58631 from cbodley/wip-66336

msg: insert PriorityDispatchers in sorted position

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>