qa: cover a custom object size in krbd_discard_granularity.t

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: adjust expected io_opt in krbd_discard_granularity.t

With linux.git commit a00d4bfce7c6 ("rbd: increase io_opt again"),
io_opt is set to object set size.

Fixes: https://tracker.ceph.com/issues/67353
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #58861 into main

* refs/pull/58861/head:
mds: only authpin on wrlock when not a locallock

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #59091 from zdover23/wip-doc-2024-08-08-README-md-alerts

doc/README.md - add "tip" alert styling

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #59030 from ljflores/wip-fix-crontab-2

qa/crontab: force priority for main rados nightlies

Merge pull request #58583 from lxbsz/wip-66946

qa/cephfs: fix TestRenameCommand and unmount the clinet before failin…

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/README.md - add "tip" alert styling

Add "tip" alert styling (what in Docbook XML is called "an admonition")
to information about Ninja in an ordered list (which is what markdown
has here instead of procedures).

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59049 from zdover23/wip-doc-2024-08-06-README-building-ceph

doc/README.md - allocate information correctly

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58913 from cyx1231st/wip-seastore-unlock-ool

crimson/os/seastore: move ool writes from collection lock to concurrent DeviceSubmission phase

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #58787 from xxhdx1985126/wip-67106

crimson/os/seastore: consider segment_header_t::modify_time as the segments' modify_time for no-tail OOL segments

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57911 from gardran/wip-gdran-mds-better-clease-handling

mds: some optimizations around client Capability and Lease tracking

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58983 from cyx1231st/wip-seastore-cleanup-cache

crimson/os/seastore/cache: cleanups and comments

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #58869 from xxhdx1985126/wip-seastore-lba-leaf-node-minor-bug-fix

crimson/os/seastore/lba_manager: correct the range end of lba mappings

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #59029 into main

* refs/pull/59029/head:
qa: simplify postmerge construction

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #44366 from orozery/rbd-crypto-migration

librbd/crypto: fix issue when live-migrating from encrypted export

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58002 from nbalacha/wip-nbalacha-sorted-snaps

cls/rbd: add group_snap_list_order method to enable sorting snapshots in creation order

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #59073 from anoopcs9/cephadm-docs-smb-spec

doc/cephadm/smb: Add missing user_sources option to service spec

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56233 from kamoltat/wip-ksirivad-fix-64802

RADOS: Generalize stretch mode pg temp handling to be usable without stretch mode
Samuel Just <sjust@redhat.com>

Merge pull request #57381 from kamoltat/wip-ksirivad-fix-65371

PeeringState.cc: Only populate want_acting when num_osds < bucket_max
Reviewed by: Samuel Just <sjust@redhat.com>

Merge pull request #58585 from anoopcs9/cephadm-update-smbd-version

cephadm/smb: Determine samba version within container

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52489 from ifed01/wip-ifed-alloc2

os/bluestore: introduce hybrid_btree2 allocator

librbd/migration: make ImageDispatch handle encryption for non-native formats

With NativeFormat now being handled via dispatch, handling encryption
for non-native formats (i.e. mapping to raw image extents and performing
decryption/mapping back on completion) in the migration layer is really
straightforward.

Note that alignment doesn't need to be performed in the migration layer
because it happens on the destination image -- the "align and resubmit"
logic in C_UnalignedObjectReadRequest should kick in before the call to
read_parent().

Fixes: https://tracker.ceph.com/issues/53674
Co-authored-by: Or Ozeri <oro@il.ibm.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: don't make an extra copy of image_extents in C_ImageReadRequest ctor

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/workunits/rbd: perform cleanup in test_clone_encryption()

... so that RAW_DEV can be unmapped and future tests can reuse testimg
and other image names without bumping into watchers and older snapshots.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/workunits/rbd: no need to chmod in luks-encryption.sh

Most workunits expect the user to be a member of "disk" group, so we
can pretty much rely on that being the case at this point.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58907 from adam-lj/rbd-throttle-perf

librbd: Reduce use of atomics in librbd throttling

Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58993 from pereman2/unused-func

os/bluestore: Unused func

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #58693 from xxhdx1985126/wip-crimson-clear-temp-objects-startup

crimson/osd: clear ondisk temp objects on startup

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57562 from perezjosibm/wip-perezjos-cpu-map

script/: add cpu-map.sh to aid manual selection of CPU cores for threads for profiling

Merge pull request #55374 from aclamk/wip-aclamk-bs-multi-label

os/bluestore: Multiple bdev labels on main block device

Merge PR #59040 into main

* refs/pull/59040/head:
doc: documenting the feature that scrub clear the entries from damage table on repair

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

qa/cephfs: fix TestRenameCommand and unmount the clinet before failing fs

We should else bring and wait for MDS to be up since it is needed
for unmounting of CephFS in CephFSTestCase.tearDown() to be successful,
or just unmount the mountpoints before failing the filesystem.

Since the mountpoint won't be used in later tests so we just unmount
it.

Fixes: https://tracker.ceph.com/issues/66946
Signed-off-by: Xiubo Li <xiubli@redhat.com>

doc/cephadm/smb: Add missing user_sources option to service spec

Signed-off-by: Anoop C S <anoopcs@cryptolab.net>

crimson/os/seastore: report outstanding requests per phase

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: wait ool writes in DeviceSubmission phase

So that it is moved out of the collection lock.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #56782 from joscollin/wip-B65171-support-replication-start-end-notifications

cephfs_mirror: provide metrics for last successful snapshot sync

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57458 from lxbsz/wip-session-evict

qa/cephfs: add test_session_evict_non_blocklisted test case

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57987 from joscollin/wip-B64752-valgrind-report-leaks

qa/suites: drop --show-reachable=yes from fs:valgrind tests

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58097 from gardran/wip-gardran-kill-count

client: eliminate duplicate lookups wherever possible

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: documenting the feature that scrub clear the entries from damage table on repair

Fixes: https://tracker.ceph.com/issues/67337
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

doc/README.md - allocate information correctly

Add information in the "Building Ceph" procedure to the steps where it
belongs. Add information about "do_cmake.sh" to the step about
"do_cmake.sh". Add information about Ninja to the step about Ninja.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59060 from qn2060/patch-3

doc/dev/peering: Change acting set num

Merge PR #52939 into main

* refs/pull/52939/head:
mon/MonClient: handle ms_handle_fast_authentication return

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #58437 into main

* refs/pull/58437/head:
mds: remove the misleading and redundant logs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #58582 into main

* refs/pull/58582/head:
test: add a test for `fs set down` on online cluster
mon: fix `fs set down` to adjust max_mds only when cluster is not down

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58511 from pereman2/ch-umount

os/bluestore: assert CollectionRef count is 1 on umount

Merge pull request #57572 from pereman2/refactors

os/bluestore: Improve Readability of Extent Map Sharding

Merge pull request #59011 from xxhdx1985126/wip-67327

crimson/osd: send empty transactions to backfill targets that haven't backfilled the objects yet

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58708 from xxhdx1985126/wip-67068

crimson/osd/osd_operation/client_request: requeue client requests from a temporary queue other than ClientRequest::Orderer::list

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58148 from xxhdx1985126/wip-65696

crimson/osd/osd_operations: hang requests if the objects are unfound

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59051 from dang/wip-dang-67321

RGW - Zipper: pull in upstream fix for zpp_bits

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/dev/peering: Change acting set num

This commit updates the peering doc page with the changed [2,1,2] to [3,1,2]

Signed-off-by: qn2060 <qn2060@users.noreply.github.com>

Merge pull request #58971 from cbodley/wip-rm-boost-redis-again-again

submodule: remove the boost_redis submodule again (again)

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #58037 from Suyashd999/refractor-PrimaryLogPG

osd/PrimaryLogPG.cc bl length would be stored before moving

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

RGW - Zipper: pull in upstream fix for zpp_bits

zpp_bits included code that required a newere compiler.  Pull in the
upstream fix from:

https://github.com/eyalz800/zpp_bits/pull/170

Fixes: https://tracker.ceph.com/issues/67321
Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

Merge pull request #59024 from zdover23/wip-doc-2024-08-05-README-building-ceph

doc/README.md - add ordered list

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58282 from NitzanMordhai/wip-nitzan-daemonwatchdog-should-terminate-thrasher-when-bark

qa/tasks: watchdog should terminate thrasher

Merge pull request #58664 from aclamk/wip-aclamk-qa-less-bluestore-debug

qa/suites/rados: Reduced BlueStore log levels

script/: add cpu-map.sh to aid manual selection of CPU cores for threads for profiling

Signed-off-by: Jose J Palacios-Perez <perezjos@uk.ibm.com>

thrashers: standardize stop and join method names

Thrashers that do not inherit from ThrasherGreenlet previously used a
method called do_join, which combined stop and join functionality. To
ensure consistency and clarity, we want all thrashers to use separate
stop, join, and stop_and_join methods.

This commit renames methods and implements missing stop and stop_and_join
methods in thrashers that did not inherit from ThrasherGreenlet.

Fixes: https://tracker.ceph.com/issues/66698
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

qa/tasks: watchdog terminate thrasher

If a thrasher exception occurs, the do_dump_ops thread will continue
looping until the Teuthology timeout is reached.
The watchdog should terminate the thrasher to free up resources.

Fixes: https://tracker.ceph.com/issues/66698
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge PR #58500 into main

* refs/pull/58500/head:
ptl-tool: Fix token file name and variable names

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58963 from rhcs-dashboard/fix-67009-main

mgr/dashboard: Dashboard not showing Object/Overview correctly.

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #58746 from rhcs-dashboard/fix-67069-main

mgr/dashboard: make multi-cluster context switcher scrollable if we have many entries in it

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

cls/rbd: add functions to get group snap orders

Added functions to get the group snap order keys.

Signed-off-by: N Balachandran <nibalach@redhat.com>

crimson/os/seastore/transaction: cleanup, drop unused getters

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction: rename ool_block_lists

Drop the inaccurate written_ prefix because they may be still writting
when added to the lists.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/epm: RandomBlockOolWriter to update extents upon submitting writes

In order to move the ool write futures out of the collection lock, which
will be after the transaction prepare phase.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

qa: simplify postmerge construction

and avoid errors when "clusternodes" is not defined.

Fixes: https://tracker.ceph.com/issues/67352
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57905 from mcpreston/docs-dashboard-typo

doc/mgr/dashboard: fix TLS typo

qa/crontab: force priority for main rados nightlies

Followup to https://github.com/ceph/ceph/pull/58800/commits/3f62db0393b2d1516a447fd13d83f6bf26bbd99b.
The suite is large, so we need to force the priority. This is warranted
since the rados team makes a lot of use of the main runs.

Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #58767 from kshtsk/wip-python-invalid-escape-sequence

mgr: fix invalid escape sequence

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58373 from ceph/wip-tasks-ceph-mkfs-has-no-f

qa: ceph.py mkfs.ext4 has no -f

os/bluestore: BluestoreRepairTest ch cleanup

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

cls/rbd: save max group snap order

Save the last used group snap order value to speed up
snapshot creation.

Signed-off-by: N Balachandran <nibalach@redhat.com>

cls/rbd: save group snapshot creation order in a new key

In order to be able to list group snapshots in creation order, a
new snapshot order key is stored on snapshot creation.

[nbalacha: removed the group_snap_unsorted function,
reverted the group_snap_list function changes]

Signed-off-by: Mykola Golub <mgolub@suse.com>
Signed-off-by: N Balachandran <nibalach@redhat.com>

doc/README.md - add ordered list

Add ordered-list formatting to the "Building Ceph" section of README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58891 from zdover23/wip-doc-2024-07-28-cephfs-file-layouts-layout-fields

doc/cephfs: edit "Layout Fields" text

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

PeeringState.cc: Only populate want_acting when num_osds < bucket_max

Currently, in `calc_replicated_acting_stretch` we would
call `pop_ancester` before checking whether the next
OSD we populate in the want_acting set would have
the ancestor count that is >= bucket_max.

Fixes: https://tracker.ceph.com/issues/65371
Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #58115 from ronen-fr/wip-rf-memmodel

common: MemoryModel: performance improvements and API changes

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

librbd: Reduce use of atomics in librbd throttling

Signed-off-by: Adam Lyon-Jones <adamlyon@uk.ibm.com>

crimson/os/seastore/cache: introduce maybe_add_to_read_set()

And only touch extent if it is absent in transaction.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: verify pending extents in get_extent_viewable_by_trans()

Distinguish specific cases carefully.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57786 from Svelar/asan_cli_test

crush: avoid out-of-bound access and simplify enlarging buckets

crimson/os/seastore: consider segment_header_t::modify_time as the
segments' modify_time for no-tail OOL segments

OOL segments don't contain record headers anymore, we use
segment_header_t::modify_time as the approximation of segment
modify_time

Fixes: https://tracker.ceph.com/issues/67106
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/epm: SegmentedOolWriter to update ool paddr upon submitting writes

In order to move the ool write futures out of the collection lock, which
will be after the transaction prepare phase.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/record_submitter: return record-base immediately upon submit

For out-of-line write only.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/record_submitter: cleanup around write_base

No longer need to pass write_base through the write promise.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/record_submitter: minor cleanups

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/record_submitter: refactor to make write base available

Make sure the write base is available upon RecordSubmitter::submit().

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/record_submitter: refactor JournalAllocator::write()

Decouple get_written_to() so it can be queried upon writes.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #58718 from wanglinke521/wlk-kv-stats

kv/RocksDBStore: add kv_stats function to print all CF info

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #58470 from mohit84/issue_66867

mon: Remove any pg_upmap_primary mapping during remove a pool

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Josh Salomon <josh.salomon@gmail.com>

Merge pull request #58302 from jianwei1216/feat_add_mclock_perfcounter_main

osd/scheduler: add mclock queue length perfcounter

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

crimson/osd: send empty transactions to backfill targets that haven't
backfilled the objects yet

Fixes: https://tracker.ceph.com/issues/67327
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #59006 from zdover23/wip-doc-2024-08-03-glossary-ceph-ansible

doc/glossary: add "ceph-ansible"

doc/glossary: add "ceph-ansible"

Add a "ceph-ansible" entry to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58969 from gbregman/main

mgr/cephadm: Add SPDK log level to nvmeof configuration file

Merge pull request #58402 from rkachach/fix_issue_mtls_support

mgr/cephadm: adding mTLS for ceph mgmt-gateway and backend services communication

Reviewed-by: Adam King <adking@redhat.com>

mgr/cephadm: add SPDK log level to nvmeof configuration
Fixes https://tracker.ceph.com/issues/67258

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>