qa/suites/krbd: override extra_system_packages directly on install task

Follow commit c61cb16223fa ("qa/suites/rbd: override
extra_system_packages directly on install task").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: stress test for recovering from watch errors for -o exclusive

This is based on a test added in commit 237aa221ebad ("qa/suites/krbd:
stress test for recovering from watch errors") for regular mappings.

Fixes: https://tracker.ceph.com/issues/67097
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58703 from kshtsk/wip-fix-unused-in-erasure-code

erasure-code/clay: cleanup unused but set variable

Merge pull request #58215 from badone/wip-tracker-59380-admin-socket-injectfull

qa/suites/rados: Cancel injectfull to allow cleanup

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #57951 from mohit84/issue_66231

AsyncMessenger: Don't decrease l_msgr_active_connections if it is negative

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57797 from ajarr/wip-66011

rbd: add `group snap info` command

Reviewed-by: N Balachandran <nibalach@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #57485 from sseshasa/wip-fix-validator-osd-down-grace-tmout

qa/suites/rados/verify/validater: increase heartbeat grace timeout

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge PR #58276 into main

* refs/pull/58276/head:
mds/QuiesceDbManager: get requested state of members before iterating through each quiesce set members

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58709 from kshtsk/wip-assert-equal-please

mgr/dashboard: use assertEqual in test_iscsi

Merge pull request #58735 from yuvalif/wip-yuval-63786

test/cls_2pc_queue: fix multi-threaded access to non-atomic counter

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58745 from cyx1231st/wip-fix-blk-aio-compile-issue

blk/aio: fix compile issue when HAVE_LIBURING isn't defined

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/cls_2pc_queue: fix multi-threaded access to non-atomic variables

Fixes: https://tracker.ceph.com/issues/63786
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #58367 from xxhdx1985126/wip-seastore-lba-mapping-iterator

crimson/os/seastore/cached_extent: add the "refresh" ability to lba mappings

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

blk/aio: fix compile issue when HAVE_LIBURING isn't defined

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57782 from myoungwon/wip-nvme-pi

crimson/os/seastore: add checksum offload to RBM

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #58705 from gitkenan/crimsonDocChanges

doc/dev/crimson: making it more user-friendly, re-arranging chapters and fixing typos

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58275 from NitzanMordhai/wip-nitzn-host-thraser-fix-min-in-checks

suites: host thrasher should check min_in before thrashing host

Merge pull request #58393 from NitzanMordhai/wip-nitzan-old-client-thrasher-slow-ops-ignorelist

suites: thrash old client missing SLOW_OPS in the ignore-list

Merge pull request #58277 from NitzanMordhai/wip-nitzan-upgrade-suites-quincy-ignore-list-adding

suites/upgrade/quincy-x: update the ignore list

Merge pull request #58600 from zdover23/wip-doc-2024-07-16-dev-guide-basic-workflow

doc/dev: add formatting to basic workflow

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rbd: add group snap info command

... to show information about a group snapshot.

And also include group snap ID in `group snap ls` output.

Fixes: https://tracker.ceph.com/issues/66011
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58725 from zdover23/wip-doc-2024-07-23-dev-dev-guide-basic-workflow

doc/dev_guide: add needs-upgrade-testing label info

Reviewed-by: Laura Flores <lflores@ibm.com>

doc/dev_guide: add needs-upgrade-testing label info

Add an explanation of the "needs-upgrade-testing" label and explain when
to use it.

Laura is the true author of this text.

Co-authored-by: Laura Flores <lflores@ibm.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/dev/crimson: clarify and rearrange for userability

Signed-off-by: gitkenan <kenan.mth@gmail.com>

Merge pull request #56517 from gukaifeng/doc-amend

doc/start/os-recommendations: remove 16.2.z support for CentOS 7

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #58353 from pereman2/bad-collection-objectstore

tools/objectstore: check for wrong coll open_collection

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57002 from aclamk/wip-aclamk-bs-storetest-expand-synthetic

Improved structure for objectstore unit tests.

doc/start/os-recommendations: remove 16.2.z support for CentOS 7

Merge pull request #58639 from zdover23/wip-doc-2024-07-17-start-os-recommendations

doc/start: add tested container host oses

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: add tested container host oses

Add a list of OSes that have been tested as container hosts. This commit
is added in response to a request made by Josh Durgin on Slack.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: use assertEqual in test_iscsi

Python asks in multiple places to use assertEqual instead of
assertEquals:

  DeprecationWarning: Please use assertEqual instead.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

crimson/os/seastore/btree: drop unused lines

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cached_extent: add the "refresh" ability to lba
mappings

So that we don't have to re-iterate through the whole lba tree to get
a "parent-viewable" mapping in most cases.

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: rename PhysicalNodeMapping::is_parent_valid() to
PhysicalNodeMapping::is_parent_viewable()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/btree: add copy_source->pending links to FixedKVNode

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cbj: skip crc calculation for cbj header if device supports the checksum offload

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: disable crc calculation if end to end data protection is enabled

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: add handle_exception() to pass_through_io()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: remove multistream related codes

Current codes allow the device to allocate multiple namespace without specific policy
if the nvme device report that it is capable of mutistream functionality.
So, this commit removes the multistream related code, leaving it as a TODO.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: add nvme_read and nvme_write to be used when end to end data protection is enabled

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/rbm: turn on end-to-end-data-protection during mkfs if possible

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: replace rbm_metadata_header_t with rbm_superblock_t

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

erasure-code/clay: cleanup unused but set variable count_retrieved_sub_chunks

Addresses clang 17 error:

/home/rocky/ceph/src/erasure-code/clay/ErasureCodeClay.cc:481:7: error: variable 'count_retrieved_sub_chunks' set but not used [-Werror,-Wunused-but-set-variable]

Signed-off-by: kshtsk <kyrylo.shatskyy@clyso.com>

Merge pull request #58452 from Matan-B/wip-matanb-seastar-july7

seastar: update submodule

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #58256 from NitzanMordhai/wip-nitzan-wait-osd-admin-command

crimson/osd: adding osdmap subscribe

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58700 from kshtsk/wip-fix-unused-in-msg-async-rdma

msg/async/rdma: cleanup unused variables

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #58688 from idryomov/wip-67051

qa/workunits/rbd: avoid caching effects in luks-encryption.sh

Reviewed-by: Mykola Golub <mgolub@suse.com>

msg/async/rdma: cleanup unused variables

Fixes: 2fb8be43fa2a758cc7934f7a82a48ad2d47d1112
Signed-off-by: kshtsk <kyrylo.shatskyy@clyso.com>

Merge pull request #58695 from pecastro/correct_mailing_list_instructions

doc: Amend dev mailing list subscribe instructions.

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc: Amend dev mailing list subscribe instructions.

Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com>

Merge pull request #58591 from idryomov/wip-aio-completion-prep-for-atomic-wait

librbd: fix inconsistency between AioCompletion is_complete() and wait_for_complete()

Reviewed-by: N Balachandran <nibalach@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58180 from guojidan/format

rbd: add the validate of the format and clone_format

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

qa/workunits/rbd: avoid caching effects in luks-encryption.sh

Commit 40f6f5224bce ("qa/workunits/rbd: fix issues in
luks-encryption.sh") did the right thing for reads, which solved
most of the issue.  However, it actually made a step in the opposite
direction for writes -- depending on the RBD cache settings, rbd-nbd
virtual devices can behave as physical devices with a volatile write
cache, so fsync is required.

While at it, involving O_DIRECT for reads isn't needed outside of
test_encryption_format().

Fixes: https://tracker.ceph.com/issues/67051
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52639 from ljflores/wip-tracker-58925

qa: suppress Leak_StillReachable mon leak in centos 9 jobs

Merge pull request #58671 from zdover23/wip-doc-2024-07-19-start-get-involved

doc/start: update mailing list links

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #58678 from cbodley/wip-doc-cephadm-quay-ci

doc/cephadm/upgrade: ceph-ci containers are hosted at quay.ceph.io

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

doc/cephadm/upgrade: ceph-ci containers are hosted by quay.ceph.io

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57737 from Svelar/asan_rgw_amqp

rgw/amqp: lock erase and create connection before emplace

Merge pull request #58662 from yuvalif/wip-yuval-64690-again

test/rgw/amqp: skip TestAMQP.MaxConnections test

Merge pull request #58656 from rhcs-dashboard/rgw-sync-policy-routing-fix

mgr/dashboard: Rgw sync policy routing fix

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #56331 from rhcs-dashboard/update-mon-stacks

mgr/cephadm: bump monitoring stacks versions

doc/start: update mailing list links

Update the User and Dev mailing list subscription links. This commit is
made in response to a request from Dan van der Ster.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58636 from afreen23/wip-nvmeof-namespace

mgr/dashboard: Add namespaces views in dashboard

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #58519 from smanjara/wip-fix-copy-obj-replication

rgw/multisite: don't retain RGW_ATTR_OBJ_REPLICATION_TRACE attr on copy_object

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56937 from Svelar/asan_bluestore_type

bluestore/bluestore_types: avoid heap-buffer-overflow in another way to keep code uniformity

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #56352 from ifed01/wip-ifed-many-many-extents-read

blk/aio: fix long batch (64+K entries) submission.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #54988 from Matan-B/wip-matanb-mapgap-fix

osd/OSD: rewrite track_pools_and_pg_num_changes logic

Reviewed-by: Samuel Just <sjust@redhat.com>

qa: suppress Leak_StillReachable mon leaks

These leaks came about when we changed the distro
from centos 8 to centos 9.

Fixes: https://tracker.ceph.com/issues/61774
Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #57817 from phlogistonjohn/jjm-smb-sqlite

mgr/smb: add sqlite internal store backend for smb mgr module

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #58518 from anoopcs9/smb-mgr-updates

mgr/smb: improvements to smb mgr module and docs

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

mgr/dashboard: Add namespaces views in dashboard

Fixes https://tracker.ceph.com/issues/66990

- list namespaces
- create namespaces
- edit namespaces
- delete namespaces
- update NQN regex

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #58644 from rhcs-dashboard/fix-66994-main

mgr/dashboard: Add a new chart for replication delta per shard in rgw sync overview grafana dashboard

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #58649 from cbodley/wip-64173

rgw/rados: set_attrs() falls back to existing attrs for index update

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

mgr/dashboard: rgw sync policy routing fixes

Fixes: https://tracker.ceph.com/issues/66977
Signed-off-by: Naman Munet <nmunet@redhat.com>

Merge pull request #58173 from ronen-fr/wip-rf-targets-j13

osd/scrub: no shared scrub-job ownership between PGs and the scrub queue

Reviewed-by: Samuel Just <sjust@redhat.com>

test/rgw/amqp: skip TestAMQP.MaxConnections test

not really a fix, mainly a workaround until we do
the amqp refactoring: https://tracker.ceph.com/issues/67011

Fixes: https://tracker.ceph.com/issues/64690
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #58555 from rhcs-dashboard/rgw-multisite-sync-pipe

 mgr/dashboard: RGW multisite sync pipe

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #58648 from afreen23/wip-nvmeof-listener

Allow listeners creation and deletion

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #58551 from xxhdx1985126/wip-66924

crimson/os/seastore/transaction_manager: consider inconsistency between backrefs and lbas acceptable when cleaning segments

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #58441 from rhcs-dashboard/add-nfs-export-route

mgr/dashboard: add NFS export button for subvolume/ grp

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #57591 from aclamk/wip-aclamk-denc-compat-check

common, os/bluestore: Fix lack of checking for compat on DENC_START

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Create and delete listeners

Fixes https://tracker.ceph.com/issues/66996

- list listener under subsystems
- delete listener
- create listener

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

mgr/dashboard: RGW multisite sync pipe

Fixes: https://tracker.ceph.com/issues/66926
Signed-off-by: Naman Munet <nmunet@redhat.com>

rgw/rados: set_attrs() falls back to existing attrs for index update

set_attrs() needs the ACLOwner, etag, content-type and storage class for
the bucket index update. if those attrs aren't being updated, consult
their existing attrs in state->attrset

Fixes: https://tracker.ceph.com/issues/64173
Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: add NFS export route from subvolume/subvolume grp

Signed-off-by: Avan Thakkar <athakkar@redhat.com>

mgr/dashboard: Add a new chart for replication delta per shard in rgw sync overview grafana dashboard

Fixes: https://tracker.ceph.com/issues/66994
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #57722 from sajibreadd/wip-62500

os/bluestore: Warning added for slow operations and stalled read

librbd: get rid of AIO_STATE_CALLBACK in AioCompletion

After commit 002afa0fe375 ("librbd: avoid using lock within AIO
completion where possible"), the only method whose behavior would
change if AIO_STATE_CALLBACK is removed is is_complete() and it
actually needs fixing anyway: because of state != AIO_STATE_PENDING
test, is_complete() returns true both for AIO_STATE_CALLBACK and
AIO_STATE_COMPLETE, while wait_for_complete() still blocks on
AIO_STATE_CALLBACK and returns only on AIO_STATE_COMPLETE.  These
methods back public APIs, so this inconsistency is exposed to users.

If we move to setting state to AIO_STATE_COMPLETE at the top of
mark_complete_and_notify() (i.e. before event socket notification), the
transient state for callbacks can be eliminated entirely and the
inconsistency goes away.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58571 from zdover23/wip-doc-2024-07-13-cephfs-mount-using-fuse

doc/cephfs: improve mount-using-fuse

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: improve mount-using-fuse

Improve the first twenty percent of doc/cephfs/mount-using-fuse.rst.
This commit changes the text so that it is clearer that FUSE is an
alternative to mounting CephFS with the kernel driver. This commit also
links to the kernel driver page from the "FUSE-mount" page.

Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/transaction_manager: consider inconsistency between
backrefs and lbas acceptable when cleaning segments

Consider the following scene:
1. Trans.A writes the final record of Segment S, in which it overwrite
   another extent E in the same segment S;
2. Before Trans.A "complete_commit", Trans.B tries to rewrite new
   records and roll the segments, which closes Segment S;
3. Before Trans.A "complete_commit", a new cleaner Transaction C tries
   to clean the segment;

In this scenario, C might see a part of extent E's laddr space mapped
to another location within the same segment S. This is actually a valid
case.

Fixes: https://tracker.ceph.com/issues/66924
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #57169 from jagombar/fixCpatch

cpatch: Search base image when patching files.

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #57906 from kamoltat/wip-ksirivad-stretch-mode-netsplit-test

mon/ElectionLogic: tie-breaker mon ignore proposal from marked down mon

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #57592 from Svelar/asan_ec_shec_t

ec: free plugin memory when it destructs and repeated loads

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #57201 from NitzanMordhai/wip-nitzan-standalone-pg-split-merge-daemon-commands

test: ceph daemon command with asok path

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #56982 from NitzanMordhai/wip-nitzan-neorados-watch-notify-test-reconnect

test/neorados: timeout test won't reconnect at timeout

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #56605 from myoungwon/wip-65241

osd: fix use-after-free before calling finish_unreadable_object()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #58354 from ivancich/wip-appropriate-min-objs-shard

rgw: make sure min objs per shard is appropriate

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58442 from clwluvw/getlocation-contenttype

rgw: add missing content-type for RGWGetBucketLocation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57640 from cbodley/wip-rgw-beast-realm-reload

rgw/beast: realm reload closes connections so requests drain quickly

Reviewed-by: Oguzhan Ozmen <oozmen@bloomberg.net>