os/bluestore: Fix unbalanced new[]/delete[] in read_bdev_label functions

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Modify read_meta()

When bdev is not opened, do not fallback to file-backed values
immediately.
First try to open temporary open bdev.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: fix fsck fixing multiple bdev label (superblock)

The problem was that after fixing label location,
location was not marked as "good label".
In result the region was not excluded when saving allocator state.
After reading allocator state, label was treated as corrupted.

Fixed fixing bdev label when freelist is bitmap (HDD).

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Review fixes

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

test/store_test: Add double check for repair

Add fsck after repair to verify.
Also drop some unnecesary calls to mount().

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

ceph-bluestore-tool: Fix set-label-key and rm-label-key

Functions were not adapted to multiple superblocks.
Now for superblock labels that handle multiple copies, all are updated.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: multiple bdev label (superblock) fixes

- fixed logic of bluestore_debug_permit_any_bdev_label
- minor fixes

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Adapt unittest for multi bdev label

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Small fix to mkfs

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Made read/write_meta bdev aware

When bdev is not set, do not even try to operate on label.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add read/write_bdev_label functions

Made _read/_write_bdev_label private.
Instead created read/write_bdev_label.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Modify bdev-label functions operate on bdev

Now bdev-label related function operate on BlockDevices.
It used to open own file descriptor to operate.
It could no longer be supported because we need bdev->get_size().

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Bluefs, expand api for getting BlockDevice on BD/WAL

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Minor fixes

These were noticed during review.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Update test for resilience of bdev label against bad UUID

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Only accept bdev labels with our UUID

This fixed the problem that some saboteur label of different UUID
with high "epoch" will trick label discovery.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Split check* and set* _main_bdev_label

Splitting makes them clearer.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Adapt multi bdev label tests for !bdev->supported_bdev_label()

Just skip tests.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Make bdev multi label compatible with !bdev->supported_bdev_label()

Some devices (spdk, nvme) claim that they do not support bdev label.
It does not make sense since it just means ability to write [0-4k] on the device.
Regardless, special care is taken of the devices that do not have label written.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix of bdev label fsck

Procedure that verifies allocations vs freelist was not aware of bdev labels.
Now it properly omits them in comparisons between free and freelist.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Tests for multi bdev label feature

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix of funny constructor misuse

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix problem with marking unavailable bdev label positions

It was attempted to mark bdev label locations as taken before adding bluefs allocation on main.
This in turn caused collision when bluefs finally tried to mark regions.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix repair for multi label

When there was no bdev labels to repair, assert happened.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add ability to upgrade into multi label

By default original behaviour is preserved.
One can upgrade into multi label by:

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Hack for fsck with multi label

Config bluestore_bdev_label_require_all interferes with fsck / repair.
For those functions disable the flag.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix bdev_label_main_try_reserve

It added valid_locations which it should not been doing.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix write_bdev_label

Fix default value of parameter 'location'.
It should consist of one element BDEV_LABEL_POSITION (0),
not just create with size 0.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: fix DB_SUPER_RESERVED->SUPER_RESERVED

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Small fix to check_or_set_bdev_label

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix write_meta; increase epoch with each write

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix fsck() for multi label

Do not operate if bdev label is outside device size.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix _check_or_set_main_bdev_label

Change use of local label to BlueStore's bdev_label.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix _main_bdev_label_try_reserve

When location of label is acceptable add it as proper position for use.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: minor fix in _check_or_set_main_bdev_label

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix read_main_bdev_label

When valid label with new epoch is found,
all that we have marked as good before are released.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix write_bdev_label

If required location is outside device size, just skip it.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Fix read_meta and write_meta

Remove asserts if optional reading of bdev label fails.
Pass all valid bdev locations when writing bdev labels.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add fsck procedure for bdev multi labels

Now fsck can properly detect collision between labels and object data / bluefs files.
Additional labels have lower precedence, they never overwrite other data.
If collision label - object data happens, the object is moved somewhere else.
If collision label - bluefs file happens, it is left unsolved.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Adapt read_meta / write_meta

Modified to align with multi label properly.
Now functions still cache, but use read_main_bdev_label.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Give label multiple positions to replicate to

Bdev label for main device can now be present in multiple locations.
The locations of valid labels are memorized and only those locations are used.
This is to preserve from overwriting data, should collision label - object or bluefs occur.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Create read_bdev_main_label function

Duplicate read_bdev_label into dedicated read_bdev_main_label.
New function reads multiple labels.
Also created check_or_set_main_bdev_label in similiar way.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Modify read/write_bdev_label functions

Modify read_bdev_label and write_bdev_label functions to operate on any disk location.
Default falls back to original position 0, which is now named BDEV_LABEL_POSITION.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Remove bluestore_debug_prefill

Remove outdated and useless debug setting.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Make read_meta / write_meta cached

Modified read_meta and write_meta to keep current bdev_label cached,
instead of reading it all over again.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #58353 from pereman2/bad-collection-objectstore

tools/objectstore: check for wrong coll open_collection

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57002 from aclamk/wip-aclamk-bs-storetest-expand-synthetic

Improved structure for objectstore unit tests.

Merge pull request #58639 from zdover23/wip-doc-2024-07-17-start-os-recommendations

doc/start: add tested container host oses

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: add tested container host oses

Add a list of OSes that have been tested as container hosts. This commit
is added in response to a request made by Josh Durgin on Slack.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58452 from Matan-B/wip-matanb-seastar-july7

seastar: update submodule

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #58256 from NitzanMordhai/wip-nitzan-wait-osd-admin-command

crimson/osd: adding osdmap subscribe

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58700 from kshtsk/wip-fix-unused-in-msg-async-rdma

msg/async/rdma: cleanup unused variables

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #58688 from idryomov/wip-67051

qa/workunits/rbd: avoid caching effects in luks-encryption.sh

Reviewed-by: Mykola Golub <mgolub@suse.com>

msg/async/rdma: cleanup unused variables

Fixes: 2fb8be43fa2a758cc7934f7a82a48ad2d47d1112
Signed-off-by: kshtsk <kyrylo.shatskyy@clyso.com>

Merge pull request #58695 from pecastro/correct_mailing_list_instructions

doc: Amend dev mailing list subscribe instructions.

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc: Amend dev mailing list subscribe instructions.

Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com>

Merge pull request #58591 from idryomov/wip-aio-completion-prep-for-atomic-wait

librbd: fix inconsistency between AioCompletion is_complete() and wait_for_complete()

Reviewed-by: N Balachandran <nibalach@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58180 from guojidan/format

rbd: add the validate of the format and clone_format

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

qa/workunits/rbd: avoid caching effects in luks-encryption.sh

Commit 40f6f5224bce ("qa/workunits/rbd: fix issues in
luks-encryption.sh") did the right thing for reads, which solved
most of the issue.  However, it actually made a step in the opposite
direction for writes -- depending on the RBD cache settings, rbd-nbd
virtual devices can behave as physical devices with a volatile write
cache, so fsync is required.

While at it, involving O_DIRECT for reads isn't needed outside of
test_encryption_format().

Fixes: https://tracker.ceph.com/issues/67051
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52639 from ljflores/wip-tracker-58925

qa: suppress Leak_StillReachable mon leak in centos 9 jobs

Merge pull request #58671 from zdover23/wip-doc-2024-07-19-start-get-involved

doc/start: update mailing list links

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #58678 from cbodley/wip-doc-cephadm-quay-ci

doc/cephadm/upgrade: ceph-ci containers are hosted at quay.ceph.io

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

doc/cephadm/upgrade: ceph-ci containers are hosted by quay.ceph.io

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57737 from Svelar/asan_rgw_amqp

rgw/amqp: lock erase and create connection before emplace

Merge pull request #58662 from yuvalif/wip-yuval-64690-again

test/rgw/amqp: skip TestAMQP.MaxConnections test

Merge pull request #58656 from rhcs-dashboard/rgw-sync-policy-routing-fix

mgr/dashboard: Rgw sync policy routing fix

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #56331 from rhcs-dashboard/update-mon-stacks

mgr/cephadm: bump monitoring stacks versions

doc/start: update mailing list links

Update the User and Dev mailing list subscription links. This commit is
made in response to a request from Dan van der Ster.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58636 from afreen23/wip-nvmeof-namespace

mgr/dashboard: Add namespaces views in dashboard

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #58519 from smanjara/wip-fix-copy-obj-replication

rgw/multisite: don't retain RGW_ATTR_OBJ_REPLICATION_TRACE attr on copy_object

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56937 from Svelar/asan_bluestore_type

bluestore/bluestore_types: avoid heap-buffer-overflow in another way to keep code uniformity

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #56352 from ifed01/wip-ifed-many-many-extents-read

blk/aio: fix long batch (64+K entries) submission.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #54988 from Matan-B/wip-matanb-mapgap-fix

osd/OSD: rewrite track_pools_and_pg_num_changes logic

Reviewed-by: Samuel Just <sjust@redhat.com>

qa: suppress Leak_StillReachable mon leaks

These leaks came about when we changed the distro
from centos 8 to centos 9.

Fixes: https://tracker.ceph.com/issues/61774
Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #57817 from phlogistonjohn/jjm-smb-sqlite

mgr/smb: add sqlite internal store backend for smb mgr module

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #58518 from anoopcs9/smb-mgr-updates

mgr/smb: improvements to smb mgr module and docs

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

mgr/dashboard: Add namespaces views in dashboard

Fixes https://tracker.ceph.com/issues/66990

- list namespaces
- create namespaces
- edit namespaces
- delete namespaces
- update NQN regex

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #58644 from rhcs-dashboard/fix-66994-main

mgr/dashboard: Add a new chart for replication delta per shard in rgw sync overview grafana dashboard

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #58649 from cbodley/wip-64173

rgw/rados: set_attrs() falls back to existing attrs for index update

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

mgr/dashboard: rgw sync policy routing fixes

Fixes: https://tracker.ceph.com/issues/66977
Signed-off-by: Naman Munet <nmunet@redhat.com>

Merge pull request #58173 from ronen-fr/wip-rf-targets-j13

osd/scrub: no shared scrub-job ownership between PGs and the scrub queue

Reviewed-by: Samuel Just <sjust@redhat.com>

test/rgw/amqp: skip TestAMQP.MaxConnections test

not really a fix, mainly a workaround until we do
the amqp refactoring: https://tracker.ceph.com/issues/67011

Fixes: https://tracker.ceph.com/issues/64690
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #58555 from rhcs-dashboard/rgw-multisite-sync-pipe

 mgr/dashboard: RGW multisite sync pipe

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #58648 from afreen23/wip-nvmeof-listener

Allow listeners creation and deletion

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #58551 from xxhdx1985126/wip-66924

crimson/os/seastore/transaction_manager: consider inconsistency between backrefs and lbas acceptable when cleaning segments

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #58441 from rhcs-dashboard/add-nfs-export-route

mgr/dashboard: add NFS export button for subvolume/ grp

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #57591 from aclamk/wip-aclamk-denc-compat-check

common, os/bluestore: Fix lack of checking for compat on DENC_START

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Create and delete listeners

Fixes https://tracker.ceph.com/issues/66996

- list listener under subsystems
- delete listener
- create listener

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

mgr/dashboard: RGW multisite sync pipe

Fixes: https://tracker.ceph.com/issues/66926
Signed-off-by: Naman Munet <nmunet@redhat.com>

rgw/rados: set_attrs() falls back to existing attrs for index update

set_attrs() needs the ACLOwner, etag, content-type and storage class for
the bucket index update. if those attrs aren't being updated, consult
their existing attrs in state->attrset

Fixes: https://tracker.ceph.com/issues/64173
Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: add NFS export route from subvolume/subvolume grp

Signed-off-by: Avan Thakkar <athakkar@redhat.com>

mgr/dashboard: Add a new chart for replication delta per shard in rgw sync overview grafana dashboard

Fixes: https://tracker.ceph.com/issues/66994
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #57722 from sajibreadd/wip-62500

os/bluestore: Warning added for slow operations and stalled read

librbd: get rid of AIO_STATE_CALLBACK in AioCompletion

After commit 002afa0fe375 ("librbd: avoid using lock within AIO
completion where possible"), the only method whose behavior would
change if AIO_STATE_CALLBACK is removed is is_complete() and it
actually needs fixing anyway: because of state != AIO_STATE_PENDING
test, is_complete() returns true both for AIO_STATE_CALLBACK and
AIO_STATE_COMPLETE, while wait_for_complete() still blocks on
AIO_STATE_CALLBACK and returns only on AIO_STATE_COMPLETE.  These
methods back public APIs, so this inconsistency is exposed to users.

If we move to setting state to AIO_STATE_COMPLETE at the top of
mark_complete_and_notify() (i.e. before event socket notification), the
transient state for callbacks can be eliminated entirely and the
inconsistency goes away.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58571 from zdover23/wip-doc-2024-07-13-cephfs-mount-using-fuse

doc/cephfs: improve mount-using-fuse

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: improve mount-using-fuse

Improve the first twenty percent of doc/cephfs/mount-using-fuse.rst.
This commit changes the text so that it is clearer that FUSE is an
alternative to mounting CephFS with the kernel driver. This commit also
links to the kernel driver page from the "FUSE-mount" page.

Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/transaction_manager: consider inconsistency between
backrefs and lbas acceptable when cleaning segments

Consider the following scene:
1. Trans.A writes the final record of Segment S, in which it overwrite
   another extent E in the same segment S;
2. Before Trans.A "complete_commit", Trans.B tries to rewrite new
   records and roll the segments, which closes Segment S;
3. Before Trans.A "complete_commit", a new cleaner Transaction C tries
   to clean the segment;

In this scenario, C might see a part of extent E's laddr space mapped
to another location within the same segment S. This is actually a valid
case.

Fixes: https://tracker.ceph.com/issues/66924
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #57169 from jagombar/fixCpatch

cpatch: Search base image when patching files.

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #57906 from kamoltat/wip-ksirivad-stretch-mode-netsplit-test

mon/ElectionLogic: tie-breaker mon ignore proposal from marked down mon

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #57592 from Svelar/asan_ec_shec_t

ec: free plugin memory when it destructs and repeated loads

Reviewed-by: Kefu Chai <tchaikov@gmail.com>