os/bluestore: remove wrong assert

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: Fix using 'near' hint in cache

We can now get near hint that refers to buffer that is in writing state.
Such buffer is not in cache yet and cannot provide in-queue location.
We only extact cache_private.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: fix buffercacheshard store arg bluestore_tyeps

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: get rid off global Writings and use TransContext instead

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

test/store_test: wait for completion not sleep.

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

os/bluestore: get rid off TransContext::seq

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

os/bluestore: introduce centralized BlueStore::Writings entity

Which performs tracking of being-in-write Buffer instances
and hence takes this reliability from BufferSpace.

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

test/store_test: add DeferredAndClone test case

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

os/bluestore: refactor maybe_prune_tail can_prune_tail

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: fix small write deferred buffer_cache_write

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

Revert "os/bluestore: fix bad buffer_cache_write"

This reverts commit eff64c1095fa7567a6590368e348ef3ec11f632c.

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: add missing move(bl) in BufferSpace::did_read

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: remove redundant bufferspace discard in truncate

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: fix bad buffer_cache_write

we can just cache write before padding or extra reading to decrease
complexity

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: remove BufferSpace::split

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: onode::finish_write

Use same finish_write->_finish_write tandem from blob->bufferspace now
with onode->bufferspace instead of a single finish_write function.

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: maybe_prune_tail

previously, discard_unallocated pruned tail. Now we have an explicit
maybe_prune_tail procedure

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: remove length from did_read

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: bring back cache change protection in finish_write

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: add comment explaining BufferSpace::_discard loop

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: remove blob from buffer_cache_write arguments

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: remove cache lock from can_split

since we don't base splitting blob un buffer space, we don't need to
lock there anymore

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: remove discard_unallocated

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: BufferSpace::read assert b->end() > offset

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: change shared_blob references to getter

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: split blob remove writing check

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: move all buffers when splittting cache

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: fix overlap function

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: deferred seq dependency lock

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: fix dup_writing offsets after rebase

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: lock collection on deferred _finish_write

Leaving deferred map unprotected caused race conditions that must be
dealt for correctness. This might impact performance.

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: use blob_offset in compressed read results

Using `front` as offset for compressed blobs was awfully incorrect and
the required was blob_offset.

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: check collection nullptr on deletion dummy onode

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore remove duplicated buffer_cache_write and bounds check

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: use dependant onode collection's cache

Fix to use onode_dependant cache instead of using onode

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: BufferSpace clear on Onode dctor

Previously we cleared BufferSpace on Blob's destructor and now it should
be in Onode destructor.

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: acquire cache lock on _finish_write

There were race conditions while finishing buffer writes due to a
missing lock. This adds that lock_guard to protect cache in _finish_write

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: undo cache audit removed from lru cache

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

os/bluestore: BufferSpace per Onode

Instead of indexing buffers by blob offset we do it by logical offset.
Previously we tried using physical offsets but it was a pretty bad idea
is it required extra bound checks of physical extents and extra loops
for to add them, that includes chopping off buffer data.

Logical offsets don't have these problems but we have different ones.
Bufferlist data is padded before being used in `_do_alloc_write` which
means before adding it to BufferSpace, we are required to get the
unpadded substr_of data because padded data might overflow to an invalid
logical offset in contrast to when we used blob offsets where padded
data was wonderful. This is true in in `generate_read_result_bl` too, we
read by regions of blob extents but we must translate it back to logical
offsets, that means an extra loop to include correct logical regions was

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #56860 from clwluvw/rgw-crypt-ssl

rgw: eliminate ssl enforcement for sse-s3 encryption

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56868 from clwluvw/request-payer

rgw: exclude logging of request payer for 403 requests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56394 from cbodley/wip-rgw-cache-size

rgw: increase default metadata cache size for accounts

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #56606 from guojidan/num_bytes

crimson/osd/ops_executer: fix snap overlap range error

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #56942 from Svelar/fix_tox_cephadm

test/cephadm: extend timeout again

Merge pull request #57057 from adk3798/no-downgrade

doc/cephadm: remove downgrade reference from upgrade docs

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #56644 from tchaikov/wip-warning-not-fatal

test: do not consider warning from Sanitizer as fatal

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55950 from ifed01/wip-ifed-omap-pcounters

os/bluestore: add omap_set* perf counters

Reviewed-by: akupczyk@ibm.com
Reviewed-by: rfriedma@ibm.com

Merge pull request #57071 from zdover23/wip-doc-2024-04-24-rados-troubleshooting-pg

doc/rados: improve t-shooting pg

Merge pull request #57051 from zdover23/wip-doc-2024-04-23-rados-dual-stack

doc/rados: remove dual-stack docs

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: improve t-shooting pg

Incorporate Anthony D'Atri's suggestions from
https://github.com/ceph/ceph/pull/57022 into the text in
doc/rados/troubleshooting/troubleshooting-pg.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56745 from ronen-fr/wip-rf-pgtell

tests/scripts: use 'tell pg deep-scrub pgid' instead of 'tell pgid deep-scrub'

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #56420 from rzarzynski/wip-tools-rados-get-with-offset

tools: rados get takes --offset now

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #56067 from myoungwon/wip-ceph-dedup-daemon

tool: separating ceph-dedup-tool into tool and daemon

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55949 from NitzanMordhai/wip-nitzan-safe-to-destroy-test-build-vars

test/osd/safe-to-destroy.sh: Add checks for the test

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54979 from pdvian/wip-mgr-optracker

mgr: integrate optracker in ceph-mgr

Merge pull request #57053 from idryomov/wip-labeler-rbd

.github: sync the list of paths for rbd label with CODEOWNERS

Reviewed-by: Laura Flores <lflores@redhat.com>

doc/cephadm: remove downgrade reference from upgrade docs

This has been in here for years, but cephadm will block
attempted upgrades to lower versions and we generally
don't want people to think this is supported or safe.

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #56986 from cbodley/wip-65551

rgw/account: add bucket_quota to RGWAccountInfo

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Oguzhan Ozmen <oozmen@bloomberg.net>

Merge pull request #57038 from adk3798/mixin-max-nvme-gws-per-group

monitoring/ceph-mixin: set NVMeoFMaxGatewaysPerGroup to 4

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #56990 from cbodley/wip-doc-rgw-realm-change

doc/rgw: add release note for changes to rgw_realm init

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #55983 from lxbsz/wip-64572

qa/fsx: use the master branch to build the xfstest-dev

.github: sync the list of paths for rbd label with CODEOWNERS

With https://github.com/actions/labeler/issues/483 still pending, do it
manually.

A couple of entries matching both files and directories were expanded
since, unlike fnmatch(3), minimatch treats slashes specially:

- src/include/rbd* -> src/include/rbd* src/include/rbd/**
- src/rbd*         -> src/rbd* src/rbd*/**
- src/test/rbd*    -> src/test/rbd* src/test/rbd*/**

A bunch of entries under src/tools were combined into:

- src/tools/rbd*/**

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

CODEOWNERS: assign ceph-rbd-mirror systemd service to RBD

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/rados: remove dual-stack docs

Remove references to dual-stack mode in
doc/rados/configuration/network-config-ref.rst and
doc/rados/configuration/msgr2.rst. This feature seems to have been
planned but never to have been completely implemented.

See the tracker issue listed below for an email exchange detailing the
confusion caused by the presence in the documentation of this
now-removed information.

Fixes: https://tracker.ceph.com/issues/65631
Signed-off-by: Zac Dover <zac.dover@proton.me>

test/cephadm: extend timeout limit again to fix timeout issue

When run make check(arm64) on omani series servers,
Shows:
python: timeout after 60 seconds
INFO     root:call_wrappers.py:284 Non-zero exit code 124 from /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/cephadm/.tox/py3/bin/python -c for i in range(1000000): print(i, flush=True)

Seems like 60s is not enough, extend to 300s would be fine.

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Merge pull request #57033 from gbregman/main

mgr/cephadm: Some changes to nvmeof conf file defaults

Merge PR #54067 into main

* refs/pull/54067/head:
mds: add dispatch killpoint and delay configs

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #56047 into main

* refs/pull/56047/head:
mgr: process map before notifying clients

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #56638 into main

* refs/pull/56638/head:
qa: do not iterate list being modified
qa: remove unnecessary background job cleanup

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56669 into main

* refs/pull/56669/head:
qa: make teardown behavior consistent for kclient/fuse mounts

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56723 into main

* refs/pull/56723/head:
mds: use mds_cache_quiesce_decay_rate to init quiesce_counter

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #57010 into main

* refs/pull/57010/head:
qa: update quiesce tests to expect ipolicy lock
mds: add missing policylock to test F_QUIESCE_BLOCK

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57022 from zdover23/wip-doc-2024-04-22-rados-operations-pg-troubleshooting

doc/rados: remove redundant pg repair commands

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw: apply default quota config on account creation

add new default quota config options for accounts analogous to
rgw_user_default_quota_max_objects/size. apply the default bucket quota
config options as-is

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57032 from stackhpc/doc-update-ms-bind-port-max

doc/rados: fix outdated value for ms_bind_port_max

Merge pull request #57036 from stackhpc/fix-typo-radosgw-metrics

doc/radosgw: fix typo in metrics.rst

monitoring/ceph-mixin: set NVMeoFMaxGatewaysPerGroup to 4

Recommendation from the nvmeof team

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #56448 from Svelar/arm_pass_check_shard_select

test/test_mempool: skip check_shard_select test on ARM64

doc/radosgw: fix typo in metrics.rst

Signed-off-by: Pierre Riteau <pierre@stackhpc.com>

mgr/cephadm: some changes to nvmeof conf file defaults
Fixes https://tracker.ceph.com/issues/65566

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

python-common/ceph/deployment: some changes to nvmeof conf file defaults
Fixes https://tracker.ceph.com/issues/65566

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

doc/rados: fix outdated value for ms_bind_port_max

The highest port number used by OSD or MDS daemons was increased from
7300 to 7568 in [1] but the documentation still refers to 7300 in
multiple locations.

[1] https://github.com/ceph/ceph/pull/42210

Fixes: https://tracker.ceph.com/issues/65609
Signed-off-by: Pierre Riteau <pierre@stackhpc.com>

Merge pull request #57016 from zdover23/wip-doc-2024-04-21-rados-operations-crush-map

doc/rados: add bucket rename command

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: remove redundant pg repair commands

Incorporate the material in /doc/rados/operations/pg-repair into
/doc/rados/troubleshooting/troubleshooting-pg. Remove
/doc/rados/operations/pg-repair from the documentation. Redirect all
links to the old location to the new location.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados: add bucket rename command

Add "ceph osd crush rename bucket" command. This commit is made in
response to a request from Michele Giacomoli.

Fixes: https://tracker.ceph.com/issues/65599
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56353 from myoungwon/wip-apply-shallow-copy-rbm-overwrite

crimson/os/seastore: avoid new allocation when overwriting data in RBM for performance

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #56855 from Svelar/asan_osd_types

test/osd/types: free PriorSet::pcontdec memory

Merge pull request #56025 from xxhdx1985126/wip-seastore-onode-loc-key

crimson/osd/osd_operations/client_requests: we don't support rados locator keys

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #56904 from zdover23/wip-doc-2024-04-16-security-cves

doc/security: update CVE list

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/security: update CVE list

Update the list of CVEs in doc/security/cves.rst.

Co-authored-by: Ilya Dryomov <idryomov@redhat.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56916 from Matan-B/wip-matanb-crimson-recovery-misc

crimson/osd/replicated_backend: misc fixes

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #56848 from xxhdx1985126/wip-65453

crimson/osd/recovery_backends: discard outdated recovery ops

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #56277 from amathuria/wip-amat-crimson-fix-pool-compression

crimson: Add support for pool compression

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #56611 from Matan-B/wip-matanb-crimson-replicated-peering

crimson/osd/replicated_recovery_backend: prepare_pull use pg_info

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #56875 from xxhdx1985126/wip-65474

crimson/osd/osd_meta: load incremental osdmap from "inc_osdmap.XXX"

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57012 from tchaikov/wip-python-common-pep8

python-common: PEP8 related cleanups

Reviewed-by: John Mulligan <jmulligan@redhat.com>

python-common: remove spaces around `=`

to be more PEP8 compliant

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

python-common: remove trailing spaces

for better readability.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

qa: update quiesce tests to expect ipolicy lock

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add missing policylock to test F_QUIESCE_BLOCK

In order to check an inode's F_QUIESCE_BLOCK, the quiesce_inode op must acquire
the policylock. Furthermore, to ensure the F_QUIESCE_BLOCK is not changed
during quiesce, the lock must be held for the duration of the op's lifetime.

Fixes: https://tracker.ceph.com/issues/65595
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>