crimson/os/seastore/lba: merge the whole mapping into the delta

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/extent_pinboard: don't add extents under promotion
to cache

This is to avoid an under-promotion extent from being added back to and
evicted from the cache again before the promotion ends.

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/lba,TM: remove shadows when trimming dirty non-data
extents

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cached_extent: let pending cached extents hold pointers to the transactions

This is necessary for the promote transaction to find the transactions
that are retiring the extents promoted

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: demote/promote background processes are also
rewrite transactions

So we need to merge the extents promoted/demoted with their counterparts
in client transactions, just like trim/cleaner transaction do.

The difference between demote/promote transactions and trim/cleaner
transactions is that the former also involves remapping extents while the
latter only involves rewriting and mutating extents.

With the logical bucket cache in place, all rewrite transactions need to
handle shadow extents carafully, too.

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: wait for the demotion when the
rbm main device is not enough

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: drop backref when all backends are RBMs

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

qa/config/crimson_qa_overrides.yaml: add test workload for LBC

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cache: count promote/demote related metrics

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

test/crimson/seastore/test_transaction_manager: the scatter_allocation case shouldn't submit the trans

In scatter_allocation, an extent is split into several
small ones, and the last allocation is supposed to fail,
in which case, the small extents that belong to the same
allocation wouldn't be attached to the parent lba leaf
node. Sumitting the transaction would trigger assertions,
and is not consistent with the behavior of nornal crimson
OSDs.

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: add test workload to promote/evict aggressively

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/random_block_manager: try to allocate consecutive rbm space when rewriting extents

Add the address of the last allocation as the hint to the current round
of allocation in rbm

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cache: add hit ratio metric

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: make cold rbm cleaner use different metric prefix

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: allow RBM backend as hot tier

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: update logical bucket cache when processing
read/write transactions

All client accessed extents are to be tracked by logical bucket cache
before eviction

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: add write through policy

Write extents larger than some threshold to the cold tier directly

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd: create cold devices when mkfs

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: throttle bandwidth to secondary devices

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: introduce RotationalDevice

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: set backend type explicitly

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/EPM/BackgroundProcess: add promote and demote process

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/EPM: initialize logical bucket and pinboard

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: don't add shadow extent to cache

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: avoid evicting promoted extent to the cold tier

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: consolidate parameters on extent allocation path

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: implement demote region

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: implement promote extent

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/lba: introduce shadow paddr

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: introduce logical bucket and demote process

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: support promote extents purged from pinboard

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: rename extent_2q_state_t to extent_pin_state_t

This is because, even for LRU caches, extents still have pin states,
although it's only Fresh. What's more, in the logical bucket cache
world, extents in both LRU and 2Q caches may be in the
Fresh/PendingPromote/Promoting state.

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: introduce PROMOTE and DEMOTE transaction type

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

doc/dev/crimson: add design the document for LogicalBucketCache

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #70051 from mheler/wip-copyobject-replace

Merge pull request #70659 from bluikko/wip-doc-cephadm-install-substitutions

doc/cephadm: enable substitutions on CLI command example

Merge pull request #69533 from qiuxinyidian/main

rgw:adding bucket index Indexless check when bi list

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #70097 from tchaikov/wip-mgr-testcase-single-bounce

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #70515 from rhcs-dashboard/fix-17067

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #70604 from rhcs-dashboard/service-form-text-label-fix

mgr/dashboard: fix cd-text-label-list string writeValue

Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/cephadm: enable substitutions on CLI command example

Substitutions were not enabled on a prompt directive that tried to use
the stable-release substitution so the substitution string was rendered
verbatim.

Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>

Merge pull request #70625 from rhcs-dashboard/fix/dashboard-gzip-mime-type

mgr/dashboard: add text/javascript to gzip mime types

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #70606 from rhcs-dashboard/add-doc-url

mgr/dashboard: Add alerts doc url to active alert list

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #68519 from edwinzrodriguez/cephfs-bench

cephfs-tool: Improve reporting and config options

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #70486 from guits/fix-nvme-reformat

ceph-volume: fix nvme preformat wiping VG on osds-per-device > 1

Merge pull request #70603 from rhcs-dashboard/snap-edit-fix

mgr/dashboard: fix snap schedule startDate

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix cd-text-label-list string writeValue

When editing services such as oauth2-proxy, backend fields like scope
arrive as a single string. Spreading that string into the list turned
each character into its own row. Normalize string values to one item.

Fixes: https://tracker.ceph.com/issues/78808
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@ibm.com>

Merge pull request #69680 from sunyuechi/wip-bufferlist-omap

test/crimson/omap: enlarge values under ASan to shrink depth-driven tests

Reviewed-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #66757 from senolcolak/wip-v2-mgr-inventory-crush-host

mgr: override OSD hostname with CRUSH physical host in all metadata paths

Merge pull request #69885 from sunyuechi/wip-rados-fix-std-fd-close

tools/rados: fix stdio/stream cleanup

Merge pull request #67023 from trociny/wip-74490

mon: allow getpoolstats for mgr modules

Merge pull request #69973 from ljflores/wip-tracker-77382

qa/suites/upgrade: ignore expected telemetry warning

Merge pull request #70206 from sunyuechi/wip-dockerfile-sccache-dnf

Dockerfile.build: install sccache from distro packages when available

Reviewed-by: Zack Cerza <zack@redhat.com>

Merge pull request #69325 from NitzanMordhai/wip-nitzan-mgr-memory-grow-debug-high-level

mgr/ClusterState: remove debug 30 memory spike

Merge pull request #69099 from MaxKellermann/osd_SnapMapper__fmt_no_sprintf

osd/SnapMapper: use fmt::format() instead of fmt::sprintf()

Merge pull request #69098 from MaxKellermann/common_options__long_desc__c_string

common/options: convert `long_desc` to C string

Merge pull request #70231 from bill-scales/issue58417

osd: Fix race hazard causing duplicated message after OSD marked down

Merge pull request #70233 from bill-scales/issue54932

osd: Fix PeeringState::proc_lease crash caused by race hazard

Merge pull request #70333 from bill-scales/issue76953

osd: Fix valgrind uninitialized variable issues in fast EC

Reviewed-by: Matty Williams <Matty.Williams@ibm.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #70505 from ivancich/wip-fix-ceph-spec-in

rgw: update rpm and debian builds

Reviewed-by: Thomas Serlin <tserlin@redhat.com>

mgr/dashboard: add text/javascript to gzip mime types

CherryPy's serve_file() uses Python's mimetypes module which returns
text/javascript for .js files (RFC 9239), but the gzip config only
listed application/javascript. This meant the main Angular bundle
was being served uncompressed. With Cheroot write timeouts,
the uncompressed transfer times out over slower networks, causing
ERR_CONTENT_LENGTH_MISMATCH in the browser.

Signed-off-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #70544 from rhcs-dashboard/switcher

mgr/dashboard: rgw gateway switcher read-only for resource page

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #70547 from xxhdx1985126/wip-crimson-recover-delete-fix

crimson/osd/pg_recovery: call on_global_recover on replica-recover-deleted objects only when it's not missing on the primary too

Reviewed-by: Kefu Chai <k.chai@proxmox.com>

Merge PR #70581 into main

* refs/pull/70581/head:
doc/governance: update CEC term length

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge PR #70574 into main

* refs/pull/70574/head:
.github/workflows: bump checkout action version
.github/workflows/qa-symlink: sparse checkout verify-qa
.github/workflows/qa-symlink: run only for active branches

Reviewed-by: Joseph Mundackal <jmundackal@bloomberg.net>

Merge PR #70582 into main

* refs/pull/70582/head:
doc/governance: add Redouane to CSC

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge PR #70586 into main

* refs/pull/70586/head:
.githubmap: add BBoozmen

Reviewed-by: Oguzhan Ozmen <oozmen@bloomberg.net>

Merge PR #70598 into main

* refs/pull/70598/head:
.githubmap: add mheler

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #70614 from VallariAg/wip-nvmeof-upgrade-refresh

qa/suites/nvmeof/upgrade: use refresh True for wait_for_service

Merge pull request #69662 from aainscow/truncate_and_write_tests

test/osd: Add truncate-then-write tests

Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>

Merge pull request #69758 from tchaikov/wip-bumpup-seastar

crimson: bump seastar submodule and adapt to its new future API

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

mgr/dashboard: Add alerts doc url to active alert list

Add documentation link in Learn more column for every alert in the
active alerts list table

Fixes: https://tracker-origin.ceph.com/issues/78747
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

qa/suites/nvmeof/upgrade: use refresh True for wait_for_service

Add refresh: True to upgrade tests, same as we added for
https://tracker.ceph.com/issues/73538

Fixes: https://tracker.ceph.com/issues/78770
Co-authored-by: barakda <barak.davidov@gmail.com>
Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

crimson: migrate off deprecated seastar::smp::count and all_cpus()

The updated seastar deprecates the global smp::count and smp::all_cpus() in
favour of the per-instance smp::shard_count() and smp::all_shards(). Under the
CI -Werror build these deprecation warnings are fatal.

Replace the uses with the reactor-local free functions
seastar::this_smp_shard_count() and seastar::this_smp_all_shards(), which read
the current smp instance's shard count and shard-id range. crimson runs a
single smp instance and every migrated site executes on a reactor thread (the
alienstore worker threads do not touch these), so the behaviour is unchanged.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

crimson: use the seastar span sink API and build at API level 10

At SEASTAR_API_LEVEL >= 9 seastar dropped output_stream::write(net::packet) in
favour of write(std::span<temporary_buffer<char>>). Convert Socket's writes to
the span form, writing the packet's released fragments and keeping them alive
across the write with do_with, then raise Seastar_API_LEVEL from 6 to 10 to
match the updated seastar. The span write overloads are unconditional, so the
Socket change is valid at every API level.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

crimson: bump seastar submodule and adapt to its new future API

Update src/seastar to ceph-umbrella-25.05.0-1074-gcced0236, which is
scylladb/seastar master (fd71a5b4c7) with the ceph patches reapplied,
and adapt crimson to the future API changes the bump brings in:

  - interruptible_future.h: seastar split its single ready_future_marker into
    set_ready_future_marker and set_from_tuple_ready_future_marker; route
    futurize::from_tuple through the matching markers.
  - thread_pool.h: future_state::set() is now templated and get_value() is
    ref-qualified; pass seastar::internal::monostate{} for the void case and
    move the state before reading its value.

These crimson changes need the new seastar, so they share one commit with the
submodule bump. The build stays at Seastar_API_LEVEL 6 here; the level is
raised in the following commit.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #70180 from Matan-B/wip-matanb-seastore-coll-batch

crimson/seastore: batch same-collection transactions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #70467 from bluikko/wip-doc-nginx-capitalization

doc: standardize capitalization of NGINX following the vendor

Merge pull request #70209 from Hezko/warn-on-deprecated-param

mgr/dashboard: add deprecation warning on ns add cli command

Merge pull request #70560 from rhcs-dashboard/fix-gateway-readonly

Reviewed-by: Naman Munet <nmunet@redhat.com>

Merge pull request #70566 from rhcs-dashboard/service-form-helper-fix

Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix snap schedule startDate

Fixes date time picker that was wrongly setting the form date

Fixes: https://tracker-origin.ceph.com/issues/78742
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@ibm.com>

Merge PR #70375 into main

* refs/pull/70375/head:
src/pybind: fixing the test_invalid_client_id failure

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #69726 from tchaikov/wip-drop-bluestore-pmem

blk,build: drop BlueStore PMEM support

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

mgr/dashboard: add deprecation warning on ns add nvme cli command

Fixes: https://tracker.ceph.com/issues/78228
Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

Merge pull request #68209 from sunyuechi/riscv-bench-crc

test/common: add RISC-V CRC32C performance benchmark to unittest_crc32c

Reviewed-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #70578 from gbregman/main

mgr/cephadm: Add CNC fields to NVMEoF spec file

.githubmap: add mheler

Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>

mgr/dashboard: rgw gateway switcher read-only for resource page

- Fixes: https://tracker.ceph.com/issues/78513

Signed-off-by: Syed Ali Ul Hasan <syedaliulhasan19@gmail.com>

Merge pull request #69790 from syedali237/dashboard/user-management

mgr/dashboard : migrated User Table tabs to resource pages

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Abhishek Desai <abhishek.desai1@ibm.com>

test/osd: cover truncate+write EC transaction behaviour

Add unit-tests to recreate the problems found by the IO Sequencer in the previous commit.

Some tests here rely on frameworks added in Umbrella and as such we are keeping them
independent (in their own PRs).

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Signed-off-by: Matty Williams <Matty.Williams@ibm.com>
Assisted-by: IBM Bob:Claude/GPT

mgr/cephadm: Add CNC fields to NVMEoF spec file
Fixes: https://tracker-origin.ceph.com/issues/78675
Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

.githubmap: add BBoozmen

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

rgw: update build config to streamline builds

Some binaries and man pages were not included in rpm builds or debian
builds.  This adds those artifacts to ceph.spec.in
debian/ceph-common.install and debian/radosgw.install.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #70576 into main

* refs/pull/70576/head:
script/ptl-tool: add more refs for main branch

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

mgr/dashboard: migrated user table tabs to resource pages

- Fixes: https://tracker.ceph.com/issues/77475

Signed-off-by: Syed Ali Ul Hasan <syedaliulhasan19@gmail.com>

doc/governance: update CEC term length

Fixes: https://tracker.ceph.com/issues/77966
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

doc/governance: add Redouane to CSC

Fixes: https://tracker.ceph.com/issues/77965
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

script/ptl-tool: add more refs for main branch

Start with "upstream", then "origin", then finally the local main branch.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #69844 from kginonredhat/issue-77835-fail-fs-mds-upgrade-wait

mgr/cephadm: wait for active MDS during fail_fs upgrade

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>