fixup! fixup! crimson/os/seastore: throttle bandwidth to secondary devices

crimson/os/seastore/epm: coroutinize RandomBlockOolWriter::alloc_write_ool_extents()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/epm: coroutinize SegmentedOolWriter::alloc_write_ool_extents()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

fixup! crimson/os/seastore: throttle bandwidth to secondary devices

crimson/os/seastore/epm: reserve the space in the cold tier before
demoting

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

fixup! crimson/os/seastore: add test workload to promote/evict aggressively

crimson/os/seastore/epm: add try_reserve_main and abort_main_usage

As the counterpart of try_reserve_cold and abort_cold_usage

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

fixup! crimson/os/seastore: consolidate parameters on extent allocation path

fixup! crimson/os/seastore: add write through policy

fixup! crimson/os/seastore/EPM/BackgroundProcess: add promote and demote process

fixup! crimson/os/seastore/transaction_manager: wait for the demotion when the rbm main device is not enough

debug

fixup! crimson/os/seastore: implement promote extent

crimson/os/seastore/transaction_manager: add the shadow to the trans'
read_set when getting extents

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction: mutation_pending extents' paddrs should
be on the same device as the stable prior's old paddr

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/journal/segmented_journal: scan the journal for the
allocation map on boot if the cold tier is an RBM one

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/journal: refactor SegmentedJournal::replay() to
adapt to Journal::scan_valid_record_delta()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/journal: coroutinize SegmentedJournal::replay()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/journal: lift the alloc_map scanning from CircularBoundedJournal to Journal

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/btree: define fmt::formatter for lba_val_map_t in
btree_types.h

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/lba: make sure all mappings in the copied range is
examined when updating paddrs for the copied mappings

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/lba: add more rigorous assertions

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: indicate the extents as COLD
when demoting them

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/epm: avoid adjust generation for eviction if the
main device is not SEGMENTED

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cache: allow LogNode to be remapped

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: also update lba mappings' shadow fields when
updating paddr synchronously

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/lba: merge the whole mapping into the delta

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/extent_pinboard: don't add extents under promotion
to cache

This is to avoid an under-promotion extent from being added back to and
evicted from the cache again before the promotion ends.

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/lba,TM: remove shadows when trimming dirty non-data
extents

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: invalidate the transaction if it has written
lba/backref nodes to the disk when the rewrite transaction tries to
merge those nodes' content

Otherwise, the result of the merging won't be persisted to the disk

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: allow concurrent rewrite transactions

Since promote/demote background processes can run concurrently with
trimming and cleaning, there maybe more than one sources that block
client transactions from submitting. We need to allow such cases.

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cached_extent: let pending cached extents hold pointers to the transactions

This is necessary for the promote transaction to find the transactions
that are retiring the extents promoted

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: demote/promote background processes are also
rewrite transactions

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: wait for the demotion when the
rbm main device is not enough

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: drop backref when all backends are RBMs

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

qa/config/crimson_qa_overrides.yaml: add test workload for LBC

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cache: count promote/demote related metrics

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

test/crimson/seastore/test_transaction_manager: the scatter_allocation case shouldn't submit the trans

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: add test workload to promote/evict aggressively

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/random_block_manager: try to allocate consecutive rbm space when rewriting extents

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cache: add hit ratio metric

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: make cold rbm cleaner use different metric prefix

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: allow RBM backend as hot tier

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: update logical bucket cache

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: add write through policy

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd: create secondary devices when mkfs

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: throttle bandwidth to secondary devices

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: introduce RotationalDevice

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: set backend type explicitly

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/EPM/BackgroundProcess: add promote and demote process

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/EPM: initialize logical bucket and pinboard

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: don't add shadow extent to cache

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: avoid evicting promoted extent to the cold tier

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: consolidate parameters on extent allocation path

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: implement demote region

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: implement promote extent

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/lba: introduce shadow paddr

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: introduce logical bucket and demote process

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: support promote extents purged from pinboard

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: rename extent_2q_state_t to extent_pin_state_t

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: introduce PROMOTE and DEMOTE transaction type

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

doc/dev/crimson: add design the document for LogicalBucketCache

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #69905 from ronen-fr/wip-rf-328786-emptyflush

crimson/osd: skip store flush for empty peering transactions

Reviewed-by: Kefu Chai <k.chai@proxmox.com>
Reviewed-by: Jose J Palacios-Perez <perezjos@uk.ibm.com>

Merge pull request #69638 from AliMasarweh/wip-nbalacha-posix-versioned

rgw/posix: fix delete markers

reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #69349 from NitzanMordhai/wip-nitzan-mgr-dispatch-throttle-increase

mgr: fix dispatch throttle bottleneck causing OSD connection timeouts

Merge pull request #69353 from batrick/librados-nspace-dout

librados: print nspace only if present

Reviewed-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #69819 from sunyuechi/wip-blk-spdk-env-opts-init

blk/spdk: call spdk_env_opts_init() before setting pci options

Reviewed-by: Kefu Chai <k.chai@proxmox.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #69053 from tchaikov/wip-seastore-overwrite-update-checksum

crimson/seastore: skip LBA-leaf crc check for in-place delta-overwrite

Reviewed-by: Myoungwon Oh <ohmyoungwon@gmail.com>

Merge pull request #69793 from anthonyeleven/caps

doc: Improve src/common/options/mds-client.yaml.in

Merge pull request #67779 from anthonyeleven/egregrioustypo

doc/rados/operations: Fix typos in erasure-code.rst

doc/rados/operations: Fix typos in erasure-code.rst

Signed-off-by: adatri <anthony.datri@gmail.com>

Merge pull request #69838 from ronen-fr/wip-rf-322936

crimson/os/alienstore: make CnLog::_flush non-blocking

Reviewed-by: Kefu Chai <k.chai@proxmox.com>
Reviewed-by: Jose J Palacios-Perez <perezjos@uk.ibm.com>

Merge pull request #69815 from rhcs-dashboard/subsystem-resourcepage

mgr/dashboard: NVMe-oF – Subsystem Resource page enhancements

Reviewed-by: Sagar Gopale <sagar.gopale@ibm.com>

Merge pull request #69937 from tchaikov/wip-promql-test-deterministic

monitoring/ceph-mixin: scope test queries per dashboard

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #69656 from tchaikov/wip-crimson-common-dump-metric-value-map

crimson/common: use the value_map passed to dump_metric_value_map()

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #69697 from tchaikov/wip-crimson-throttle-releaser

crimson/osd: mark ThrottleReleaser [[nodiscard]]

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

monitoring/ceph-mixin: scope test queries per dashboard

get_dashboards_data() keeps every query in a single map keyed by
"<panel title>-<legendFormat>", but that id is not unique: several
dashboards have a panel titled "IOPS", "OSDs" or "Throughput". The
dashboard read last overwrites the earlier entry, so one query shadows
another and never gets tested. glob() walks the files in filesystem
order, so which query survives, and whether the test passes, depends on
readdir.

run-tox-promql-query-test hit this in "Test IOPS Read"
(ceph-cluster.feature): the scenario feeds ceph_osd_op_r but the id
resolved to a CephFS pool panel.

    FAILED:
      expr: "sum(rate(ceph_pool_rd{cluster=~"mycluster|",  pool_id=~"UNSET VARIABLE"}[1m]))", time: 1m,
          exp:"{} 2.5E+01"
          got:"nil"
    HOOK-ERROR in after_scenario: AssertionError:

pool_id is "UNSET VARIABLE" because the scenario does not set $mdatapool,
and no ceph_pool_rd series were fed, so the query returns nil. It only
fails when readdir returns cephfsdashboard after ceph-cluster-advanced,
so the earlier change that walked nested rows and made the winner
deterministic did not fix it; it fixed which query wins, not that the
wrong one can win.

Key the queries per dashboard (data['queries'][<dashboard>][<id>]) and
have each .feature name its dashboard in a Background step. The lookup is
confined to that dashboard, so a title/legend used elsewhere no longer
shadows it. A duplicate within one dashboard is still an error, unless it
is in a collapsed row.

ceph-cluster.feature covers ceph-cluster-advanced; the rest map to one
dashboard each.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

mgr/dashboard: NVMe-oF – Subsystem Resource page enhancements

fixes:https://tracker.ceph.com/issues/77789

Signed-off-by: pujaoshahu <pshahu@redhat.com>
# Conflicts:
# src/pybind/mgr/dashboard/frontend/src/app/ceph/block/nvmeof-group-form/nvmeof-group-form.component.html
# src/pybind/mgr/dashboard/frontend/src/app/ceph/block/nvmeof-subsystem-overview/nvmeof-subsystem-overview.component.html
# src/pybind/mgr/dashboard/frontend/src/app/ceph/block/nvmeof-subsystem-overview/nvmeof-subsystem-overview.component.spec.ts
# src/pybind/mgr/dashboard/frontend/src/app/ceph/block/nvmeof-subsystem-overview/nvmeof-subsystem-overview.component.ts

Merge pull request #69850 from rhcs-dashboard/edit-auth-subsystem

mgr/dashboard: NVMe-oF – Add edit authentication support for subsystem

Reviewed-by: Sagar Gopale <sagar.gopale@ibm.com>

doc: Improve src/common/options/mds-client.yaml.in

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

mgr/dashboard: NVMe-oF – Add edit authentication support for subsystem

Fixes: https://tracker.ceph.com/issues/77840
Signed-off-by: pujaoshahu <pshahu@redhat.com>

mgr/dashboard: NVMe-oF – Subsystem Resource page enhancements

fixes:https://tracker.ceph.com/issues/77789

Signed-off-by: pujaoshahu <pshahu@redhat.com>

Merge pull request #69441 from rhcs-dashboard/edit-gateway

mgr/dashboard: Implement support for editing existing nvme-oF gateway groups through UI

Reviewed-by: Sagar Gopale <sagar.gopale@ibm.com>

Merge pull request #69883 from imran-imtiaz/wip-dashboard-clone-snap-id

mgr/dashboard: support snapshot_id in clone API for group snapshots

Merge pull request #69243 from rhcs-dashboard/bug-3990-pr2

mgr/dashboard : Add gateway filtering support for subsystem and namespace lists

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Sagar Gopale <sagar.gopale@ibm.com>

Merge pull request #69912 from Sodani/shsodani_rgw_caps_final

mgr/smb: Allow pool creation for empty RGW buckets

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

mgr/dashboard: Implement support for editing existing nvme-oF gateway groups through UI

Fixes: https://tracker.ceph.com/issues/77386
Signed-off-by: pujaoshahu <pshahu@redhat.com>

mgr/dashboard: Implement backword support for Create existing nvme-oF gateway groups through UI

Fixes : https://tracker.ceph.com/issues/77632

Signed-off-by: pujaoshahu <pshahu@redhat.com>
(cherry picked from commit 26b7e9e4849f8f0f1d044cb23cc94a5ce910061d)

mgr/dashboard: Implement NVMe-oF Gateway Group Resource Overview Page

Fixes: https://tracker.ceph.com/issues/77383
Signed-off-by: pujaoshahu <pshahu@redhat.com>
(cherry picked from commit 254ab5383505962460d06b483dd931bf91c1f0b8)

mgr/dashboard : Enable cephadm-signed TLS for NVMeOf
fixes: https://tracker.ceph.com/issues/77406
Signed-off-by: Abhishek Desai <abhishek.desai1@ibm.com>
(cherry picked from commit 9623f6847cecf7f2e79f461420fce4821fd1b159)

mgr/dashboard : Add gateway filtering support for subsystem and namespace lists

Fixes: https://tracker.ceph.com/issues/77068
Signed-off-by: pujaoshahu <pshahu@redhat.com>
(cherry picked from commit e838bb19897ed4a8f6a548eca08e4fdd726e62f5)

 Conflicts:
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/nvmeof-namespaces-list/nvmeof-namespaces-list.component.spec.ts
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/nvmeof-subsystems/nvmeof-subsystems.component.spec.ts

Merge pull request #69856 from tchaikov/wip-seastar-sanitizer

crimson: keep seastar's sanitizers in lockstep with WITH_ASAN

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #69439 from rhcs-dashboard/fix-4877

mgr/dashboard: Implement NVMe-oF Gateway Group Resource Overview Page

Reviewed-by: Sagar Gopale <sagar.gopale@ibm.com>

Merge pull request #69753 from rhcs-dashboard/atollon-fixes

mgr/dashboard: Add compression panels and PSU temperature

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #69757 from rhcs-dashboard/atollon-alerts

mgr/dashboard: Add node proxy hardware alerts

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #69864 from tchaikov/wip-rpm-build-deps

ceph.spec.in: drop rhel 8 support and use pkgconfig() when possible

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #69795 from tchaikov/wip-qa-cephfs-escape-warnings

qa/cephfs: fix invalid escape sequence SyntaxWarnings

Reviewed-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>

Merge pull request #69908 from sunyuechi/crc32c-riscv-zbc-no-gp-tp

common/crc32c: stop using gp/tp as scratch in RISC-V Zbc CRC32C

Reviewed-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #69652 from yzaken/bugfix_ceph_cli_watch_flag

ceph.in: reject -w/--watch flags when used with subcommands

Reviewed-by: Kobi Ginon <kginon@redhat.com>
Reviewed-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #69609 from mheler/wip-77538

mgr: filter root logger fallback