qa/workunits/nvmeof/basic_tests.sh: Assert number of devices

Check number of devices connected after connect-all.
It should be equal to number of namespaces created.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

qa/workunits/nvmeof/setup_subsystem.sh: add list_namespaces() func

Add list_namespaces function which could be useful for debugging later.
Remove extra call of list_subsystems so it's only logged once after
subsystems are completely setup.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

qa/tasks/nvmeof.py: Improve thrasher and rbd image creation

Create rbd images in one command using ";" to queue them,
instead of running "cephadm shell -- rbd create" again
and again for each image.

Improve the method to select to-be-thrashed daemons.
Use randint() and sample(), instead of weights/skip.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

qa/suites/nvmeof/thrasher: use 120 subsystems and 8 ns each

For tharsher test:
1. Run it on 120 subsystems with 8 namespaces each
2. Run FIO for 20 mins (instead of 15mins)
2. Run FIO for few randomly picked devices
    (using `--random_devices 200`)

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

qa/suites/nvmeof/thrash: Add "is unavailable" to log-ignorelist

This commit also:
- Remove --rbd_iostat from thrasher fio
- Log iteration details before printing stats in nvmeof_tharsher

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

qa/suites/nvmeof/basic: use default image in nvmeof_initiator.yaml

Instead of using quay.io/ceph/nvmeof:latest, use default
image in ceph build.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

qa/suite/nvmeof/thrash: increase number of thrashing

- Run fio for 15 mins (instead of 10min).
- nvmeof.py: change daemon_max_thrash_times default from 3 to 5
- nvmeof.py: run nvme list in do_checks()

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

qa/suite/nvmeof: add asserts to scalability_test.sh

Add assertions to 'status_checks()' function.
Use "apply" and "redeploy", instead of "orch rm" and
"apply" to upscale/downscale gateways.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

qa/suites/nvmeof: fix nvmeof_namespaces.yaml

When basic_tests.sh is executed in parallel
with namespace_test.sh, sometimes namespace_test.sh
starts before fio_test.sh which would break the test.

So this change ensures "fio_test.sh" is started before
and executed in parallel with "namespace_test.sh".

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

Merge pull request #60736 from guits/fix-stale-bot

.github/workflows: Do not exempt PRs with milestone

Merge pull request #59897 from avanthakkar/note-cephfs-earmark

doc: add pendingreleasenotes for cephfs subvolume earmarking feature

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #60768 from VallariAg/wip-prometheus-tested-limits

monitoring: Add alert NVMeoFTooManyNamespaces

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60702 from ceph/wip-yuriw-release-17.2.8-main

doc: 17.2.8 Release Notes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/releases: fix some formatting for the 17.2.8 release notes

Signed-off-by: Laura Flores <lflores@ibm.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #60756 from ArbitCode/wip-raja-rgw-fix-68211

rgw:Wrong x-rgw-quota-bucket-objects header when bucket quota is disabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60813 from anthonyeleven/improve-subsystem

doc/rados/troubleshooting: Improve log-and-debug.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60818 from gbregman/main

Add key encryption support to nvmeof configuration

python-common/ceph/deployment: Add key encryption support to nvmeof configuration
Fixes https://tracker.ceph.com/issues/69036

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

mgr/cephadm: Add key encryption support to nvmeof configuration
Fixes https://tracker.ceph.com/issues/69036

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

doc/rados/troubleshooting: Improve log-and-debug.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #60805 from zdover23/wip-doc-2024-11-23-cephadm-install-curl-based-installation

doc/cephadm: Improve "Curl-based Installation"

:Reviewd-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #60804 from zdover23/wip-doc-2024-11-23-cephadm-install

doc/cephadm: Clarify "Deploying a new Cluster"

Reviewed-by: Anthony D'Atri<anthony.datri@gmail.com>

doc/cephadm: Improve "Curl-based Installation"

Change the name of the section "curl-based installation" to "Using Curl to Install Cephadm", and improve the English in that section.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/cephadm: Clarify "Deploying a new Cluster"

Change the title of the section "Deploying a new Ceph cluster" to "Using
cephadm to Deploy a New Ceph Cluster". This is part of the initiative to
separate package-related documentation from container-based
documenation.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60775 from cbodley/wip-rgw-access-key-active-json

rgw: RGWAccessKey::decode_json() preserves default value of 'active'

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #58645 from vshankar/wip-omap-commit-header-size

mds: account for header size during omap commit

Merge pull request #60488 from jmolmo/exporter_manage_sigterm

exporter: SIGTERM/SIGINT/SIGHUP management

rgw:Wrong x-rgw-quota-bucket-objects header when bucket quota is disabled

This commit introduces to get quota settings if bucket quota is enabled

Fixes: https://tracker.ceph.com/issues/68211
Signed-off-by: Raja Sharma <raja@ibm.com>

Merge pull request #60787 from ronen-fr/wip-rf-reservStat

osd/scrub: fixing reservations dump format

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #60636 from mohit84/issue_68585

TEST_backfill_grow fails after finding "num_bytes mismatch" in osd log

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #60253 from MaxKellermann/includes

common: add missing include, reduce header dependencies

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #59524 from liangmingyuanneo/wip-standalone-test-pg-repair

qa/standalone: bugfix for wait_for_scrub

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #60523 from ljflores/wip-tracker-68657

mgr/balancer: optimize 'balancer status detail'

Merge pull request #60542 from adk3798/sign-generated-rgw-certs

mgr/cephadm: sign generated RGW certs

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #60303 from adk3798/handle-failed-daemon-start

cephadm: handle "systemctl start" failures during deployment better

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60531 from Kushal-deb/fix_issue_2124175

mgr/cephadm: Added fix for Modifying Profiles(add-setting and rm-setting) to allow multiple setting changes

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60587 from phlogistonjohn/jjm-more-py312-fixes

various python 3.12 fixes

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60319 from baum/no_huge

cephadm: support no huge pages for nvmeof spdk

osd/scrub: fixing reservations dump format

Headers were missing in the faulty dump output. This commit
fixes the section type to 'object section'.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

cephadm/nvmeof: support no huge pages for nvmeof spdk

depends on: https://github.com/ceph/ceph-nvmeof/pull/898

Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

Merge pull request #60668 from ronen-fr/wip-rf-mconf

osd/scrub: cache frequently used configuration parameters

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #60769 from gbregman/main

mgr/cephadm: Add resource limits to nvmeof configuration

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60588 from AlexonOliveiraRH/wip-doc-templates

doc/cephadm/services: Update monitoring.rst to add additional templates

rgw: RGWAccessKey::decode_json() preserves default value of 'active'

squid added the new "bool active" member that defaults to true, but
RGWAccessKey::decode_json() sets it to false when "active" isn't present

this is an issue for multisite when a squid zone replicates user
metadata from a master zone running reef

Fixes: https://tracker.ceph.com/issues/68985
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60598 from xxhdx1985126/wip-68808

crimson/osd/replicate_backend: add the skipped newly created clone object to the push queue after the clone request completes

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

common: rm the cast operator of md_config_cacher_t

as its replacement - the type-deduced operator() - is now in use,
and the resulting code is more readable and less error-prone.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd: rm all uses of the cast operator of md_config_cacher_t

as this interface is to be removed in the next commit.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: cache frequently used configuration parameters

using the md_config_cacher_t, which is a cache object that registers
itself to the config observer and caches the up-to-date configuration
value.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

common: a simple API to extract md_config_cacher_t cached value

Using a type-deduced operator() (i.e. my_opt() ), instead of the
existing ValueT() operator (which requires a type-matching cast).

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

mgr/cephadm: Add resource limits to nvmeof configuration
Fixes https://tracker.ceph.com/issues/68967

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

python-common/ceph/deployment: Add resource limits to nvmeof configuration
Fixes https://tracker.ceph.com/issues/68967

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #60572 from xxhdx1985126/wip-68787

crimson/osd/backfill_state: don't try to do object backfills when primary_bi is empty

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #60661 from ronen-fr/wip-rf-scheduledeep

osd/scrub: fix 'schedule-deepscrub' test asok command

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

monitoring: Add alert NVMeoFTooManyNamespaces

NVMeoFTooManyNamespaces helps to alert user if total
number of namespaces across subsystems are more than
1024.

Change NVMeoFTooManySubsystems limit to 128 from 16.

Fixes: https://github.com/ceph/ceph-nvmeof/issues/948
Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

Merge pull request #60674 from oshrey16/bugfix-rgw-ps-list-topics-op

rgw/notification: Fix segmentation fault in RGWPSListTopicsOp::execute() and correct topic listing to use get_topics_v2

Reviewed-By: Joseph Mundackal <jmundackal@bloomberg.net>, Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #60765 from zdover23/wip-doc-2024-11-19-os-recommendations

doc/start: update os-recommendations.rst

doc/start: update os-recommendations.rst

Remove information about the operating systems that support Ceph's
official container images from the "Platforms" table in
doc/start/os-recommendations.rst and add that information to the (new)
table that shows the operating systems that support Ceph's official
container images.

Credit for this change should go to Enrico Bocchi, who noticed a
discrepancy that motivated it.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60309 from linuxbox2/wip-cksum-68537

rgw_cksum:  treat mpu checksums w/o part count as valid in CompleteMu…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60711 from soumyakoduri/wip-skoduri-dbstore

rgw: Update README for DBStore and Posix drivers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mds: account for header size during omap commit

fnode_t is set in the omap header during directory commit
operation which isn't accounted when tracking operation
size.

Fixes: http://tracker.ceph.com/issues/67597
Signed-off-by: Venky Shankar <vshankar@redhat.com>

rgw/notification: fix segmentation fault and topic listing logic

- Fixed a segmentation fault caused by a null bucket pointer in RGWPSListTopicsOp::execute()
- Corrected logic to use get_topics_v2 when supported, with fallback otherwise

Fixes: https://tracker.ceph.com/issues/68756
Signed-off-by: Oshrey Avraham <oshrey16@gmail.com>

doc: add pendingreleasenotes for cephfs subvolume earmarking feature

Signed-off-by: Avan Thakkar <athakkar@redhat.com>

.github/workflows: Do not exempt PRs with milestone

The stale bot doesn't close certain PRs, even if they have been inactive
for several years, because almost all of these PRs have a milestone

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

crimson/osd/replicate_backend: add the skipped newly created clone
object to the push queue after the clone request completes

Fixes: https://tracker.ceph.com/issues/68808
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

exporter: SIGTERM/SIGINT/SIGHUP management

Ceph exporter manages SIGINT, SIGTERM and SIGHUP signals

Fixes: https://tracker.ceph.com/issues/68721
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #60655 from xxhdx1985126/wip-seastore-move-out-root-meta

crimson/os/seastore: move the root meta out of the root block

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #60209 from myoungwon/wip-init-onode-layout

crimson/os/seastore: initialize oi and ss in onode_layout_t to zero

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc: 17.2.8 Release Notes

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #60715 from ronen-fr/wip-rf-dumpsched

osd/scrub: list additional information when dumping the queue

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #60727 from guits/dmcrypt-activate

ceph-volume: fix dmcrypt activation regression

Merge pull request #60404 from VallariAg/wip-nvmeof-listeners-prometheus-alerts

monitoring: add 2 nvmeof alerts to prometheus_alerts.yaml

Merge pull request #51766 from zhsgao/mds_clear_session_failed

mds: session in the importing state cannot be cleared if an export subtree task is interrupted while the state of importer is acking

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #60067 from VallariAg/wip-nvmeof-healthcheck

mon: add nvmeof healthchecks

ceph-volume: fix dmcrypt activation regression

`ceph.with_tpm` tag is actually a `str`, which means that
`bool('0')` will return `True`.

Fixes: https://tracker.ceph.com/issues/68944
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #60554 from jshen28/fix-legacy-addr-mismatch

Make mon_info_t consistent with mon_addr

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mds: client is evicted when an export subtree task is interrupted

The importer will force open some sessions provided by the exporter but the client does not know about
the new sessions until the exporter notifies it, and the notifications cannot be sent if the exporter
is interrupted. The client does not renew the sessions regularly that it does not know about, so the client
will be evicted by the importer after `session_autoclose` seconds (300 seconds by default).

The sessions that are forced opened in the importer need to be closed when the import process is reversed.

Signed-off-by: Zhansong Gao <zhsgao@hotmail.com>

qa: add test for importer's unexpected client eviction after an export subtree task is interrupted

Signed-off-by: Zhansong Gao <zhsgao@hotmail.com>

mds: session in the importing state cannot be cleared if an export subtree task is interrupted while the state of importer is acking

The related sessions in the importer are in the importing state(`Session::is_importing` return true) when the state of importer is `acking`,
`Migrator::import_reverse` called by `MDCache::handle_resolve` should reverse the process to clear the importing state if the exporter restarts
at this time, but it doesn't do that actually because of its bug. And it will cause these sessions to not be cleared when the client is
unmounted(evicted or timeout) until the mds is restarted.

The bug in `import_reverse` is that it contains the code to handle state `IMPORT_ACKING` but it will never be executed because
the state is modified to `IMPORT_ABORTING` at the beginning. Move `stat.state = IMPORT_ABORTING` to the end of import_reverse
so that it can handle the state `IMPORT_ACKING`.

Fixes: https://tracker.ceph.com/issues/61459
Signed-off-by: Zhansong Gao <zhsgao@hotmail.com>

qa: add test for importer's session cleanup after an export subtree task is interrupted

Signed-off-by: Zhansong Gao <zhsgao@hotmail.com>

qa/suites/nvmeof: add nvmeof warnings to log-ignorelist

Add NVMEOF_SINGLE_GATEWAY and NVMEOF_GATEWAY_DOWN
warnings to nvmeof:thrash job's log-ignorelist

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

mds: the assert should be before the journal entry submit otherwise it's racy

Signed-off-by: Zhansong Gao <zhsgao@hotmail.com>

mds: add `importing_count` to session dump

Signed-off-by: Zhansong Gao <zhsgao@hotmail.com>

Merge PR #60464 into main

* refs/pull/60464/head:
mds: add or update MDS thread names
log: cache recent threads up to a day
common: cache pthread names
log: concatenate thread names and print once per thread

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge PR #60327 into main

* refs/pull/60327/head:
mon/MDSMonitor: send reply to beacons with `state=DNE`

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #60381 into main

* refs/pull/60381/head:
doc: remove refrences to `mds_log_major_segment_event_ratio`
mds: start a new major segment after reaching minor segment threshold
mds: make parts of mdlog reusable to be used by beacon

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #60325 into main

* refs/pull/60325/head:
mds/Beacon: wake up the thread in shutdown()

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #60283 into main

* refs/pull/60283/head:
mds: add issue_seq to all cap messages
include/ceph_fs: correct ceph_mds_cap_peer field name
include/ceph_fs: correct ceph_mds_cap_item field name
messages/MClientCaps: use correct ceph_seq_t for cap sequence types
messages/MClientCaps: dump issue_seq for debugging
mds: remove dead code

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #60226 into main

* refs/pull/60226/head:
mds/QuiesceDbEncoding: add `inline` to work around linker error

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #58586 into main

* refs/pull/58586/head:
qa/suites/fs/thrash: add convolve operator for overrides
qa/cephfs: s/prefetch_dirfrags/prefetch_oft_dirfrags

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #60675 from zmc/make-dist-quiet

osd/scrub: list additional information when dumping the queue

Extend the information provided for operator dump commands, to
include the basic identity and scheduling information of the
entries in the scrub queue.

This change mostly benefits automatic QA and our internal
testing.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

rgw: Update README for DBStore and Posix drivers

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

crimson/osd/replicated_backend: add the dedicated method to construct
MOSDRepOp

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: add is_missing_on_peer()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: reuse already defined methods as many as possible to
improve readability

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/ops_executor: call OpsExecutor::apply_stats() before
OpsExecutor::obc is moved out.

OpsExecutor::apply_stats() accesses OpsExecutor::obc indirectly

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: fix dangerous lambda captures

In PG::submit_transaction(), osd_op_p was first moved to
ReplicatedBackend::submit_transaction(), and later its at_version is
captured in another lambda. The latter may be already empty at the time
of the capture.

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #60703 from zdover23/wip-doc-2024-11-12-rados-operations-add-or-rm-osds

doc/rados: correct "full ratio" note

Merge pull request #60704 from zdover23/wip-doc-2024-11-12-start-hardware-recommendations

doc/start: fix "are are" typo

Merge pull request #60705 from zdover23/wip-doc-2024-11-12-radosgw-account

doc/radosgw: s/Poliicy/Policy/