doc/cephadm: Clarify "Deploying a new Cluster"

Change the title of the section "Deploying a new Ceph cluster" to "Using
cephadm to Deploy a New Ceph Cluster". This is part of the initiative to
separate package-related documentation from container-based
documenation.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60775 from cbodley/wip-rgw-access-key-active-json

rgw: RGWAccessKey::decode_json() preserves default value of 'active'

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #58645 from vshankar/wip-omap-commit-header-size

mds: account for header size during omap commit

Merge pull request #60488 from jmolmo/exporter_manage_sigterm

exporter: SIGTERM/SIGINT/SIGHUP management

Merge pull request #60787 from ronen-fr/wip-rf-reservStat

osd/scrub: fixing reservations dump format

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #60636 from mohit84/issue_68585

TEST_backfill_grow fails after finding "num_bytes mismatch" in osd log

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #60253 from MaxKellermann/includes

common: add missing include, reduce header dependencies

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #59524 from liangmingyuanneo/wip-standalone-test-pg-repair

qa/standalone: bugfix for wait_for_scrub

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #60523 from ljflores/wip-tracker-68657

mgr/balancer: optimize 'balancer status detail'

Merge pull request #60542 from adk3798/sign-generated-rgw-certs

mgr/cephadm: sign generated RGW certs

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #60303 from adk3798/handle-failed-daemon-start

cephadm: handle "systemctl start" failures during deployment better

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60531 from Kushal-deb/fix_issue_2124175

mgr/cephadm: Added fix for Modifying Profiles(add-setting and rm-setting) to allow multiple setting changes

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60587 from phlogistonjohn/jjm-more-py312-fixes

various python 3.12 fixes

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60319 from baum/no_huge

cephadm: support no huge pages for nvmeof spdk

osd/scrub: fixing reservations dump format

Headers were missing in the faulty dump output. This commit
fixes the section type to 'object section'.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

cephadm/nvmeof: support no huge pages for nvmeof spdk

depends on: https://github.com/ceph/ceph-nvmeof/pull/898

Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

Merge pull request #60668 from ronen-fr/wip-rf-mconf

osd/scrub: cache frequently used configuration parameters

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #60769 from gbregman/main

mgr/cephadm: Add resource limits to nvmeof configuration

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60588 from AlexonOliveiraRH/wip-doc-templates

doc/cephadm/services: Update monitoring.rst to add additional templates

rgw: RGWAccessKey::decode_json() preserves default value of 'active'

squid added the new "bool active" member that defaults to true, but
RGWAccessKey::decode_json() sets it to false when "active" isn't present

this is an issue for multisite when a squid zone replicates user
metadata from a master zone running reef

Fixes: https://tracker.ceph.com/issues/68985
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60598 from xxhdx1985126/wip-68808

crimson/osd/replicate_backend: add the skipped newly created clone object to the push queue after the clone request completes

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

common: rm the cast operator of md_config_cacher_t

as its replacement - the type-deduced operator() - is now in use,
and the resulting code is more readable and less error-prone.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd: rm all uses of the cast operator of md_config_cacher_t

as this interface is to be removed in the next commit.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: cache frequently used configuration parameters

using the md_config_cacher_t, which is a cache object that registers
itself to the config observer and caches the up-to-date configuration
value.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

common: a simple API to extract md_config_cacher_t cached value

Using a type-deduced operator() (i.e. my_opt() ), instead of the
existing ValueT() operator (which requires a type-matching cast).

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

mgr/cephadm: Add resource limits to nvmeof configuration
Fixes https://tracker.ceph.com/issues/68967

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

python-common/ceph/deployment: Add resource limits to nvmeof configuration
Fixes https://tracker.ceph.com/issues/68967

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #60572 from xxhdx1985126/wip-68787

crimson/osd/backfill_state: don't try to do object backfills when primary_bi is empty

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #60661 from ronen-fr/wip-rf-scheduledeep

osd/scrub: fix 'schedule-deepscrub' test asok command

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #60674 from oshrey16/bugfix-rgw-ps-list-topics-op

rgw/notification: Fix segmentation fault in RGWPSListTopicsOp::execute() and correct topic listing to use get_topics_v2

Reviewed-By: Joseph Mundackal <jmundackal@bloomberg.net>, Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #60765 from zdover23/wip-doc-2024-11-19-os-recommendations

doc/start: update os-recommendations.rst

doc/start: update os-recommendations.rst

Remove information about the operating systems that support Ceph's
official container images from the "Platforms" table in
doc/start/os-recommendations.rst and add that information to the (new)
table that shows the operating systems that support Ceph's official
container images.

Credit for this change should go to Enrico Bocchi, who noticed a
discrepancy that motivated it.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60309 from linuxbox2/wip-cksum-68537

rgw_cksum:  treat mpu checksums w/o part count as valid in CompleteMu…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60711 from soumyakoduri/wip-skoduri-dbstore

rgw: Update README for DBStore and Posix drivers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mds: account for header size during omap commit

fnode_t is set in the omap header during directory commit
operation which isn't accounted when tracking operation
size.

Fixes: http://tracker.ceph.com/issues/67597
Signed-off-by: Venky Shankar <vshankar@redhat.com>

rgw/notification: fix segmentation fault and topic listing logic

- Fixed a segmentation fault caused by a null bucket pointer in RGWPSListTopicsOp::execute()
- Corrected logic to use get_topics_v2 when supported, with fallback otherwise

Fixes: https://tracker.ceph.com/issues/68756
Signed-off-by: Oshrey Avraham <oshrey16@gmail.com>

crimson/osd/replicate_backend: add the skipped newly created clone
object to the push queue after the clone request completes

Fixes: https://tracker.ceph.com/issues/68808
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

exporter: SIGTERM/SIGINT/SIGHUP management

Ceph exporter manages SIGINT, SIGTERM and SIGHUP signals

Fixes: https://tracker.ceph.com/issues/68721
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #60655 from xxhdx1985126/wip-seastore-move-out-root-meta

crimson/os/seastore: move the root meta out of the root block

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #60209 from myoungwon/wip-init-onode-layout

crimson/os/seastore: initialize oi and ss in onode_layout_t to zero

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #60715 from ronen-fr/wip-rf-dumpsched

osd/scrub: list additional information when dumping the queue

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #60727 from guits/dmcrypt-activate

ceph-volume: fix dmcrypt activation regression

Merge pull request #60404 from VallariAg/wip-nvmeof-listeners-prometheus-alerts

monitoring: add 2 nvmeof alerts to prometheus_alerts.yaml

Merge pull request #51766 from zhsgao/mds_clear_session_failed

mds: session in the importing state cannot be cleared if an export subtree task is interrupted while the state of importer is acking

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #60067 from VallariAg/wip-nvmeof-healthcheck

mon: add nvmeof healthchecks

ceph-volume: fix dmcrypt activation regression

`ceph.with_tpm` tag is actually a `str`, which means that
`bool('0')` will return `True`.

Fixes: https://tracker.ceph.com/issues/68944
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #60554 from jshen28/fix-legacy-addr-mismatch

Make mon_info_t consistent with mon_addr

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mds: client is evicted when an export subtree task is interrupted

The importer will force open some sessions provided by the exporter but the client does not know about
the new sessions until the exporter notifies it, and the notifications cannot be sent if the exporter
is interrupted. The client does not renew the sessions regularly that it does not know about, so the client
will be evicted by the importer after `session_autoclose` seconds (300 seconds by default).

The sessions that are forced opened in the importer need to be closed when the import process is reversed.

Signed-off-by: Zhansong Gao <zhsgao@hotmail.com>

qa: add test for importer's unexpected client eviction after an export subtree task is interrupted

Signed-off-by: Zhansong Gao <zhsgao@hotmail.com>

mds: session in the importing state cannot be cleared if an export subtree task is interrupted while the state of importer is acking

The related sessions in the importer are in the importing state(`Session::is_importing` return true) when the state of importer is `acking`,
`Migrator::import_reverse` called by `MDCache::handle_resolve` should reverse the process to clear the importing state if the exporter restarts
at this time, but it doesn't do that actually because of its bug. And it will cause these sessions to not be cleared when the client is
unmounted(evicted or timeout) until the mds is restarted.

The bug in `import_reverse` is that it contains the code to handle state `IMPORT_ACKING` but it will never be executed because
the state is modified to `IMPORT_ABORTING` at the beginning. Move `stat.state = IMPORT_ABORTING` to the end of import_reverse
so that it can handle the state `IMPORT_ACKING`.

Fixes: https://tracker.ceph.com/issues/61459
Signed-off-by: Zhansong Gao <zhsgao@hotmail.com>

qa: add test for importer's session cleanup after an export subtree task is interrupted

Signed-off-by: Zhansong Gao <zhsgao@hotmail.com>

qa/suites/nvmeof: add nvmeof warnings to log-ignorelist

Add NVMEOF_SINGLE_GATEWAY and NVMEOF_GATEWAY_DOWN
warnings to nvmeof:thrash job's log-ignorelist

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

mds: the assert should be before the journal entry submit otherwise it's racy

Signed-off-by: Zhansong Gao <zhsgao@hotmail.com>

mds: add `importing_count` to session dump

Signed-off-by: Zhansong Gao <zhsgao@hotmail.com>

Merge PR #60464 into main

* refs/pull/60464/head:
mds: add or update MDS thread names
log: cache recent threads up to a day
common: cache pthread names
log: concatenate thread names and print once per thread

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge PR #60327 into main

* refs/pull/60327/head:
mon/MDSMonitor: send reply to beacons with `state=DNE`

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #60381 into main

* refs/pull/60381/head:
doc: remove refrences to `mds_log_major_segment_event_ratio`
mds: start a new major segment after reaching minor segment threshold
mds: make parts of mdlog reusable to be used by beacon

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #60325 into main

* refs/pull/60325/head:
mds/Beacon: wake up the thread in shutdown()

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #60283 into main

* refs/pull/60283/head:
mds: add issue_seq to all cap messages
include/ceph_fs: correct ceph_mds_cap_peer field name
include/ceph_fs: correct ceph_mds_cap_item field name
messages/MClientCaps: use correct ceph_seq_t for cap sequence types
messages/MClientCaps: dump issue_seq for debugging
mds: remove dead code

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #60226 into main

* refs/pull/60226/head:
mds/QuiesceDbEncoding: add `inline` to work around linker error

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #58586 into main

* refs/pull/58586/head:
qa/suites/fs/thrash: add convolve operator for overrides
qa/cephfs: s/prefetch_dirfrags/prefetch_oft_dirfrags

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #60675 from zmc/make-dist-quiet

osd/scrub: list additional information when dumping the queue

Extend the information provided for operator dump commands, to
include the basic identity and scheduling information of the
entries in the scrub queue.

This change mostly benefits automatic QA and our internal
testing.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

rgw: Update README for DBStore and Posix drivers

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

crimson/osd/replicated_backend: add the dedicated method to construct
MOSDRepOp

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: add is_missing_on_peer()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: reuse already defined methods as many as possible to
improve readability

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/ops_executor: call OpsExecutor::apply_stats() before
OpsExecutor::obc is moved out.

OpsExecutor::apply_stats() accesses OpsExecutor::obc indirectly

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: fix dangerous lambda captures

In PG::submit_transaction(), osd_op_p was first moved to
ReplicatedBackend::submit_transaction(), and later its at_version is
captured in another lambda. The latter may be already empty at the time
of the capture.

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #60703 from zdover23/wip-doc-2024-11-12-rados-operations-add-or-rm-osds

doc/rados: correct "full ratio" note

Merge pull request #60704 from zdover23/wip-doc-2024-11-12-start-hardware-recommendations

doc/start: fix "are are" typo

Merge pull request #60705 from zdover23/wip-doc-2024-11-12-radosgw-account

doc/radosgw: s/Poliicy/Policy/

doc/radosgw: s/Poliicy/Policy/

Correct "AttachGroupPoliicy" to "AttachGroupPolicy".

Fixes: https://tracker.ceph.com/issues/68902
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/start: fix "are are" typo

Fix typo reading "They are are single-threaded".
s/are are/are/

Fixes: https://tracker.ceph.com/issues/68901
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60092 from connorfawcett/vstart_runner_bytes_fix

qa/vstart-runner: Only remove non-ASCII characters from string-based command outputs

doc/rados: correct "full ratio" note

Correct a note that directed users not to add an OSD after the cluster
has reached its "full ratio". The note now says "Do not let your cluster
reach its full ratio before adding an OSD."

Hat tip: Oskar Berggren

Fixes: https://tracker.ceph.com/issues/68900
Co-authored-by: Oskar Berggren <oskar.berggren@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

monitoring: add tests for 2 new nvmeof alerts

Add test for alerts NVMeoFMissingListener and
NVMeoFZeroListenerSubsystem to test_alerts.yml.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

Merge pull request #60529 from zdover23/wip-doc-2024-10-29-start-os-recommendations

doc/start: separate package chart from container chart

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #59328 from mkogan1/wip-d3n-umask

rgw: d3n: set access permissions for the D3N cache directory when creating

doc/start: separate package chart from container chart

Separate the packages-and-containers chart into two charts:
   (1) a chart that shows which OSes Ceph builds packages for
   (2) a chart that shows which OSes support Ceph's containers

Signed-off-by: Zac Dover <zac.dover@proton.me>

monitoring: add 2 new nvmeof alerts

Add NVMeoFMissingListener and NVMeoFZeroListenerSubsystem
alerts to prometheus_alerts.libsonnet.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

monitoring: add 2 nvmeof alerts to prometheus_alerts.yaml

- `NVMeoFMissingListener`: trigger if all listeners
     are not created for each gateway in a subsystem
- `NVMeoFZeroListenerSubsystem`: trigger if a subsystem has no listeners

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

Merge pull request #60626 from ivoalmeida/carbon-version-update

mgr/dashboard: update carbon-components-angular

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60511 from rhcs-dashboard/health-test-fix

qa/dashboard: fix tasks.mgr.dashboard.test_health.HealthTest

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

test/crimson/seastore/test_object_data_handler: always access object
data based on offsets relative to the start of the object data base

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

test/crimson/seastore/test_transaction_manager: take RootMetaBlock into
account

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cache: throw enospc when failed to alloc new extents

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: move the root meta out of the root block

During massive data backfilling, new osdmaps keep being created due to
frequent pg status changes, which can lead to frequent osd meta updates.
Those updates will be translated into "SeaStore::write_meta"s, which
modifies the root block's meta field and invalidates all inflight
transactions. Since the osd meta updates can be very frequent, long
transactions may be kept invalidated and the corresponding IO requests
hang.

This commit moves the root meta out of the root block, so that updates
to it won't invalidate irrelevant transactions

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

qa/vstart-runner: Only remove non-ASCII characters from string-based command outputs

Signed-off-by: Connor Fawcett <connorfa@uk.ibm.com>
Fixes: https://tracker.ceph.com/issues/68368

Merge pull request #60663 from leonidc/wip-publish-rebalance-ana-group

 mon/nvmeofgw*: monitors publish in nvme-gw show ana group responsible

Merge pull request #60605 from zdover23/wip-doc-2024-11-04-cephadm-services-osd

doc/cephadm: clean "Adv. OSD Service Specs"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60592 from athanatos/sjust/wip-crimson-io-2

crimson: switch to simple RAII obc locking mechanism

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

make-dist: Quiet submodule operations in Jenkins

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #60578 from zmc/rpm-sccache-toggle

Merge pull request #60621 from galsalomon66/mask_parquet_dependencies

rgw: hiding parquet dependencies upon parquet-select option is off

qa/dashboard: fix test_list_enabled_module failure

Check the ports availability and go for a new port if the current one is
not available

Fixes: https://tracker.ceph.com/issues/62972
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #60448 from anthonyeleven/improve-mgr-disable

doc: improve pending release notes and CephFS

Reviewed-by: Zac Dover <zac.dover@proton.me>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

 mon/nvmeofgw*: monitors publish in nvme-gw show ana group responsible
 for  namespace rebalance

Signed-off-by: Leonid Chernin <leonidc@il.ibm.com>

ceph.spec.in: Add option to enable sccache

Signed-off-by: Zack Cerza <zack@redhat.com>

mgr/dashboard: update carbon-components-angular

* update carbon-components-angular pkg to v5.48.0
* fixed change detection errors on unit tests
* fixed pagination page length when limit is 0 and data is empty

Fixes: https://tracker.ceph.com/issues/68837
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>