crimson/os/seastore/cache: monitor cache access by extent and trans

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: add asserts

Currently, cached must be stable, and absent must be clean.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction_manager: minor cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: report cache index sizes

Also rename Cache::extents to extents_index.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #58547 into main

* refs/pull/58547/head:
qa: failfast mount for better performance

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #59510 from afreen23/main

mgr/dashboard: Fix banner and scroll issue

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59182 from zhscn/wip-4k-laddr

crimson/os/seastore: make laddr_t block aligned

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #59507 from nbalacha/wip-nbalacha-rbd-manpage

doc/rbd: add namespace information for mirror commands

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #58950 into main

* refs/pull/58950/head:
mds: add the event when creating the batch head
mds: print the path depth to help debug mds issue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #59310 into main

* refs/pull/59310/head:
qa: load all dirfrags before testing altname recovery

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

doc/rbd: add namespace information for mirror commands

Updates the rbd man page to clarify  that some mirror commands
can apply to pools and namespaces.

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge pull request #59469 from smanjara/fix-multisite-script-default-realm

test-rgw-multisite: create default realm in multisite test script

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Fix banner and scroll issue

- includes fix to make banner visible again
- includes fix to allow scrolling on longer pages like, expand cluster and upgrade

Fixes https://tracker.ceph.com/issues/67795

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59363 from adk3798/cephadm-mount-nvmeof-certs

cephadm: mount nvmeof certs into container

Reviewed-by: Alexander Indenbaum <aindenba@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58778 from rhcs-dashboard/fix-67103-main

mgr/dashboard: Zone details showing incorrect data for data pool values and compression info for Storage Classes

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59495 from zhscn/fix-fmt-missing-arg

crimson/osd/pg: fix missing priority argument

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59391 from cyx1231st/wip-seastore-more-reports2

crimson/os/seastore/cache: report dirty cache usage and rewrite stats

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: fix missing priority argument

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Merge pull request #59390 from ceph/wip-oozmen-67656

doc/rgw/account: Handling notification topics when migrating an existing user into an account

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #53915 from pritha-srivastava/wip-rgw-sts-update-oidc-provider

rgw/iam: add AddClientIdToOIDCProvider/UpdateOidcProviderThumbprint

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #59249 from pritha-srivastava/wip-rgw-sts-err-message

rgw/sts: correcting the error message returned for an sts key

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #59300 into main

* refs/pull/59300/head:
client: calls to _ll_fh_exists() should hold client_lock

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #59419 from phlogistonjohn/jjm-smb-ctdb-vips

smb: cluster public ip addresses support

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Michael Adam <obnox@samba.org>

Merge pull request #59434 from VallariAg/fix-nvmeof-apply-teuthology

qa/tasks/nvmeof.py: add nvmeof gw-group to deployment

Merge pull request #59385 from leonidc/wip-leonidc-20242108-fixing-gw-bugs

mon: handle gw fast-reboot, proper handle of gw delete scenarios

Merge pull request #59240 from leonidc/wip-leonidc-20241508-upgrade-rules-centos9-only

upgrade rules for NVMeofGW monitors and gateways

Merge pull request #59401 from nbalacha/wip-nbalacha-check-mirror-ns

rbd-mirror: use correct ioctx for namespace

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

mgr/dashboard: Zone details showing incorrect data for data pool values and compression info for Storage Classes

Fixes: https://tracker.ceph.com/issues/67103
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #57876 from Suyashd999/rgw-false-positives

rgw: rgw_auth.cc: disable false use-after-move clang-tidy warning

Reviewed-By: Ronen Friedman <rfriedma@ibm.com>, Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #58959 from rhcs-dashboard/fix-67192-main

mgr/dashboard: Add Performance Details grafana charts for individual clusters in Manage-clusters page

Reviewed-by: Nizamudeen A <nia@redhat.com>

qa/suites/orch: add test for smb with ctdb and cluster public ips

Signed-off-by: John Mulligan <jmulligan@redhat.com>

doc: add documentation for (cluster_)public_addrs options

Document the spec and resource options (they're basically the same) for
specifying public addresses that will be managed automatically
by CTDB.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: add cluster public ip information to service spec

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: extend cluster resource type to define public ip addrs

When a cluster defines public IPs it will pass this information along to
the smb service spec.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: pass public addresses for a cluster to cephadm binary

Add the strictly-formed public addresses list as one of the config blobs
we pass to the binary for smb container deployment.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/deployment: add a cluster public ip spec for smb

This spec can be used to define one or more public addresses that will
be automatically assigned to hosts by CTDB. The address can be specified
in the "interface" form - an address plus prefix length.  Optionally,
networks to bind to can be specified. The network value will be
converted to a network device name later by cephadm.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

test/multisite: create default realm in multisite config

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #58380 from adk3798/squid-base-mds-upgrade-sequence-cephadm

qa/suites/fs: pull compiled cephadm for squid branch in mds_upgrade_sequence

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59421 from phlogistonjohn/jjm-teuth-cephadm-from-ctr

qa/tasks: add a new cephadm_from_container feature to cephadm task

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #59171 into main

* refs/pull/59171/head:
client: use vectors for context lists

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #59176 into main

* refs/pull/59176/head:
mds: encode quiesce payload on demand
mds: print quiesce message name in debug log

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #58419 into main

* refs/pull/58419/head:
mds: generate correct path for unlinked snapped files
qa: add test for cephx path check on unlinked snapped dir tree
mds: add debugging for stray_prior_path

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge PR #58987 into main

* refs/pull/58987/head:
qa/cephfs: update ignorelist

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #59088 into main

* refs/pull/59088/head:
mds: add compile time checks for sortedness
mds: sort conf keys

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #59095 into main

* refs/pull/59095/head:
qa: wait for file creation before changing mode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #59162 into main

* refs/pull/59162/head:
client: Prevent race condition when printing Inode in ll_sync_inode

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #59173 into main

* refs/pull/59173/head:
mds: fix spelling typo

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #59423 from idryomov/wip-67698

rbd: "rbd bench" always writes the same byte

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #59409 from adk3798/teuth-reinstall-nvme-cli

qa/distros: reinstall nvme-cli on centos 9 nodes

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #57952 from NitzanMordhai/wip-nitzan-bench-osd-admin-command

crimson: Add support for bench osd command

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59189 from xxhdx1985126/wip-67508

crimson/osd/recovery_backend: restart object pulls that are blocked by down osds

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59085 from VallariAg/update-default-nvmeof-img

mgr/cephadm: bump DEFAULT_NVMEOF_IMAGE to 1.2.17

Merge pull request #59433 from idryomov/wip-drop-xmlstarlet-variable

qa: drop XMLSTARLET variable, use xmlstarlet directly

Reviewed-by: Ramana Raja <rraja@redhat.com>

crimson/os/seastore: ensure the block size of device is always greater than laddr_t::UNIT_SIZE

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

rbd-mirror: use correct ioctx for namespace

The PoolReplayer uses the ioctx for the default namespace
to check if other namespaces are enabled for mirroring, causing
it to incorrectly conclude that they are all enabled.

Fixes: https://tracker.ceph.com/issues/67676
Signed-off-by: N Balachandran <nibalach@redhat.com>

crimson/osd/pg: add logs for repeating pulls

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #58870 from rhcs-dashboard/fix-67194-main

mgr/dashboard: fix typo in Multi-Cluster > Manager Cluster to Manage Clusters

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59376 from rhcs-dashboard/Upgrade-page-scroll-issue

mgr/dashboard: can't scroll to the end of the page

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

crimson/osd/recovery_backend: restart object pulling for recoveries that
are blocked pulling from down osds

Fixes: https://tracker.ceph.com/issues/67508
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/common/interruptible_future: new interruptor function `repeat_eagain`

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #59332 from afreen23/nvmeof-group-mtls

mgr/dashboard: Add group field in nvmeof service form

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59422 from cbodley/wip-67697

rgw: ignore zoneless default realm when not configured

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

cephadm: add support for cluster public ip addresses to smb daemon

When a list of public addresses (and optional network destination(s))
are supplied at deploy time, convert the networks to device names
and pass that result to the sambcc ctdb configuration.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/smb: simplify orch backend enablement

We have a developer/debug module option that allows one to disable
triggering orchestration. When I tried to use it I thought it was
buggy and I had trouble diagnosing it. The mistake was on my side,
but the code change makes it much clearer what is being enabled
so I want to keep it.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

doc/rgw/account: Handling notification topics when migrating an existing user into an account

Add a subsection under "Migrate an existing User into an Account" to
describe how a client can seamlessly migrate the notification topics
after account migration.

Fixes https://tracker.ceph.com/issues/67656

Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>

Merge pull request #59227 from xxhdx1985126/wip-67564

crimson/osd/pg: implement PG::PGLogEntryHandler::remove()

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59117 from cbodley/wip-67468

rgw/rados: zero-init shard_count in RGWBucket::check_index_unlinked()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #59172 from clwluvw/enoent-loglevel

rgw: increase log level for enoent caused by clients

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #59243 from cbodley/wip-67522

rgw/http: finish_request() after logging errors

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #59329 from smanjara/wip-data-sync-full-initialize

rgw/multisite: initialize sync_status in RGWDataFullSyncSingleEntryCR ctor

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57956 from tobias-urdin/remove-keystone-v2

rgw/auth: Remove Keystone v2.0 API support

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/rgw: include --rgw-realm/zonegroup/zone args for 'account create'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: test_multi.py creates realm with --default

mstart.sh relies on default realm/zonegroup/zone configuration, because
it doesn't supply them to radosgw as config options

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: ignore zoneless default realm when not configured

"default" zone/zonegroup deployments without a realm can be broken by
the creation of an unrelated realm, because that realm is (was)
automatically set as the default

when startup detects an incomplete default realm (one that doesn't have
a default zone), fall back to the realmless "default" zone/zonegroup
instead

Fixes: https://tracker.ceph.com/issues/67697
Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add 'realm default rm' command

the 'realm default' command could only set a different realm as the
default, and provided no way to clear the default

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #59301 from xxhdx1985126/wip-67604

crimson/common/tri_mutex: also wake up waiters when demoting

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #58136 from xxhdx1985126/wip-66372

crimson/osd/osd: mark down connections to downed osds

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #54620 from rishabh-d-dave/mgr-vol-clone-stats

mgr/vol: show progress and stats for the subvolume snapshot clones

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa/tasks/nvmeof.py: add nvmeof gw-group to deployment

Groups was made a required parameter to be
`ceph orch apply nvmeof <pool> <group>` in
https://github.com/ceph/ceph/pull/58860.
That broke the `nvmeof` suite so this PR fixes that.

Right now, all gateway are deployed in a single group.
Later, this would be changed to have multi groups for a better test.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

 mon/NVMeofGw*: fixing bugs - handle gw fast-reboot, proper handle of gw delete scenarios

Signed-off-by: Leonid Chernin <leonidc@il.ibm.com>

Merge pull request #59428 from zdover23/wip-doc-2024-08-26-cephadm-services-osd

doc/cephadm: how to get exact size_spec from device

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

mon/NVMeofGw*: support upgrades from prior out-of-tree nvmeofha implementation (nvmeof-reef)

This commit adds upgrade support for users running an experimental
nvmeofha implementation which can be found in the nvmeof-reef branch in
ceph.git.

Signed-off-by: Leonid Chernin <leonidc@il.ibm.com>

include/ceph_features: add NVMEOFHA feature bit

Normally, we'd just use the SERVER_SQUID or SERVER_T flags instead of
using an extra feature bit.  However, the nvmeof ha monitor paxos
service has had a more complex development journey.  There are users
interested in using the nvmeof ha feature in squid, but it didn't make
the cutoff for backporting it.  There's an upstream nvmeof-squid branch
in the ceph.git repository with the patches backported for anyone
interested in building it.

However, that means that users of our normal stable releases will see
the feature added to the monitor one release after anyone who chooses to
use the nvmeof-squid branch.  We could disallow upgrades from
nvmeof-squid to T, but by adding a feature bit here we make such a
restriction unnecessary.

Signed-off-by: Samuel Just <sjust@redhat.com>

include/ceph_features: remove stray available marker

Should have been removed in caa9e7a45e.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: Add support for bench osd command

this commit adds support for the 'bench' admin command in the OSD,
allowing administrators to perform benchmark tests on the OSD. The
'bench' command accepts 4 optional parameters with the following
default values:

1. count - Total number of bytes to write (default: 1GB).
2. size - Block size for each write operation (default: 4MB).
3. object_size - Size of each object to write (default: 0).
4. object_num - Number of objects to write (default: 0).

The results of the benchmark are returned in a JSON formatted output,
which includes the following fields:

1. bytes_written - Total number of bytes written during the benchmark.
2. blocksize - Block size used for each write operation.
3. elapsed_sec - Total time taken to complete the benchmark in seconds.
4. bytes_per_sec - Write throughput in bytes per second.
5. iops - Number of input/output operations per second.

Example JSON output:

```json
{
  "osd_bench_results": {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.5,
    "bytes_per_sec": 2147483648,
    "iops": 512
  }
}

Fixes: https://tracker.ceph.com/issues/66380
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #59392 from cyx1231st/wip-inplace-rewrite-comments

crimson/os/seastore: refine documents related to inplace rewrite

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

doc/cephadm: how to get exact size_spec from device

Add instructions for retrieving the exact size of block devices.

Fixes: https://tracker.ceph.com/issues/66754
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59053 from baum/wip-baum-20240806-00

nvmeof/NVMeofGwMonitorClient: use a separate mutex for beacons

Merge pull request #58858 from ronen-fr/wip-rf-entry

osd/scrub: a scrub queue of level-specific entries

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

test/osd/scrub: fix searched-for log string

To match the modified log message in
OsdScrub::restrictions_on_scrubbing().

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: fix missing 'const' on some formatters

required to pass CI checks.

co-author: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

test/osd/scrub: disable tests for deleted scrub functionality

The scrub scheduler no longer "upgrades" shallow scrubs into
deep ones on error, so the tests that check this functionality
are no longer valid.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

test/osd: test new functionality added to the not-before queue

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: delay both targets on some failures

If the failure of a scrub-job is due to a condition that affects
both targets, both should be delayed. Otherwise, we may end up
with the following bogus scenario:

A high priority deep target is scheduled, but scrub session initiation
fails due to, for example, a concurrent snap trim. The deep target
will be delayed. A second initiation attempt may happen after the
snap trimming is done, but before the updated deep target not-before.
As a result - the lower priority target will be scheduled before the
higher priority one - which is a bug.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: reverse OSDRestrictions flags polarity

As most of the flags in OSDRestrictions are of 'true is bad' polarity,
reverse the two non-conforming flags - cpu load and time-of-day
restrictions - to match.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: fix the conditions for auto-repair scrubs

The conditions for auto-repair scrubs should have been changed
when need_auto lost some of its setters.

Also fix the rescheduling of repair scrubs
when the last scrub ended with errors.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: remove requested_scrub_t::deep_scrub_on_error

This flag was used to indicate that a deep scrub should
be performed if a shallow scrub finds an error. It was
always set true for shallow, regular, scrubs - if
can_autorepair flag was set. Thus, the ephemeral flag in
the requested_scrub_t object is not really needed.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

qa/standalone/scrub: disable scrub_extended_sleep test

Disabling osd-scrub-test.sh::TEST_scrub_extended_sleep,
as the test is no longer valid (updated code no longer
produces the same logs or the same behavior).

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: remove non-display usage of target's is_high_priority()

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: remove 'calculated_to_deep' flag

as once a sched-target was selected, we know the level of the scrub.
Also removed: the ephemeral 'time_for_deep' flag.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>