common/gated: enable ceph_assert on shard id

Checking that each call to gate on the same shard the gate
assigned to.

Fixes: https://tracker.ceph.com/issues/67773
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #59732 from zdover23/wip-doc-2024-09-11-README-md-formatting

doc/README.md: improve formatting

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #54777 from cbodley/wip-rgw-meta-handlers

rgw: remove unused backend abstraction for metadata handlers

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

rgw/service: remove unused metadata and metadata backend services

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/services: remove unused RGWSI_Role_RADOS

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/role: add rgwrados::role interface for role metadata

move everything from class RadosRole into driver/rados/role.cc and use
RGWSI_SysObj instead of the metadata backend. narrows the RGWRole
interface by handling the name/path objects internally

rgwrados::role::write() adds/remove name and path linkages in case they
change. admin ops don't allow this, but 'metadata put' could upload
arbitrary json that does change them

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: use raw RGWMetadataHandler for role metadata

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: encapsulate role metadata handler in rgw_role.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: use raw RGWMetadataHandler for user metadata

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/user: remove unused 'meta backend ctx' arguments

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/user: call sysobj directly instead of 'sysobj meta backend'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: use raw RGWMetadataHandler for bucket instances

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: use raw RGWMetadataHandler for bucket entrypoints

RGWBucketMetadataHandler inherits from RGWMetadataHandler directly
instead of RGWMetadataHandler_GenericMetaBE

replace RGWSI_Bucket_SObj_Module with the RGWMetadataLister abstraction

RGWSI_Bucket_SObj now depends on RGWSI_MDLog and writes mdlog entries
itself

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/bucket: remove unused 'meta backend ctx' arguments

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/bucket: call sysobj directly instead of 'sysobj meta backend'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/bucket: free functions for bucket instance metadata keys

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/services: remove otp service and metadata backend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: remove backend stuff from otp metadata handler

inherit from RGWMetadataHandler instead of
RGWMetadataHandler_GenericMetaBE to avoid all of the metadata backend
abstractions, and just call into the cls and sysobj services directly

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/services: remove unused RGWOTPCtl

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: clear output keys in RGWMetadataLister::get_next()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw/posix: add missing include for cpp_strerror()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #59696 from wanglinke521/wlk-perfect-comment

os/bluestore: perfect comments in hybrid_allocator

Reviewed-by: Adam Kupczyk <kupczyk@ibm.com>
Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #59438 from mkogan1/wip-d3n-ex-co-ar

doc/rgw/d3n: pass cache dir volume to extra_container_args

Merge pull request #59634 from rkachach/fix_issue_67934

mgr/cephadm: adding spec fields for oauth2-proxy whitelist_domains

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #59456 from xxhdx1985126/wip-67731

crimson/osd/pg: clear backfill_state when the PG goes clean

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59652 from xxhdx1985126/wip-67945

crimson/osd/osd_operations/snaptrim_event: only increase osd_osd_param_t::at_version after clone adjustments

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59564 from Matan-B/wip-matanb-crimson-osd-release

crimson/osd: write require_osd_release only when needed

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #59506 from Matan-B/wip-matanb-crimson-seastore-logs

qa/config/seastore: enable all subsystem debug

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #59753 from wodeshijie33941/main

doc/install: Keep the name field of the created user consistent with …

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59741 from yuvalif/wip-yuval-68033

rgw/kafka: refactor topic creation to avoid rd_kafka_topic_name()

Reviewed-By: Casey Bodley <cbodley@ibm.com>

doc/install: Keep the name field of the created user consistent with the node name in the Start RADOSGW service command

If the user name does not match the name of the node that started the RADOSGW service, this will cause confusion for those who are new to ceph. Because they can't start the radosgw service normally as shown in the tutorial.

Signed-off-by: hejindong <wodeshijie3394@126.com>

Merge pull request #59482 from rhcs-dashboard/nvmeof-api-group-support

mgr/dashboard: add gw_groups support to nvmeof api

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: caroav <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #59576 from rhcs-dashboard/hosts-selection-show-only-10-entries-fix

mgr/dashboard: service form hosts selection only show up to 10 entries

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59738 from gbregman/main

Make the discovery and gateway IPs configurable in NVMEof configuration

Merge pull request #59681 from adk3798/cephadm-enable-module-bootstrap-retry

cephadm: bootstrap: verify orch module is running before setting cephadm backend

Reviewed-by: John Mulligan <jmulligan@redhat.com>

rgw/kafka: refactor topic creation to avoid rd_kafka_topic_name()

also, use unique_ptr for safer memory management.

Fixes: https://tracker.ceph.com/issues/68033
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

mgr/cephadm: Make the discovery and gateway IPs configurable in NVMEof configuration
Fixes https://tracker.ceph.com/issues/68030

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

python-common/ceph/deployment: Make the discovery and gateway IPs configurable in NVMEof configuration
Fixes https://tracker.ceph.com/issues/68030

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #59635 from zdover23/wip-doc-2024-09-06-rados-ops-health-checks

doc/rados: add "pgs not deep scrubbed in time" info

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/README.md: improve formatting

Improve the formatting in the section "Building Ceph" in the file
README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: expose gw_groups list api

also if the cephadm is available, make the api smart enough to
understand the running daemons and get its gateway_addr instead of
taking the first one in the list so that the HA will be there for the UI
and API as well

Fixes: https://tracker.ceph.com/issues/67774
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: add gw_groups support to nvmeof api

Introduces a gw_group query-param to gateway and subsystem requests.

Modified the dashboard config to support adding more than one gateway to
the list.

Fixes: https://tracker.ceph.com/issues/67774
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59066 from xxhdx1985126/wip-67380

crimson/osd: cancel ongoing pglog-based recoveries on recovery defering

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59485 from mcv21/cephadm_image_warn

cephadm: emit warning if daemon's image is not to be used

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #59599 from rhcs-dashboard/fix-67893-main

mgr/dashboard: Rephrase dedicated pool helper in rbd create form

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59447 from afreen23/wip-nvmeof-microcopy

mgr/dashboard: Update nvmeof microcopies

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59569 from afreen23/wip-nvme-ns

mgr/dashboard: Increase maximum namespace count to 1024

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #56858 from rhcs-dashboard/fix-grafana-server-depreciation

mgr/dashboard: use grafana server instead of grafana-server in grafana 10.4.0

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@ibm.com>

Merge pull request #59610 from adk3798/cephadm-ssh-configurable-keepalive

mgr/cephadm: make ssh keepalive settings configurable

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59643 from rkachach/fix_issue_67939

mgr/cephadm: changing the nginx upstream image used by mgmt-gateway

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #59658 from phlogistonjohn/jjm-remove-object-deleting-func

mgr/smb: stop trying to clean external store during cluster sync

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #58994 from phlogistonjohn/jjm-smb-vfs-ceph-new

mgr/smb: add support for new ceph vfs module

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>

Merge pull request #58880 from aclamk/wip-aclamk-bs-faster-objectstore-test

test/store_test: Make ceph_test_objectstore run faster

Merge pull request #59107 from nbalacha/wip-nbalacha-async-sorted-snaps

librbd: make "group snap list" async and optionally sorted by snap creation time

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

crimson/osd/ops_executer: cleanup snap_map_*

Since https://github.com/ceph/ceph/pull/58868, snap map updates
are now maintained via PG::update_snap_map.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/snaptrim_event: only increase
osd_osd_param_t::at_version after clone adjustments

Fixes: https://tracker.ceph.com/issues/67945
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/osd_operations/snaptrim_event: we do SnapMapper updates in
ReplicatedBackend::submit_transaction(), drop all SnapMapper updates
before that

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #59675 from zdover23/wip-doc-2024-09-10-README-md-vstart

doc/README.md: improve formatting

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

os/bluestore: perfect comments in hybrid_allocator

co-author: Jrchyang Yu <yuzhiqiang_yewu@cmss.chinamobile.com>
Signed-off-by: Wang Linke <wanglinke_yewu@cmss.chinamobile.com>

Merge pull request #58986 from NitzanMordhai/wip-nitzan-crimson-op_gate-alianstore-multi-core

crimson: use gate per shard for AlienStore and OSD

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #58868 from xxhdx1985126/wip-crimson-clean_region-based-clone-recovery

crimson/osd/pg: properly propagate snap mapper updates and do clean-region-based clone objects recovery

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

mgr/dashboard: service form hosts selection only show up to 10 entries

exposed new UIRouter API to get all hosts

Fixes: https://tracker.ceph.com/issues/67413
Signed-off-by: Naman Munet <nmunet@redhat.com>

Merge pull request #59677 from zdover23/wip-doc-2024-09-10-man-ceph-bluestore-tool

doc/man: edit ceph-bluestore-tool.rst

cephadm: bootstrap: verify orch module is running before setting cephadm backend

There have been occasional failures when bootstrapping with error
messages like

```
Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it
```

despite the orchestrator being meant to be an always on module.

Since trying to enable an already enabled module causes no issues,
trying to enable it ourselves seems like a sensible workaround.

Fixes: https://tracker.ceph.com/issues/67969
Signed-off-by: Adam King <adking@redhat.com>

doc/man: edit ceph-bluestore-tool.rst

Edit the section "bluefs-bdev-migrate" in
doc/man/8/ceph-bluestore-tool.rst to add the information that this
operation expands the target storage by updating its size label, making
"bluefs-bdev-expand" unnecessary.

Improve the subject-verb agreement in this section, and supply some
absent definite articles.

Co-authored-by: Peter Gervai <grin@drop.grin.hu>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58603 from cbodley/wip-rgw-sal-concrete-lc

rgw/sal: LCHead and LCEntry don't need abstraction

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

doc/README.md: improve formatting

Improve the formatting of the section "Building Ceph" in the file
README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/cephadm: changing the nginx upstream image used by mgmt-gateway

pointing mgmt-gateway nginx image to a different upstream repository

Fixes: https://tracker.ceph.com/issues/67939
Signed-off-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #59297 from ivoalmeida/carbon-datatable-performance

mgr/dashboard: datatable performance improvement

Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/smb: stop trying to clean external store during cluster sync

It was found during testing that a sequence of commands like:
```
ceph smb cluster create slow1 user --define-user-pass=user1%badf00d --clustering=always
--placement=3
sleep 0.5
ceph smb share create slow1 share1 cephfs --subvolume=g1/sv1 --path=/
sleep 0.5
ceph smb share create slow1 share2 cephfs --subvolume=g1/sv2 --path=/
```
would create a CTDB enabled cluster that would fail to start up
correctly. The issue was due to the call to `external.rm_other_in_ns`
during the cluster sync operation. In the CTDB enabled mode, objects are
written to the pool outside of the smb mgr module's direct control, in
particular `cluster.meta.json`, and this function, intended to keep the
pool & namespace tidy, was removing objects needed by CTDB-enabled mode.
The failure is somewhat timing sensitive due to the ctdb enablement
sidecars coming up before or after the object was deleted.

Remove this function call so that these objects stop getting deleted at
inopportune times. While we could have tried making this function
"smarter" and only deleting some unexpected objects, in this case I feel
that keeping it simple is better. If we find this pool getting cluttered
in the future we can add a smarter pool-tidying-up function later.

Fixes: https://tracker.ceph.com/issues/67946
Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58815 from synarete/samba-metrics-exporter

cephadm: samba metrics exporter

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59534 from cbodley/wip-rgw-listing-max-entries

rgw: raise default rgw_max_listing_results=5000

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #57635 from clwluvw/trans-id

rgw: log x-amz-request-id along with the request

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

Merge pull request #58082 from jzhu116-bloomberg/wip-66516

rgw/multipart: use cls_version to avoid racing between part upload and multipart complete

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #59014 from liangmingyuanneo/wip-bugfix-beast-socket-accept

rgw/beast: optimize for accept when meeting error in listenning

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/osd: check_osdmap_features - add log

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/ops_executer: pass MutFunc by value

Since we now co_await mut_func, we should not pass it by rvalue ref.

```
DEBUG 2024-09-01 15:54:46,212 [shard 0:main] osd - do_osd_ops_execute: object 2:c4c92e5a:::rbd_trash:head submitting txn
=================================================================
==17416==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f590008a430 at pc 0x0000040a367a bp 0x7ffc0b1d5ff0 sp 0x7ffc0b1d5fe0
Address 0x7f590008a430 is located in stack of thread T0 at offset 48 in frame
    #0 0x40b0a2b in crimson::osd::PG::do_osd_ops_execute ... lambda(std::error_code const&)#1}&&)::{lambda()#1}::operator()() const (/usr/bin/ceph-osd+0x40b0a2b)
```

Co-authored-by: Xuehan Xu <xuxuehan@qianxin.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/ops_executer: avoid seastar::async comment

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/snaptrim_event: make use of process_and_submit

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/snaptrim_event: introduce process_and_submit

We should be carful with using capturing lambdas with coroutines.
Avoid capturing [this] by passing process_and_submit instead.

See: https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rcoro-capture

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/ops_executer: flush_changes_n_do_ops_effects to use
coroutines

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/snaptrim_event: start to use coroutines

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: make use of SnapMapper::update_snap_map

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

mgr/dashboard: datatable performance improvement

Fixes: https://tracker.ceph.com/issues/67796:
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge pull request #59657 from eriksjolund/fix-sign-of-errno

tools/rbd_ggate: fix sign of errno

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #59000 from connorfawcett/rbd-random

rbd/bench: Add ability to write a random string of bytes or a specific byte with --pattern-byte option

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #59607 from yuvalif/wip-yuval-67658

rgw/notifications: free completion pointer using unique_ptr

Reviewed-By: Adam Emerson <aemerson@ibm.com>, Casey Bodley <cbodley@ibm.com>

Merge pull request #59642 from zdover23/wip-doc-2024-09-06-cephadm-upgrade

doc/cephadm: add malformed-JSON removal instructions

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #59553 from cyx1231st/wip-seastore-more-reports3

crimson/os/seastore/cache: report cache access stats

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/cephadm: add malformed-JSON removal instructions

Add instructions for removing malformed JSON that causes the cephadm
module to fail to start, which in turn makes the orchestrator fail to
start.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59590 from ronen-fr/wip-rf-delay-conf

osd/scrub: add configuration parameters to control delay duration

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

tools/rbd_ggate: fix sign of errno

Signed-off-by: Erik Sjölund <erik.sjolund@gmail.com>

Merge pull request #59628 from zdover23/wip-doc-2024-09-06-dev-release-checklist-rtd-activation

doc/dev: add "activate latest release" RTD step

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #59592 from leonidc/wip-leonidc0409-blocklist-fix

leonidc0409 blocklist fix

Merge pull request #44640 from effi-ofer/live-migration-nbd

librbd/migration: add NBD stream

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #57388 from mchangir/mgr-snap_schedule-correctly-fetch-mds_max_snaps_per_dir-from-mds

mgr/snap_schedule: correctly fetch mds_max_snaps_per_dir from mds

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59579 from leonidc/wip-leonidc0309-gw-deleting-state

 gw deleting state

mgr/cephadm: stop running "true" before every command

It turns out this is completely unnecessary. With the keepalive
settings on the connection, we no longer need to run an
additional command with a timeout on it to quickly detect a
host is offline

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #44359 from mchangir/mds-uninline-file-during-scrub

mds: un-inline data on scrub

Reviewed-by: Xiubo Li <xiubli@redhat.com>

doc/rados: add "pgs not deep scrubbed in time" info

Add a procedure to doc/rados/operations/health-warnings.rst that
explains how to remedy the "X PGs not deep-scrubbed in time" health
warning.

This procedure was developed by Eugen Block, and is at the time of this
commit available on his blog at
https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-time/

Co-authored-by: Eugen Block <eblock@nde.ag>
Signed-off-by: Zac Dover <zac.dover@proton.me>