test/rgw: clean up unused include dirs

store/rados was renamed to driver/rados, so these include directories
aren't required

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50419 from myoungwon/wip-rbm-partial-overwrite

crimson/os/seastore: introduce delta-based overwrite

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #54379 from cbodley/wip-rgw-sal-bucket-owner

rgw/sal: Bucket owner as rgw_user

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #54440 from zdover23/wip-doc-2023-11-10-rados-troubleshooting-mon-common-issues-4-of-x

doc/rados: edit t-mon "common issues" (4 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #54415 from jincreator/cls-fmt-ftbfs

cls: fix unable to link with system-installed fmt

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/rados: edit t-mon "common issues" (4 of x)

Edit the fourth part of the section "Most Common Monitor Issues" in
doc/rados/troubleshooting/troubleshooting-mon.rst.

Follows https://github.com/ceph/ceph/pull/54432.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54387 from phlogistonjohn/jjm-cephadm-isolate-mounts-code

cephadm: isolate mounts code in methods

Reviewed-by: Adam King <adking@redhat.com>

rgw/sal: Bucket owner as rgw_user

stop maintaining a User pointer for the bucket owner, and use the
stored rgw_user owner directly

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54432 from zdover23/wip-doc-2023-11-09-rados-troubleshooting-mon-common-issues-3-of-x

doc/rados: edit t-mon "common issues" (3 of x)

crimson/os/seastore/object_data_handler: prevent from being overwritten if cloned

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

test/crimsons/seastore/object_data_handler: add overwrite test cases

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: store data as a delta using mutable extent when overwriting

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/rados: edit t-mon "common issues" (3 of x)

Edit the second part of the section "Most Common Monitor Issues" in
doc/rados/troubleshooting/troubleshooting-mon.rst.

Follows https://github.com/ceph/ceph/pull/54417.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54392 from guits/cv-revert-52429

ceph-volume: Revert "ceph-volume: fix raw list for lvm devices"

Merge pull request #54412 from zdover23/wip-doc-2023-11-08-rados-troubleshooting-mon-top-matter

doc/rados: edit t-shooting-mon.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw/sal: remove Bucket::get_acl_owner()

was only used under the line by rados and dbstores

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50599 from cbodley/wip-rgw-sal-forward

rgw: move forward_request_to_master() out of sal

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

ceph-volume: fix raw list for non-existent device

ceph-volume should not crash when given a device which doesn't exist.

Fixes: https://tracker.ceph.com/issues/63391
Signed-off-by: Matthew Booth <mbooth@redhat.com>
(cherry picked from commit e009b1a319c834a504947423ae1dd55387809235)

cls: fix unable to link with system-installed fmt

Signed-off-by: Jinkyu Yi <jincreator@jincreator.net>

Merge pull request #50212 from rishabh-d-dave/fs-swap-subcmd

cephfs: add command "ceph fs swap"

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume: Revert "ceph-volume: fix raw list for lvm devices"

This reverts commit e5e429617c1c27dcd631171f65d30571e32f7266.
This commit introduced a regression, see linked tracker for details.

Fixes: https://tracker.ceph.com/issues/63391
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #54417 from zdover23/wip-doc-2023-11-08-rados-troubleshooting-mon-common-issues-2-of-x

doc/rados: edit t-mon "common issues" (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #54397 from phlogistonjohn/jjm-buildpy-latin1-fix

cephadm: work around pip failure on some envs

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/rados: edit t-shooting-mon.rst

Improve the front matter of the section "Troubleshooting Monitors" in
doc/rados/troubleshooting/troubleshooting-mon.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54413 from zdover23/wip-doc-2023-11-08-rados-troubleshooting-mon-common-issues-1-of-x

doc/rados: edit t-mon "common issues" (1 of x)

doc/rados: edit t-mon "common issues" (2 of x)

Edit the second part of the section "Most Common Monitor Issues" in
doc/rados/troubleshooting/troubleshooting-mon.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados: edit t-mon "common issues" (1 of x)

Edit the first part of the section "Most Common Monitor Issues" in
doc/rados/troubleshooting/troublehsooting-mon.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54402 from athanatos/sjust/wip-fix-tmdriver

crimson/tools/store_nbd/tm_driver: use TransactionManager::remove

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/tools/store_nbd/tm_driver: use TransactionManager::remove

231c7ac removed TransactionManager::dec_ref, adjust store_nbd/tm_driver
to use TransactionManager::remove instead.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #54158 from michaelalang/master

[cephadm](BUG): honor --skip-firewalld in section Open ports explicitly required for the daemon

Reviewed-by: Adam King <adking@redhat.com>

cephadm: work around pip failure on some envs

Work around an encoding/locale issue when the dashboard tests are run
(ubuntu 20.04).
The build.py changes brought in a9d1c62ca86 were validated for package
builds, teuthology, and other CI jobs but a different error was masking
this failure in the dashboard ci job.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54178 from leonid-s-usov/balance-after-stop

mds/MDBalancer: ignore queued callbacks if MDS is not active

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #54149 from mihalicyn/ceph_mds_request_head_legacy_fallback_fix

MClientRequest: properly handle ceph_mds_request_head_legacy for ext_num_retry, ext_num_fwd, owner_uid, owner_gid

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53912 from joscollin/wip-B63099-fix-prev-snapshot-none

client: avoid getvxattr op for ceph.mirror* attrs

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53887 from lxbsz/wip-fix-check-access

client: two fix for checking mds auth access in client side

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53899 from rishabh-d-dave/before-fs-rename-fail-fs

 cephfs,mon: fs rename must require FS to be offline and refuse_client_session to be set

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw/sal: rename Bucket::remove_bucket() to remove()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: remove forward-to-master functions from Driver

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add free function forward_iam_request_to_master()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add free function rgw_forward_request_to_master()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: forward_request takes const req_info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: remove ZoneGroup::get_endpoint()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: move User::create_bucket() to Bucket::create()

the CreateBucket operation always has to deal with existing buckets, so
we have to load the Bucket handle first anyway

also moves the related placement and forward_request_to_master() logic
out of sal entirely

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54364 from rkachach/fix_issue_github_template

github: adding command for rook e2e jenkins job

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54391 from rhcs-dashboard/fix-63469-main

mgr/dashboard: fix rgw multi-site import form helper

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54101 from rhcs-dashboard/dashboard-v3-unit-test

mgr/dashboard: dashboard area chart unit test

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix rgw multi-site import form helper

Before : To obtain the token, generate it from your primary Ceph cluster. This token includes encoded information about the secondary cluster's endpoint, access key, and secret key.
Fix: To obtain the token, generate it from your primary Ceph cluster. This token includes encoded information about the primary cluster's endpoint, access key, and secret key.

Fixes: https://tracker.ceph.com/issues/63469
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #54372 from linuxbox2/wip-fix-deletemakers

rgwlc: fix delete marker regression in currentversion lc expiration

Merge pull request #54384 from cityofships/main

doc/cephadm: fix typo in set ssh key command

Reviewed-by: Zac Dover <zac.dover@proton.me>

cephadm: move code setting up mounts for monitoring daemon types

Move the code used to set up mounts for monitoring daemon types to
a method on the Monitoring class.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move code setting mounts for ceph daemon types

Move the code in an if-block for ceph daemon types within get_container_mounts
to a new method of the Ceph type. The logic is moved to a staticmethod
so that it can be reused within the CephExporter type. This can be fixed
up in the future, but currently the staticmethod handles both Ceph and
CephExporter. Both types gain a get_container_mounts for common usage.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: remove unused get_container_mounts CephExporter method

The get_container_mounts method on CephExporter is never called.
Remove it.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move where _get_container_mounts_for_type is called

The _get_container_mounts_for_type method is only useful for ceph based
daemon types. There's no need to call this method in a universal
context.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

doc/cephadm: fix typo in set ssh key command

Signed-off-by: Piotr Parczewski <piotr@stackhpc.com>

Merge pull request #54196 from ivancich/wip-log-bi-list-errors

rgw: log errors during "bi list"

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

Merge pull request #54257 from ivancich/wip-fix-restore-bucket-index

rgw: handle instances w/ leading hyphens correctly

Reviewed-by: Michael Kidd <linuxkidd@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53537 from cyx1231st/wip-crimson-osd-fix-complete-exit

crimson/osd: decouple cross-core pg submission out of the OrderedExclusivePhase

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #54197 from ivancich/wip-bucket-stats-add-gen

rgw: add layout generation info to "bucket stats"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54365 from cbodley/wip-63455

common: admin socket catches exception by reference

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

rgwlc: fix delete marker regression in currentversion lc expiration

Fixes a regression introduced in https://github.com/ceph/ceph/pull/50680,
which attempted to re-use an already created sal object handle as
an optimiztion.  This change suppressed creation of delete markers in
current version expiration when buckets are versioned.

Fixes: https://tracker.ceph.com/issues/63458
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #53568 into main

* refs/pull/53568/head:
common: resolve config proxy deadlock using refcounted pointers
common: add missing locks in config_proxy methods
common/ceph_mutex: note whether mutex debug methods are usable
qa: add reproducer for obs removal deadlock
qa: narrow search to debug_asok

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #54134 from phlogistonjohn/jjm-cephadm-podman-props-2

cephadm: move more container engine specific logic

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54264 from thotz/remove_realm_name_period_config

rgw/rgw_period : remove realm name from period config

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54342 from anthonyeleven/anthonyeleven/improvements

doc/radosgw: Improve dynamicresharding.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54147 from AliMasarweh/wip-alimasa-perf-counters

RGW: perf counters for persistent topic stats

reviewed-by: yuvalif, alimaredia, cbodley

common: admin socket catches exception by reference

probably not a real bug, but resolves a compiler warning:

src/common/admin_socket.cc: In member function ‘int RaiseHook::parse_signal(std::string&&, ceph::Formatter*, std::ostream&)’:
src/common/admin_socket.cc:869:21: warning: catching polymorphic type ‘class std::invalid_argument’ by value [-Wcatch-value=]
  869 |       } catch (std::invalid_argument) {
      |                     ^~~~~~~~~~~~~~~~

Fixes: https://tracker.ceph.com/issues/63455
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add free function rgw::get_zonegroup_endpoint()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use load_bucket()'s rgw_bucket overload

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_bucket() ctor for tenant+name

this avoids the need to construct a temporary rgw_bucket_key just to
construct a rgw_bucket without an instance id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: remove load_bucket() overload with tenant/name

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: rename loading get_bucket()s to load_bucket()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/services: make set_attrs const correct

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove legacy bucket placement and 'radosgw-admin pool' commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: pass obj_lock_enabled into RGWRados::create_bucket

there's a retry loop here that may end up overwriting RGWBucketInfo, so
pass obj_lock_enabled and use it to update info.flags on each iteration

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: read existing placement and swift_ver_location outside of sal

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWCreateBucket cleanup

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move rgw_op_get_bucket_policy_from_attr() out of sal

this was duplicated in rgw_op.cc and rgw_sal_rados.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: User::remove_bucket() no longer supports forward_to_master

callers of remove_bucket() are now responsible for calling
forward_request_to_master() themselves

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWBulkDelete calls forward_to_master before remove_bucket()

fix the path of the forwarded request the same way that RGWBulkUpload
does, and call forward_request_to_master() before remove_bucket()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: don't use other zone's shard count for forwarded CreateBucket requests

multisite no longer requires shard counts to match between zones. each
zone can configure its own default index shard count, so ignore the shard
counts that come from the metadata master zone

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused RGWObjectSimplePutCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused RGWBucketCreateLocalCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cephadm: move logic to build pull command to container engines

Move the logic needed to build a pull command to a function in the
container_engines.py file. This continues the effort to improve the
locality of items with regards to specific options for container
engines. I had thought to move the whole pull function but the logging
and retry code made me change my mind. We can always move it later if it
makes sense.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move option for setting unlimited pids into engine classes

Move the option for setting unlimited pids for the container engine
into the container engine classes. This continues the attempts to
improve the locality of items specific to certain classes to be
by making them part of the classes.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move some podman specific mount logic to podman class

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54285 from thmour/main

mgr/cephadm: discovery service (port 8765) fails on ipv6 only clusters

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54126 from phlogistonjohn/jjm-cephadm-remove-get_daemon_args

cephadm: remove get_daemon_args function

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #53928 from adk3798/rm-cleanup-on-failure

cephadm: remove --cleanup-on-failure flag

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #53737 from adk3798/rm-host-crushmap

mgr/cephadm: support for removing host entry from crush map during host removal

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52492 from phlogistonjohn/jjm-cephadm-jinja

Add templating, via jinja2, to cephadm

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #53842 from BBoozmen/oozmen_sts_max_duration

RGW/STS: honor configured limits when updating max session duration

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #53997 from Inode1/rgw-subuser-user-policy-check

rgw: add subuser to user policy condition check

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #54050 from shreyanshjain7174/fix_62646

RadosGW: RGW Swift info not providing max_deletes_per_request in bulk_delete

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54174 from cbodley/wip-63306

rgw: fetch_remote_obj() uses uncompressed size for encrypted objects

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #54213 from cbodley/wip-63333

rgw-check-policy: remove check for nonempty tenant

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #50680 from linuxbox2/wip-notify-lctrans

rgwlc: dispatch S3 notifications on transition and mpu abort

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #54307 from rkachach/fix_issue_63326

mgr/dashboard: fix secure_monitoring_stack check

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

github: adding command for rook e2e jenkins job

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #54352 from zdover23/wip-doc-2023-11-06-rados-troubleshooting-mon-common-issues-formatting

doc/rados: format Q&A list in tshooting-mon.rst

Merge pull request #54222 from nbalacha/deep_copy_logs

librbd: additional logs for deep-copy debugging

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>