rgw/logging: clean pending objects

* when source bucket is deleted
* when logging is disabled
* when logging conf changes
* when target bucket is deleted

also add "bucket logging info" admin command
returning logging conf for source bucket, and list
of source buckets for log bucket

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: retry attribuite set in case of race

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: log bucket must not have encryption

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: source and log bucket must be different

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: support source and destination buckets on different tenants

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #60958 from xxhdx1985126/wip-69120

crimson/osd/backfill_state: recheck whether needs to scan the primary at the end of Enqueuing::Enqueuing

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #60803 from xxhdx1985126/wip-69027

crimson/osd/replicate_backend: add standalone delete to the push queue if the object is deleted while it's being backfilled

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #60795 from taodd/fix-rbd-sparsify-on-ec-image-slow

osd: optimize extent comparison in PrimaryLogPG

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #61313 from VallariAg/wip-vallari-teuthology-docs

doc: improve tests-integration-testing-teuthology-workflow.rst

Merge pull request #61291 from cbodley/wip-69470

doc/rgw: document UserName requirements for account migration

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55259 from linuxbox2/wip-object-attributes

rgw: implement GetObjectAttributes

Merge pull request #60969 from idryomov/wip-69135

qa/suites/upgrade/reef-x: sync log-ignorelist with quincy-x

Reviewed-by: Laura Flores <lflores@ibm.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #60081 from anthonyeleven/anthonyeleven/tweak

src/exporter: improve usage message

rgw_attrs: fixes for 3 more iam/perms mappings

Help from Pritha and Casey appreciated

Plus, fix 3 explicit mapping tests in test_rgw_iam_policy.cc.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #61164 from chardan/wip-objectstore-ops-replayer-sign-warning

ops_replayer: fix signed/unsigned warning and add parameter range checking.

doc: improve tests-integration-testing-teuthology-workflow.rst

This commit adds:
1. workflow summary in the first section along with an image.
2. sub-section "Pushing to ceph-ci repository" to second section.
3. file doc/dev/developer_guide/testing_integration_tests/workflow.png

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

Merge pull request #61254 from kamoltat/wip-ksirivad-fix-stretch-mode-doc

doc/rados/operations/stretch-mode: Improve doc
Reviewed-by: zdover23
Reviewed-by: anthonyeleven

Merge pull request #61288 from adamemerson/wip-69303

rgw: Don't crash on exceptions from pool listing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: document UserName requirements for account migration

clarify the following error when migrating a user into an account:

> could not modify user: unable to modify user, UserName contains invalid characters

Fixes: https://tracker.ceph.com/issues/69470
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: Don't crash on exceptions from pool listing

Fixes: https://tracker.ceph.com/issues/69303
Signed-off-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #60330 from JonBailey1993/JonBailey1993/ceph_test_rados_io_sequence_inject_error

common/io_exerciser: Add support to ceph_test_rados_io_sequence injecting errors for testing how erasure coding deals with error scenarios

Reviewed-by: Ronen Friedman <rfriedma@ibm.com>

Merge pull request #52791 from clwluvw/location-constraint

rgw: check for location constraint on master zonegroup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #59960 from smanjara/wip-fix-missing-http-data

rgw/multisite:  include request body when CreateBucket op is forwarded to master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60881 from NitzanMordhai/wip-nitzan-ipv6-subnet-checks

common/pick_address: Add IPv6 support to is_addr_in_subnet

Merge pull request #60987 from drunkard/main

doc: fixes in doc building

Merge pull request #60883 from xxhdx1985126/wip-crimson-backfill-throttle

crimson/osd/pg_recovery: throttle backfills together with pglog based recoveries

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #61262 from VallariAg/fix-nvmeof-teuthology-basic-test

qa/workunits/nvmeof/basic_tests.sh: fix nvme list assert

Merge pull request #60623 from piyushagarwal1411/ceph-exporter

vstart: add ceph-exporter support

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #60891 from xxhdx1985126/wip-seastore-fadvise-backfill

crimson/os/seastore: add fadvise support to SeaStore and prevent recovery/backfill from polluting the cache of SeaStore

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

doc/rados/operations/stretch-mode: Improve doc

Added more content and rewrite some sections

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Fix signed/unsigned warning and add parameter range checking.

Signed-off-by: Jesse F. Williamson <1643380+chardan@users.noreply.github.com>

Merge pull request #60848 from cbodley/wip-rgw-deprecate-iam-tenant

docs/rgw: deprecate tenant-based IAM in favor of accounts

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61074 from chardan/wip-radowsgw-admin-jfw-restructure_file

rgw: migrate rgw_admin to new radosgw-admin directory

Merge pull request #60379 from cbodley/wip-librados-cancellation

librados/asio: forward asio cancellations to AioCompletion::cancel()

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rgw/multisite: the create_bucket forward request omits the
the request body, thus missing some data if specified inside
CreateBucketConfiguration xml on the non-master zone.
also, now that we perform cksum validation against empty payloads,
such a request would fail with -ERR_AMZ_CONTENT_SHA256_MISMATCH due
to a zero content-length but a non-empty payload hash.
this fix ensures that request body is forwarded during create_bucket

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #61127 from VallariAg/wip-nvmeof-delete-healthcheck

mon/NVMeofGwMap: add healthcheck warning NVMEOF_GATEWAY_DELETING

Merge pull request #61136 from cbodley/wip-69301

rgw: don't use merge_and_store_attrs() when recreating a bucket

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #61255 from Matan-B/wip-matanb-crimson-seastore-deafult-2

common/options/crimson.yaml.in: fallback to Bluestore by default

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #61251 from guits/ceph-volume-69432

ceph-volume: fix loop devices support

vstart.sh: add support for launching the ceph-exporter daemon

Signed-off-by: Piyush Agarwal <piyushagarwal14.pa@gmail.com>

Merge pull request #61252 from guits/hints-create_id

ceph-volume: add python hints to util.prepare.create_id()

qa/workunits/nvmeof/basic_tests.sh: fix connect-all assert

There seems to be change in 'nvme list' json output
which caused failures in asserts after 'nvme connect-all'
command.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

rgw:attrs: dump marker and max_parts only if sent

Suggested by Casey.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw:attrs cap max_parts at 1000

Suggested by Casey.  This is a relatively small value, but let's
be conservative.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: GetObjAttrs enforces MaxSize and returns correct IsTruncated

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: Object::list_parts() starts after marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: get_part_obj_state() fixes accounted_size when uncompressed

the part head objects don't have a RGW_ATTR_MANIFEST attribute, so
get_obj_state_impl() isn't able to set the correct
RGWObjState::accounted_size unless RGW_ATTR_COMPRESSION provides one

get_part_obj_state() builds a fake manifest in memory to represent the
part and updates state.size accordingly, but it hadn't corrected the
value of state.accounted_size

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: GetObjAttrs calls rgw_s3_prepare_decrypt()

RGWGetObj::execute() relies on the virtual get_decrypt_filter() which is
overridden by RGWGetObj_ObjStore_S3

but when called from RGWGetObjAttrs_ObjStore_S3, there is no override.
add one that just calls rgw_s3_prepare_decrypt()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw_cksum: return version_id in all cases

The version_id member in RGWGetObjAttrs shadows the one in
RGWGetObj.

Just assign it before conditionally returning the version-id header
in RGWGetObjAttrs_ObjStore_S3::send_response().

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_cksum: permit fallback to checksum-type from create-multipart, in upload-part

There appear to be workloads that provide a checksum algorithm in
create-multipart-upload, but do not provide (what must be) the
corresponding algorithm when uploading the parts.  (complete-multipart-upload
has no checksum argument, the value is implicit.)

This behavior is inconsistent with at least some SDKs, but it is
possibly accepted behavior in AWS S3, and is not logically necessary,
since the originally supplied checksum type is already known.

Therefore, change the behavior of upload-part to fall back to a
checksum type that was sent with the corresponding create-multipart-upload
request, if one is present, rather than failing with InvalidRequest.

Fixes: https://tracker.ceph.com/issues/68513
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 6b487a4c6dbadf3f470c8b12ddd5f2521c6920c6)

rgw: defensive check for part cksum in try_sum_part_cksums

If a create-multipart-upload requested additional checksum but a
part upload omits it, fail gracefully.

Possibly we can fixup the checksum uploads and remove this scenario,
but coding defensively would still be correct.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_cksum: don't display nil checksums in get-object-attributes (parts)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: GetObjectAttributes iam policy policy hooks

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_cksum: zero-initialize GetObjAttrs integral members

(code review)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: implement GetObjectAttributes

Implements the corresponding S3 operation, and
introduces a new Object::list_parts SAL interface to support it.

Includes Casey Bodley <cbodley@redhat.com>:
  * use uncompressed part size
  * local variable shadowed a member variable and broke handling of
      PartNumberMarker in request and response

Fixes: https://tracker.ceph.com/issues/64109
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #61213 from Matan-B/wip-matanb-crimson-shared-lru-dtor

crimson/common/shared_lru: invalidate Deleter's cache

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #60666 from ThomasLamprecht/debian-fix-librgw2-lua-dependencies

debian/control: fix overly broad lua dependency declaration for librgw2 package

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #60118 from kshtsk/wip-refactor-make-check

script/run-make: stop args duplication

Merge pull request #61241 from guits/ceph-volume-69430

ceph-volume: fix Zap.ensure_associated_raw()

ceph-volume: fix loop devices support

This commit updates the `is_device` function to correctly handle
loop devices.
The function now validates loop devices when they are explicitly
allowed, by checking their type (`loop`) in addition to `disk`
and `mpath`.

Changes include:
  - Extending the type check to include `loop` in the list of
    supported device types.
  - Enhancing the docstring for better documentation of the
    function's purpose and behavior.

These changes ensure that loop devices are properly recognized
and handled when configuring OSDs in ceph-volume.

Fixes: https://tracker.ceph.com/issues/69432
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

common/io_exerciser: Make chunksize so initial generated value is 4096 and random values are generated thereafter

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

common/io_exerciser: Make sure Sequence 10 removes objects after finishing running

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

src/common/json: Modified JSON structures so they take advantage of ceph_json.h fully.

Also moved and renamed JSONStructures files so they structures are more easily identifiable and usable by others if desired.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

src/common/io_exerciser: Remove unneccisairy override in data_generation::SeededRandomGenerator

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

src/common/io_exerciser: add missing override statements to JsonStructures.h

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

src/common/io_exerciser: Formatting improvements using clang format

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

common/io_exerciser: Add simple sequences for testing error injects

Add sequences to test IOs with simple error injects, along with some small fixes for previous error inject implementation.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

common/io_exerciser: Add injecterror commands to ceph_test_rados_io_sequence interactive mode

Add injecterror commands that can be used in interactive mode to inject read and write errors as well as clear them

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

osd: EC error inject interfaces

Error inject interfaces for EC reads and writes using
ceph tell osd.<n> interface

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

src/common/io_exerciser: Code readability improvements

Implement inheritence for ceph::io_exerciser::IoOp to allow better differentiation between the different Op types and allow more complex Operations to be implemented

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

Revert "doc/dev/crimson: update SeaStore as default backend"

This reverts commit 41327dcf0f733777f0d022d1bab7bf72b9807b3c.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados-exp: add Seastore/thrash

As Seastore/trash was removed from the non-exp suite, it is
moved here until fully supported.
Follow-up to: 5150dae471c

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados-exp: remove basic

"basic" directory is included in the non-exp suite and is fully supported

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados: disable thrash/seastore

Seastore supports thrash_simple only until https://tracker.ceph.com/issues/69405
is resolved.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

common/options/crimson.yaml.in: Bluestore by default

This partially reverts f4dee79c3309e5ac1e1a142b85f492851e6757e1
Until https://tracker.ceph.com/issues/69402 is resolved.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

ceph-volume: add python hints to util.prepare.create_id()

This commit introduces type annotations to the `create_id` function in `ceph_volume.util.prepare`.
The parameters and return value are now typed as follows:
  - `fsid` is a `str`.
  - `json_secrets` is a `str`.
  - `osd_id` is an optional `str` (`Optional[str]`).
  - The function returns a `str`.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

mon/NVMeofGwMap: add delay to NVMEOF_GATEWAY_DELETING warning

Instead of immediately triggering, have this healthcheck trigger
after some time has elasped. This delay can be configured by
mon_nvmeofgw_delete_grace.

Track the time when gateways go into DELETING state in a new
member var (of NVMeofGwMon) 'gws_deleting_time'.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

Merge pull request #61240 from neesingh-rh/wip-68974

doc: add snapshots under cephfs concepts

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw: migrate rgw_admin to new directory.

Some C++ fixes, macro fixes, etc.. Gathers other core
files, makes names more closely match executable.

Signed-off-by: Jesse F. Williamson <jfw@ibm.com>

rgw: don't use merge_and_store_attrs() when recreating a bucket

https://github.com/ceph/ceph/pull/56583 recently fixed
merge_and_store_attrs() to preserve existing attrs, but this broke the
swift api's ability to remove container metadata. RGWCreateBucket
handles this merging itself with prepare_add_del_attrs(), so we should
just assign createparams.attrs to the bucket and store it with
bucket->put_info()

make the same change for RGWPutMetadataBucket which swift uses to
add/remove existing metadata

Fixes: https://tracker.ceph.com/issues/69301
Signed-off-by: Casey Bodley <cbodley@redhat.com>

ceph-volume: fix Zap.ensure_associated_raw()

When an OSD creation fails, ceph-volume can zaps unrelated
existing raw based OSD as part of the 'rollback step'.

Fixes: https://tracker.ceph.com/issues/69430
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #61243 from anthonyeleven/etags-fix

doc/radosgw/s3: correct eTag op match tables

doc/radosgw/s3: correct eTag op match tables

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

src/common/options/mon.yaml.in: add mon_nvmeofgw_delete_grace

This config allows to configure the delay in triggering
NVMEOF_GATEWAY_DELETING healthcheck warning, which is
triggered when NVMeoF gateways are in DELETEING state
for too long (indicating a problem in namespace
load-balacing).
The default value for this config is 15 mins.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

doc: add snapshots in docs under Cephfs concepts

Fixes: https://tracker.ceph.com/issues/68974
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #60720 from batrick/i68913

qa: write out ESubtreeMap more frequently to find large events

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/osd/replicated_recovery_backend: prevent recovery/backfills from
polluting the cache of the underlying futurized store

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os: all "read/get" interfaces accept op_flags

To be used by exclusively by Seastore, See `seastore::cache_hint_t`

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: introduce cache_hint_t

Layers above Cache can use cache_hint_t to notify Cache whether to put
the extents to Cache::lru.

CEPH_OSD_OP_FLAG_FADVISE_DONTNEED and CEPH_OSD_OP_FLAG_FADVISE_NOCACHE
will be treated as "no need to put into Cache::lru"

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/replicated_backend: make sure the check on whether to send
ops to replica osds and the pg log append happens in the same
continuation

Since backfill relies on the pg log to discover new modifications, we
need to make sure backfill always discover the modification that's not
sent to replica osds.

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/replicated_recovery_backend: ignore the recovery of delete
objects

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/replicate_backend: add standalone delete to the push queue
if the object is deleted while it's being backfilled

Fixes: https://tracker.ceph.com/issues/69027
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg_recovery: throttle backfills together with pg-log based
recoveries

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/backfill_state: recheck whether needs to scan the primary at
the end of Enqueuing::Enqueuing

Fixes: https://tracker.ceph.com/issues/69120
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #59593 from xxhdx1985126/wip-67888

crimson/osd/backfill_state: treat `Cancelled` as a pause of the ongoing backfilling

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #60597 from xxhdx1985126/wip-68806

crimson/osd/replicated_recovery_backend: call on_global_recover() only when all replicas and the primary have been recovered

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #60041 from xxhdx1985126/wip-68286

crimson/osd/pg_shard_manager: discard outdated operations when the corresponding pgs are already removed

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #61216 from athanatos/sjust/wip-crimson-backfill-teuth

qa/suites/crimson-rados: enable short_pg_log

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #61227 from zdover23/wip-doc-2024-01-05-README

doc: README.md - improve "Tshooting" and "Tips & Tricks"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61148 from ronen-fr/wip-rf-abortReserv

osd/scrub: abort running scrub in replica-reservation if an operator-initiated scrub is requested

Reviewed-by: Samuel Just <sjust@redhat.com>