]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
15 months agorgw/pubsub: use rgw_owner for topic ownership
Casey Bodley [Tue, 12 Dec 2023 21:36:04 +0000 (16:36 -0500)]
rgw/pubsub: use rgw_owner for topic ownership

allow topics to be owned by accounts instead of users. radosgw-admin
topic list can now filter by --account-id

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 13d1c803f6a90ad1747f21faa7a6e89d7e8af8c1)

15 months agorgw: use rgw_owner in RGWBucketEntryPoint
Casey Bodley [Fri, 1 Dec 2023 21:13:02 +0000 (16:13 -0500)]
rgw: use rgw_owner in RGWBucketEntryPoint

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 469b7e83d723021555886dc7c46d178190f59829)

15 months agorgw/rados: add rgwrados::buckets namespace abstraction for cls_user
Casey Bodley [Fri, 1 Dec 2023 03:06:27 +0000 (22:06 -0500)]
rgw/rados: add rgwrados::buckets namespace abstraction for cls_user

move cls_user stuff out of RGWSI_User_RADOS into namespace
rgwrados::buckets that doesn't depend on rgw_user, so it can also be
used for account bucket owners

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 92f04d8637f78d916a3bcf955fe7cff56f053283)

15 months agorgw/rados: add account metadata handler
Casey Bodley [Fri, 1 Dec 2023 04:44:06 +0000 (23:44 -0500)]
rgw/rados: add account metadata handler

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit eb86cd410df93fa9edb959e9d2db9a56410dd0f3)

15 months agorgw/rados: implement account metadata operations
Casey Bodley [Mon, 27 Nov 2023 23:12:22 +0000 (18:12 -0500)]
rgw/rados: implement account metadata operations

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8c3fc16b3d9ef866c6b9cee558b30d78b357c8cc)

15 months agorgw/rados: generalize RGWUID for accounts
Casey Bodley [Thu, 30 Nov 2023 21:55:11 +0000 (16:55 -0500)]
rgw/rados: generalize RGWUID for accounts

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c72a51e08c5fb570a270efc3e1a7aff1e0383d16)

15 months agorgw/rados: add .rgw.meta:accounts pool
Casey Bodley [Mon, 27 Nov 2023 18:42:36 +0000 (13:42 -0500)]
rgw/rados: add .rgw.meta:accounts pool

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit e3418a9c4f2e2e00c3c066d8f8bb2db1d149745a)

15 months agorgw/quota: generalize quota to rgw_owners
Casey Bodley [Sun, 26 Nov 2023 19:47:50 +0000 (14:47 -0500)]
rgw/quota: generalize quota to rgw_owners

use rgw_owner instead of rgw_user as a key in the quota cache stats.
only fetch_stats_from_storage() and init_refresh() need to differentiate
between user and account ids

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit f1835cd78770ccd0d4ff84f68ecf2a9750182a65)

15 months agorgw/auth: Identity::is_owner/admin_of(rgw_owner)
Casey Bodley [Fri, 3 Nov 2023 19:56:58 +0000 (15:56 -0400)]
rgw/auth: Identity::is_owner/admin_of(rgw_owner)

is_owner_of() and is_admin_of() take rgw_owner instead of rgw_user so
that identities associated with an account share ownership of that
account's resources

LocalApplier is the only Identity type that supports accounts, based on
comparison with RGWUserInfo::account_id

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a1c675da7cf571457898d799206e911e23cdc711)

15 months agorgw/auth: account users also match ACL grants to their account id
Casey Bodley [Sat, 4 Nov 2023 19:09:46 +0000 (15:09 -0400)]
rgw/auth: account users also match ACL grants to their account id

ACL grants can now specify an account id for the CanonicalUser to
grant access to the entire account. this is implemented only for
LocalApplier

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1698784e2eb72e99e3eaa477ae8e53b35a6ab377)

15 months agorgw: add rgw_owner variant with json encoding
Casey Bodley [Fri, 3 Nov 2023 00:28:03 +0000 (20:28 -0400)]
rgw: add rgw_owner variant with json encoding

existing buckets and objects use `struct rgw_user` for the owner. with
the addition of accounts, we need to be able to represent ownership by
the account rather than its users

add a `rgw_owner` variant that preserves the existing json encoding of
`rgw_user` while adding a new representation for account ids

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 87a74f8603bdeccdea3fdfa7b4e0fd344fba8aae)

15 months agorgw: enable 'user modify --account-id'
Casey Bodley [Wed, 1 Nov 2023 22:28:34 +0000 (18:28 -0400)]
rgw: enable 'user modify --account-id'

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit e69ce01774be57259bef76fb3267c607513f28e7)

15 months agorgw: add /admin/account rest apis
Casey Bodley [Wed, 1 Nov 2023 22:27:42 +0000 (18:27 -0400)]
rgw: add /admin/account rest apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit cce372dc2ebd2b0d7bdb06b930fcc35ad27147d1)

15 months agoradosgw-admin: add account admin commands
Casey Bodley [Wed, 1 Nov 2023 22:26:53 +0000 (18:26 -0400)]
radosgw-admin: add account admin commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 07d5ed956275e408fc429cbef821ad01b085a2c3)

15 months agorgw: add account admin ops
Casey Bodley [Wed, 1 Nov 2023 22:16:45 +0000 (18:16 -0400)]
rgw: add account admin ops

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 6c834d6d38128dc03bb088f133f4bef6c0a711b9)

15 months agorgw/sal: add account interfaces to Driver
Casey Bodley [Wed, 1 Nov 2023 22:15:06 +0000 (18:15 -0400)]
rgw/sal: add account interfaces to Driver

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 91a58853a6265f19175586c4543a41109dc3a791)

15 months agorgw: add account_id to RGWUserInfo
Casey Bodley [Wed, 1 Nov 2023 19:48:33 +0000 (15:48 -0400)]
rgw: add account_id to RGWUserInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 3f1cd1b44a8ef60c6f8fd460df5559145a059930)

15 months agorgw: add struct RGWAccountInfo
Casey Bodley [Wed, 1 Nov 2023 19:47:59 +0000 (15:47 -0400)]
rgw: add struct RGWAccountInfo

initial design and prototype by Abhishek

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 59bbd65bac0ea41ec8cca844b9b65b9def026467)

15 months agorgw: define account ids and names
Casey Bodley [Wed, 1 Nov 2023 22:11:58 +0000 (18:11 -0400)]
rgw: define account ids and names

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit f678c780195c81d86533740456d843c08c7db1c0)

15 months agocommon: add gen_rand_numeric()
Casey Bodley [Mon, 27 Jun 2022 20:49:11 +0000 (16:49 -0400)]
common: add gen_rand_numeric()

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit cac60663f553cf6a985050c068d906778e66214e)

15 months agorgw/sal: pass in ACLOwner for object writes
Casey Bodley [Thu, 7 Dec 2023 03:38:27 +0000 (22:38 -0500)]
rgw/sal: pass in ACLOwner for object writes

`ACLOwner` contains both the user id and display name. the bucket index
needs both values for:
```
struct rgw_bucket_dir_entry_meta {
  ...
  std::string owner;
  std::string owner_display_name;
```
`RGWRados::Bucket::UpdateIndex::complete()` relied on the parsing of
`RGW_ATTR_ACL` to get those values. but object write operations already
had that information earlier in the call stack, so we might as well pass
them in directly

for other operations like the copy/rewrite/transition of existing objects,
we decode the owner from the source object's `RGW_ATTR_ACL`

the existing `owner` param was confusing, as it represented the bucket
owner for quota stats updates. this get renamed to `bucket_owner` inside
of `RGWRados`, and hidden from the outside. bucket stats are attributed to
`RGWBucketInfo::owner`, not the `ACLOwner`, so we use that instead of
`s->bucket_owner`

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1d29b7056983825e7095c518edc7b3d4dcdf0b05)

Conflicts:
src/test/rgw/test_d4n_filter.cc removed on main

15 months agorgw/acl: use ACLOwners for create_default()
Casey Bodley [Thu, 7 Dec 2023 23:00:37 +0000 (18:00 -0500)]
rgw/acl: use ACLOwners for create_default()

initialize RGWAccessControlPolicy with ACLOwners from the auth identity

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a2b41a27ebc2f14f3d2079fc4a2c2a682e871bcc)

15 months agorgw/auth: initialize s->owner in Strategy::apply()
Casey Bodley [Thu, 7 Dec 2023 21:56:27 +0000 (16:56 -0500)]
rgw/auth: initialize s->owner in Strategy::apply()

for a common location that applies to all rest handlers that implement
authorize()

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9989a86dd7a586d292cae0de32f7723b93147edc)

15 months agorgw/auth: Identity::get_aclowner() for resources it creates
Casey Bodley [Thu, 7 Dec 2023 21:16:16 +0000 (16:16 -0500)]
rgw/auth: Identity::get_aclowner() for resources it creates

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4b8cea086a44019486418d818687cc1502b1ed2d)

15 months agorgw/auth: generalize Identity::get_role_tenant() as get_tenant()
Casey Bodley [Thu, 7 Dec 2023 20:21:52 +0000 (15:21 -0500)]
rgw/auth: generalize Identity::get_role_tenant() as get_tenant()

all identities can return a tenant. rgw ops should consult the auth
identity for this instead of a rgw_user or rgw::sal::User

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d82ff164505cd7a7abe6937879052ad03f1da939)

15 months agorgw/iam: zero-pad milliseconds for role and oidc-provider CreateDate
Casey Bodley [Sun, 11 Feb 2024 17:24:54 +0000 (12:24 -0500)]
rgw/iam: zero-pad milliseconds for role and oidc-provider CreateDate

make sure that tv_usec=41000 gets rendered as '.041Z' instead of '.41Z'

Fixes: https://tracker.ceph.com/issues/64381
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit efb2a91138e76c5e163916093d5e37f776479bcf)

15 months agoMerge pull request #56670 from cbodley/wip-65292-squid
Casey Bodley [Fri, 12 Apr 2024 18:01:41 +0000 (19:01 +0100)]
Merge pull request #56670 from cbodley/wip-65292-squid

squid: rgw/pubsub: CreateTopic validates topic name

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>
15 months agoMerge pull request #56069 from cbodley/wip-64818-squid
Casey Bodley [Fri, 12 Apr 2024 17:54:18 +0000 (18:54 +0100)]
Merge pull request #56069 from cbodley/wip-64818-squid

squid: rgw: replicate v2 topic/notification metadata

Reviewed-by: Adam Emerson <aemerson@redhat.com>
15 months agoMerge pull request #56856 from guits/update-cv-func-tests-squid
Guillaume Abrioux [Fri, 12 Apr 2024 14:19:04 +0000 (16:19 +0200)]
Merge pull request #56856 from guits/update-cv-func-tests-squid

ceph-volume: update functional testing

15 months agoMerge pull request #56850 from zdover23/wip-doc-2024-04-12-backport-56831-to-squid
Anthony D'Atri [Fri, 12 Apr 2024 14:04:10 +0000 (10:04 -0400)]
Merge pull request #56850 from zdover23/wip-doc-2024-04-12-backport-56831-to-squid

squid: doc/cephfs: refine client-auth (3 of 3)

15 months agoMerge pull request #56750 from ronen-fr/wip-rf-unreserved-squid
Ronen Friedman [Fri, 12 Apr 2024 12:17:43 +0000 (15:17 +0300)]
Merge pull request #56750 from ronen-fr/wip-rf-unreserved-squid

squid:osd/scrub: disable scrub reservation queuing

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
15 months agoceph-volume: update functional testing 56856/head
Guillaume Abrioux [Thu, 11 Apr 2024 07:43:47 +0000 (09:43 +0200)]
ceph-volume: update functional testing

various changes for fixing ceph-volume functional testing.
cleaning up deploy.yml (drop py2 references)

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit 90bfaaa2f0ab7e28c0f78189ccb83635e944e758)

15 months agosquid:osd/scrub: disable scrub reservation queuing 56750/head
Ronen Friedman [Sun, 7 Apr 2024 18:18:50 +0000 (13:18 -0500)]
squid:osd/scrub: disable scrub reservation queuing

as a temporary measure for Squid RC0. There are known problems in the
existing code, with patch PRs being tested. This change disables the
Reserver functionality, until such time as the patches are merged.

Note the following trackers:
- https://tracker.ceph.com/issues/65044
- https://tracker.ceph.com/issues/65072

Once these are solved, this commit should be reverted.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
15 months agoMerge pull request #56766 from ronen-fr/wip-65374-squid
Ronen Friedman [Fri, 12 Apr 2024 09:09:33 +0000 (12:09 +0300)]
Merge pull request #56766 from ronen-fr/wip-65374-squid

squid: tests/scripts: use 'tell pg deep-scrub pgid' instead of 'tell pgid deep-scrub'

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
15 months agodoc/cephfs: refine client-auth (3 of 3) 56850/head
Zac Dover [Thu, 11 Apr 2024 11:18:53 +0000 (21:18 +1000)]
doc/cephfs: refine client-auth (3 of 3)

Refine the final third of doc/cephfs/client-auth.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 44d1eeed703e968b374ec4d12fff3a2466a4c276)

15 months agoMerge pull request #56841 from zdover23/wip-doc-2024-04-12-backport-56784-to-squid
Anthony D'Atri [Fri, 12 Apr 2024 03:12:13 +0000 (23:12 -0400)]
Merge pull request #56841 from zdover23/wip-doc-2024-04-12-backport-56784-to-squid

doc/cephfs: refine client-auth (2 of 3)

15 months agodoc/cephfs: refine client-auth (2 of 3) 56841/head
Zac Dover [Tue, 9 Apr 2024 12:04:46 +0000 (22:04 +1000)]
doc/cephfs: refine client-auth (2 of 3)

Refine the second third of doc/cephfs/client-auth.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 083f56352b4fff662dff3c78b6a0c455605d02ff)

15 months agotest/rgw/notification: fix threading issue in test 56069/head
Yuval Lifshitz [Tue, 9 Apr 2024 10:00:08 +0000 (10:00 +0000)]
test/rgw/notification: fix threading issue in test

test_ps_s3_persistent_topic_stats may delete boto client threads
before they are done.
also, no need to run the http server twice

Fixes: https://tracker.ceph.com/issues/63909
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit 51e365927361db19b2ae2b270bab7f30fe3e51ee)

15 months agotest/rgw/notification: use polling instead of sleep
Yuval Lifshitz [Thu, 4 Apr 2024 17:14:31 +0000 (17:14 +0000)]
test/rgw/notification: use polling instead of sleep

when checking persistent stats queue length

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit 5581bc7d14f4c5814a7d43597aa77cacd6a2d125)

15 months agotest/rgw/notification: cleanup leftover topics
Yuval Lifshitz [Thu, 4 Apr 2024 16:25:33 +0000 (16:25 +0000)]
test/rgw/notification: cleanup leftover topics

this may happen when a test fails, and does not cleanup topics
it created. other tests that verify the number of topics may fail
because of that.
all tests that verify number of topics, should delete all topics at the
start of the test.

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit 9ae0de17e1a8c3a0527b04981eeb7e632da985ac)

15 months agotest/rgw/notification: use simpler multithreaded http server
Yuval Lifshitz [Mon, 25 Mar 2024 11:11:31 +0000 (11:11 +0000)]
test/rgw/notification: use simpler multithreaded http server

Fixes: https://tracker.ceph.com/issues/63909
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit 673adcbdbd01e64c8b76c3176e062571fb8710ac)

15 months agotest/rgw/notifications: do not delay http responses to notifications
Yuval Lifshitz [Tue, 20 Feb 2024 10:12:26 +0000 (10:12 +0000)]
test/rgw/notifications: do not delay http responses to notifications

we also increase the wating time after rerunning the http server
to allow fo the persistent notifications retries to finish

Fixes: https://tracker.ceph.com/issues/63909
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit ca6688696b9ac9f838ffe21951331b6a1974dae8)

15 months agorgw/notification: Load bucket attrs before calling publish_reserve.
kchheda3 [Thu, 4 Apr 2024 18:30:40 +0000 (14:30 -0400)]
rgw/notification: Load bucket attrs before calling publish_reserve.

As part of PR# 55657, publish_reserve would reload bucket to ensure bucket_attrs are loaded. However for lc events, where the bucket attrs were already loaded, the reloading was causing crash but there was no obvious root cause, so to avoid the crashes, remove reloading of bucket in publish_reserve and put the onus on callers to load the bucket before calling publish_reserve.

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>
(cherry picked from commit fa5d370663804db9752c99e79659aae031a7bf85)

15 months agotest/rgw/noitifications: run basic/http/migration tests with 2 RGWs
Yuval Lifshitz [Mon, 18 Mar 2024 16:44:39 +0000 (16:44 +0000)]
test/rgw/noitifications: run basic/http/migration tests with 2 RGWs

this is useful to:
* cover the cls_lock code with persistent notifications
* cover v1 to v2 migration collisions between RGWs

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit c0e540e6f11aa32b347ea6e7e9f03cad5d1913e8)

15 months agorgw/notifications: run v2 migration tests in teuthology
Yuval Lifshitz [Thu, 14 Mar 2024 12:21:19 +0000 (12:21 +0000)]
rgw/notifications: run v2 migration tests in teuthology

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit 67d43ae2538a611373159346855dab8701e176fb)

15 months agoRGW/Rados: Migrate topics to data path v2
Ali Masarwa [Tue, 9 Jan 2024 21:51:54 +0000 (23:51 +0200)]
RGW/Rados: Migrate topics to data path v2

also add migration tests

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>
(cherry picked from commit 1a7d1454801e3d7b888aa734c4e3b609febf417f)

15 months agorgw/notification: Support generating multisite sync delete events.
kchheda3 [Tue, 27 Feb 2024 20:11:48 +0000 (15:11 -0500)]
rgw/notification: Support generating multisite sync delete events.

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>
(cherry picked from commit 37069ac96b53d2d4927b9d6f743790847861ec8e)

15 months agorgw/notification: Make the Replication events to be aws compatible.
kchheda3 [Tue, 27 Feb 2024 19:24:43 +0000 (14:24 -0500)]
rgw/notification: Make the Replication events to be aws compatible.

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>
(cherry picked from commit 3ea110bb89a7f666de25f8a128fa8bbb231e304f)

15 months agorgw/notification: Make the Lifecycle events AWS compatible
kchheda3 [Tue, 27 Feb 2024 19:06:47 +0000 (14:06 -0500)]
rgw/notification: Make the Lifecycle events AWS compatible

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>
(cherry picked from commit ea6611327c9aa678c524e0d71e954bd735d60789)

15 months agoPendingReleaseNotes: announce the notification_v2 feature and its migration
Casey Bodley [Wed, 13 Mar 2024 22:47:35 +0000 (18:47 -0400)]
PendingReleaseNotes: announce the notification_v2 feature and its migration

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4f0ff0f80ba0ef6273c31345e93244ccc37cfa69)

15 months agodoc/radosgw: document the notification_v2 zone feature
Casey Bodley [Wed, 13 Mar 2024 22:35:40 +0000 (18:35 -0400)]
doc/radosgw: document the notification_v2 zone feature

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 30d97250f945e0afdfe9ca52a97a42922091d4dd)

15 months agodoc/radosgw: generalize description of Zone Features
Casey Bodley [Wed, 13 Mar 2024 22:20:55 +0000 (18:20 -0400)]
doc/radosgw: generalize description of Zone Features

so they make sense outside of multisite configurations

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 0a97eb89e1d9c99e96fd215e9dc31febc2e44de5)

15 months agodoc/radosgw: move the Zone Features section to its own page
Casey Bodley [Wed, 13 Mar 2024 22:12:42 +0000 (18:12 -0400)]
doc/radosgw: move the Zone Features section to its own page

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d5ac275694bce35d37d7226b2f63934e3c277cbd)

15 months agoqa/rgw: notification suite creates a realm
Casey Bodley [Thu, 7 Mar 2024 23:07:38 +0000 (18:07 -0500)]
qa/rgw: notification suite creates a realm

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 7ef94cdb1c97f537561a7c8bb226d9d22ff709a8)

15 months agoqa/rgw: teach rgw.py to create an optional realm
Casey Bodley [Wed, 6 Mar 2024 19:04:46 +0000 (14:04 -0500)]
qa/rgw: teach rgw.py to create an optional realm

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4ac0b7df3c15140bc9d904a1e12fac482c7130be)

15 months agorgw/notifications: delete persistent queue only if topic is deleted
Yuval Lifshitz [Tue, 13 Feb 2024 16:36:51 +0000 (16:36 +0000)]
rgw/notifications: delete persistent queue only if topic is deleted

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 666e79f1fb78fe8128791e9e23159571f76cfe70)

15 months agorgw/notifications: handle migration state between v1 and v2
Yuval Lifshitz [Sat, 10 Feb 2024 16:38:30 +0000 (16:38 +0000)]
rgw/notifications: handle migration state between v1 and v2

test instructions:
https://gist.github.com/yuvalif/21449e301732b719cd1ed97c3eeeabb2

* during migration all topic and notification operations must fail with HTTP error code 503
* read operations should return the values of the v1 topics and notifications
* sending notifications should continue based on v1 values

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit bcd79d23df63e72839c652cbc20ebb67a024ba85)

15 months agorgw: move rgw_rest_pubsub.h out of driver/rados/
Casey Bodley [Tue, 13 Feb 2024 15:57:08 +0000 (10:57 -0500)]
rgw: move rgw_rest_pubsub.h out of driver/rados/

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 5b2ae726a95c9922a5c5ba984d6e170dd115128b)

15 months agorgw/pubsub: topic apis forward post body
Casey Bodley [Mon, 12 Feb 2024 21:44:00 +0000 (16:44 -0500)]
rgw/pubsub: topic apis forward post body

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9d46c21bc6e522b1d85056e1e545125743aace65)

15 months agorgw/topic: metadata handler adds/removes persistent queues
Casey Bodley [Thu, 18 Jan 2024 20:34:16 +0000 (15:34 -0500)]
rgw/topic: metadata handler adds/removes persistent queues

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 633a5bef314b5fbc3087271d684f22f9beafb21a)

15 months agorgw/topic: remove unused RGWSI_Topic_RADOS
Casey Bodley [Wed, 10 Jan 2024 22:40:11 +0000 (17:40 -0500)]
rgw/topic: remove unused RGWSI_Topic_RADOS

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit be9cc0c7bc163f4e578bd5459866a2c1e5c8a2dd)

15 months agorgw/topic: add rgwrados::topic interface for topic metadata
Casey Bodley [Wed, 10 Jan 2024 20:33:25 +0000 (15:33 -0500)]
rgw/topic: add rgwrados::topic interface for topic metadata

add a new interface for topic metadata that doesn't depend on metadata
backends. this low-level interface is used by both RadosStore and the
topic metadata handler

remove Driver::delete_bucket_topic_mapping() from sal because the omap
object is deleted internally by rgwrados::topic::remove()

remove the RGWRados::topics_pool_ctx member

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 6d1d036afb2d1624674fef43f2e70ef3b3ae2859)

15 months agorgw/topic: metadata key format in rgw_pubsub.h
Casey Bodley [Wed, 10 Jan 2024 22:07:27 +0000 (17:07 -0500)]
rgw/topic: metadata key format in rgw_pubsub.h

the format of topic metadata keys is agnostic to the backend, so the
parsing/formatting functions should be in rgw_pubsub.h

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1caf954dab136e7d5cff97f76ce362c1bfb8d3f4)

15 months agorgw: paginate ListTopics
Casey Bodley [Tue, 9 Jan 2024 23:55:40 +0000 (18:55 -0500)]
rgw: paginate ListTopics

rename read_topics()/write_topics() to 'v1' and only call them from
internal v1 call paths

public get_topics() now calls read_topics_v1() for the v1 case, and does
the paginated listing with driver->meta_list_keys_next() for v2

RGWPSListTopicsOp now uses the NextToken request/response params with
the paginated get_topics(), limiting responses to 100 entries like AWS

'radosgw-admin topic list' also paginates the listing according to
--max-entries to avoid reading everything into memory at once

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit db6c73a0cdcf60a920c91b6d4506df36d98b7308)

15 months agorgw/mdlog: add complete_entry() for most common usage
Casey Bodley [Fri, 5 Jan 2024 15:26:00 +0000 (10:26 -0500)]
rgw/mdlog: add complete_entry() for most common usage

metadata sync ignores entries other than MDLOG_STATUS_COMPLETE, so we
don't need to write separate prepare/complete entries. metadata
mutations can just call complete_entry() on success

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 7827b2527f6b656331ae337ebc196c17b2716e48)

15 months agorgw/metadata: add RGWMetadataLister abstraction
Casey Bodley [Sun, 3 Dec 2023 19:10:30 +0000 (14:10 -0500)]
rgw/metadata: add RGWMetadataLister abstraction

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 6000732706890fa5d276d22972d1d83be32bb5c8)

15 months agorgw: RGWPubSub requires SiteConfig
Casey Bodley [Sun, 4 Feb 2024 20:17:47 +0000 (15:17 -0500)]
rgw: RGWPubSub requires SiteConfig

RGWPubSub constructor takes SiteConfig instead of zonegroup map

replace do_all_zonegroups_support_notification_v2() with a generic
function rgw::all_zonegroups_support() that handles non-realm
configurations too

remove unused sal::ZoneGroup::supports_feature()

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1fccdad21d5bcdebbdd7167b7ed18391a1e8d1e2)

15 months agorgw/rados: RGWTopicMetadataHandler doesn't depend on RGWPubSub
Casey Bodley [Sun, 4 Feb 2024 19:04:11 +0000 (14:04 -0500)]
rgw/rados: RGWTopicMetadataHandler doesn't depend on RGWPubSub

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 0b8b62bed0e79dd54d1baac240695b2a398d407f)

15 months agorgw/rados: store SiteConfig with RGWServices
Casey Bodley [Sun, 4 Feb 2024 18:43:10 +0000 (13:43 -0500)]
rgw/rados: store SiteConfig with RGWServices

make the SiteConfig available to all of RGWRados via svc.site instead
of storing it in sal::RadosStore

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 0057bb7d0f02974f3a8cb1d34e73ead8d92c3062)

15 months agoradosgw-admin: make SiteConfig available to commands
Casey Bodley [Sun, 4 Feb 2024 16:37:57 +0000 (11:37 -0500)]
radosgw-admin: make SiteConfig available to commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 0f333f1cc5a265015d4d436e1275ffadc2d0d8d4)

15 months agorgw/multisite-notification: allow enabling v1/v2 in tests
Yuval Lifshitz [Thu, 28 Dec 2023 12:05:09 +0000 (12:05 +0000)]
rgw/multisite-notification: allow enabling v1/v2 in tests

v1 could be enabled only in local tests. teuthology tests would run with v2

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit d2db9df224659cf711aee80479b5d462ff3ca716)

15 months agotest/rgw/notifications: support running tests in multisite environment
Yuval Lifshitz [Thu, 21 Dec 2023 18:18:45 +0000 (18:18 +0000)]
test/rgw/notifications: support running tests in multisite environment

both locally and in teuthology

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 7502215d70a423e642e5147a0d444f627a7979f2)

15 months agorgw/multisite-notification: Add integration test for bucket notifications in multisit...
kchheda3 [Mon, 11 Dec 2023 20:47:30 +0000 (15:47 -0500)]
rgw/multisite-notification: Add integration test for bucket notifications in multisite config.

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>
(cherry picked from commit 1ed8df24ae0f279a6e7d294231b5e6c1e45fb663)

15 months agorgw/multisite-notification: Add omap object to store the mapping between bucket and...
kchheda3 [Thu, 21 Dec 2023 22:23:05 +0000 (17:23 -0500)]
rgw/multisite-notification: Add omap object to store the mapping between bucket and topics.

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>
(cherry picked from commit df668e9f65cec50227f671eb2e624c0ee02f2c37)

15 months agorgw/multisite-notification: Add support to replicate bucket notifications in multisit...
kchheda3 [Tue, 5 Dec 2023 17:26:48 +0000 (12:26 -0500)]
rgw/multisite-notification: Add support to replicate bucket notifications in multisite config.

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>
(cherry picked from commit ab74e2cfe4e3738b85d38a132f835d1dca113258)

15 months agorgw/multisite-notification: Add support to replicate topic creation in multisite...
kchheda3 [Mon, 27 Nov 2023 20:51:02 +0000 (15:51 -0500)]
rgw/multisite-notification: Add support to replicate topic creation  in multisite config

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>
(cherry picked from commit c4a147d079cb9f268340e5f79ec5350a94296658)

15 months agotest/rgw/notifications: split tests between basic, kafka and amqp
Yuval Lifshitz [Thu, 22 Feb 2024 09:58:00 +0000 (09:58 +0000)]
test/rgw/notifications: split tests between basic, kafka and amqp

see comment: https://tracker.ceph.com/issues/64184#note-3

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 6f28930602a2dfdcd402537c1a274df1836109ce)

15 months agoMerge pull request #56778 from epuertat/wip-65368-squid
Ernesto Puerta [Wed, 10 Apr 2024 08:55:38 +0000 (10:55 +0200)]
Merge pull request #56778 from epuertat/wip-65368-squid

squid: install-deps: enable copr ceph/grpc

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
15 months agoMerge pull request #56779 from zdover23/wip-doc-2024-04-09-backport-56754-to-squid
Anthony D'Atri [Tue, 9 Apr 2024 21:26:56 +0000 (17:26 -0400)]
Merge pull request #56779 from zdover23/wip-doc-2024-04-09-backport-56754-to-squid

squid: doc/cephfs: refine client-auth (1 of 3)

15 months agoMerge pull request #56797 from zdover23/wip-doc-2024-04-10-backport-56796-to-squid
Anthony D'Atri [Tue, 9 Apr 2024 20:59:01 +0000 (16:59 -0400)]
Merge pull request #56797 from zdover23/wip-doc-2024-04-10-backport-56796-to-squid

squid: doc/mgr: remove Zabbix 1 information

15 months agodoc/mgr: remove Zabbix 1 information 56797/head
Zac Dover [Tue, 9 Apr 2024 20:04:28 +0000 (06:04 +1000)]
doc/mgr: remove Zabbix 1 information

Remove information about the installation of the Zabbix module and link
to a discussion of the reasoning behind Ceph's refusal to support
Zabbix.

John Jasen developed a procedure explaining how to install "Zabbix 2".
This commit removes outdated procedures and explains why those
procedures were removed. Immediately following this explanation, the
text includes an explanation of how to install "Zabbix 2".

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 29ac627956c5e5cdd4ed5f1316c20bc9e1fe0ff7)

15 months agoMerge pull request #56783 from epuertat/wip-65390-squid
Ernesto Puerta [Tue, 9 Apr 2024 14:49:24 +0000 (16:49 +0200)]
Merge pull request #56783 from epuertat/wip-65390-squid

squid: mgr/dashboard: update NVMeoF gRPC: "listener add" changes

15 months agomgr/dashboard: update NVMeoF gRPC 56783/head
Ernesto Puerta [Tue, 2 Apr 2024 17:36:21 +0000 (19:36 +0200)]
mgr/dashboard: update NVMeoF gRPC

To include latest changes to the listener add call.

Fixes: https://tracker.ceph.com/issues/65268
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
(cherry picked from commit 7357e1f326c03baabfb870fb424756537dc8830a)

15 months agodoc/cephfs: refine client-auth (1 of 3) 56779/head
Zac Dover [Mon, 8 Apr 2024 12:07:38 +0000 (22:07 +1000)]
doc/cephfs: refine client-auth (1 of 3)

Refine the first third of doc/cephfs/client-auth.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 82c5bac5df553f1e13a2df3a107b2692efa6fc27)

15 months agoinstall-deps: enable copr ceph/grpc 56778/head
Nizamudeen A [Wed, 27 Mar 2024 16:29:55 +0000 (21:59 +0530)]
install-deps: enable copr ceph/grpc

In dashboard, to generate nvmeof apis in el8 this is needed so that it
can download the python3-grpcio packages.
https://copr.fedorainfracloud.org/coprs/ceph/grpc/

Fixes: https://tracker.ceph.com/issues/65184
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 257dc69d324445f6c71c33d9ac248f0bf0e961ec)

15 months agoMerge pull request #56668 from batrick/wip-65289-squid
Leonid S. Usov [Tue, 9 Apr 2024 10:10:55 +0000 (13:10 +0300)]
Merge pull request #56668 from batrick/wip-65289-squid

squid: mds/quiesce: prevent an overflow of the wait duration

15 months agoMerge pull request #56695 from ceph/wip-lusov-qdb-clear-squid
Leonid S. Usov [Mon, 8 Apr 2024 22:31:48 +0000 (01:31 +0300)]
Merge pull request #56695 from ceph/wip-lusov-qdb-clear-squid

squid: mds/quiesce-db: always clear the db if a membership is lost

15 months agoMerge pull request #56761 from adk3798/squid-cephadm-pin-pyfakefs-version
Adam King [Mon, 8 Apr 2024 18:24:42 +0000 (14:24 -0400)]
Merge pull request #56761 from adk3798/squid-cephadm-pin-pyfakefs-version

squid: cephadm: pin pyfakefs version for tox tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>
15 months agotests/scripts: use 'tell pg deep-scrub pgid' instead of 'tell pgid 56766/head
Ronen Friedman [Sun, 7 Apr 2024 07:48:35 +0000 (02:48 -0500)]
tests/scripts: use 'tell pg deep-scrub pgid' instead of 'tell pgid
deep-scrub'

as older OSD versions do not support the former.

Fixes: https://tracker.ceph.com/issues/64972
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
(cherry picked from commit dda89e77ae6d0a1c7f351c74192417bf60fde3f4)

15 months agocephadm: pin pyfakefs version for tox tests 56761/head
Adam King [Mon, 8 Apr 2024 14:48:18 +0000 (10:48 -0400)]
cephadm: pin pyfakefs version for tox tests

5.4.0 seems to have caused a number of permission issues
in the test. In order to unblock make check, let's pin it
to 5.3.5 for now.

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit ce71ce10b7551a649c9a64be992c0eddfa7e7243)

15 months agoMerge pull request #56739 from zdover23/wip-doc-2024-05-07-backport-56738-to-squid
Anthony D'Atri [Sun, 7 Apr 2024 12:28:13 +0000 (08:28 -0400)]
Merge pull request #56739 from zdover23/wip-doc-2024-05-07-backport-56738-to-squid

squid: doc/dev: refine "Concepts" 4 of 3

15 months agodoc/dev: refine "Concepts" 4 of 3 56739/head
Zac Dover [Sat, 6 Apr 2024 04:43:16 +0000 (14:43 +1000)]
doc/dev: refine "Concepts" 4 of 3

s/PG's PG logs/PG's logs/

re https://github.com/ceph/ceph/pull/56727#discussion_r1553541922

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 221d734760b6f680e4d17cd2f0bfcd39f914c8d6)

15 months agoMerge pull request #56731 from zdover23/wip-doc-2024-04-05-backport-56727-to-squid
Zac Dover [Sat, 6 Apr 2024 04:40:04 +0000 (14:40 +1000)]
Merge pull request #56731 from zdover23/wip-doc-2024-04-05-backport-56727-to-squid

squid: doc/dev: refine "Concepts" 3 of 3

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
15 months agoMerge pull request #56706 from afreen23/wip-65208-squid
Nizamudeen A [Fri, 5 Apr 2024 17:27:45 +0000 (22:57 +0530)]
Merge pull request #56706 from afreen23/wip-65208-squid

squid: mgr/dashboard:Update encryption and tags in bucket form

Reviewed-by: Nizamudeen A <nia@redhat.com>
15 months agoMerge pull request #56708 from afreen23/wip-65303-squid
Nizamudeen A [Fri, 5 Apr 2024 17:26:36 +0000 (22:56 +0530)]
Merge pull request #56708 from afreen23/wip-65303-squid

squid: mgr/dashboard:Use advanced fieldset for rbd image

Reviewed-by: Nizamudeen A <nia@redhat.com>
15 months agoMerge pull request #56564 from batrick/wip-65214-squid
Yuri Weinstein [Fri, 5 Apr 2024 14:59:24 +0000 (07:59 -0700)]
Merge pull request #56564 from batrick/wip-65214-squid

squid: mds: do not dispatch aborted internal requests

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
15 months agoMerge pull request #56423 from batrick/wip-65093-squid
Yuri Weinstein [Fri, 5 Apr 2024 14:58:25 +0000 (07:58 -0700)]
Merge pull request #56423 from batrick/wip-65093-squid

squid: qa/suites/fs/workload: enable snap_schedule early

Reviewed-by: Venky Shankar <vshankar@redhat.com>
15 months agoMerge pull request #56402 from batrick/wip-65063-squid
Yuri Weinstein [Fri, 5 Apr 2024 14:57:41 +0000 (07:57 -0700)]
Merge pull request #56402 from batrick/wip-65063-squid

squid: qa/cephfs: ignorelist clog of MDS_UP_LESS_THAN_MAX

Reviewed-by: Venky Shankar <vshankar@redhat.com>
15 months agodoc/dev: refine "Concepts" 3 of 3 56731/head
Zac Dover [Fri, 5 Apr 2024 06:43:47 +0000 (16:43 +1000)]
doc/dev: refine "Concepts" 3 of 3

Refine the third third of the entries in the section "Concepts" in
doc/dev/peering.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 512e2dc9d52fdd14dfdd87ccd141e535d3098fb7)

15 months agoMerge pull request #56724 from zdover23/wip-doc-2024-04-05-backport-56646-to-squid
Anthony D'Atri [Fri, 5 Apr 2024 12:33:58 +0000 (08:33 -0400)]
Merge pull request #56724 from zdover23/wip-doc-2024-04-05-backport-56646-to-squid

squid: doc/dev: refine "Concepts" 2 of 3