rgw/sal: add interfaces for account roles

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: add rgwrados::roles namespace abstraction for cls_user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: init_quota() loads owner quota unconditionally

now that owners can be accounts, don't default to s->user when s->owner
matches s->bucket_owner

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: quota commands can set account quota

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: add IAM AccessKey apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/user: add 'create_date' to RGWAccessKey

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/user: expose functions to generate access/secret keys

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: add initial IAM User apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rest: wrap iam/sns/sts Error responses with ErrorResponse

all iam/sns/sts requests wrap the s3 <Error> xml response in another
<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">

without this, boto3 fails to fully parse error responses, leading to
generic Unknown ClientError exceptions of the form:
    botocore.exceptions.ClientError: An error occurred (Unknown) when calling the PutUserPolicy operation: Unknown

with the ErrorResponse part, boto3 throws more specific exceptions that
include the error Code and Message:
    botocore.errorfactory.NoSuchEntityException: An error occurred (NoSuchEntity) when calling the PutUserPolicy operation: No such UserName in the account

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: user metadata links to account index

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: add interfaces for account users

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: add rgwrados::users namespace abstraction for cls_user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: use switch for is_non_s3_op()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/op: separate IAM ops in enum RGWOpType

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: use enum for action values

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: add iam user metadata (path, create_date, tags)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/cls: add ceph_test_cls_user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/user: add interfaces to index user account resources

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: move WITH_RADOSGW checks up to test/CMakeLists.txt

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/multisite: RGWBucketMetadataHandler updates linkage on owner change

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: update bucket ownership when adding user to account

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: Bucket::chown() updates owner on bucket instance

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWIdentityType::TYPE_ROOT for account root user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: create accounts for random s3test users

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: create_policy_from_headers() can yield

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: s3 CanonicalUser grants support account ids

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: add load_aclowner_by_email()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: move list_buckets() to Driver

move User::list_buckets() to Driver and take rgw_owner to serve bucket
listings for account owners

also unifies the user/account stats interfaces around rgw_owner in
Driver

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use rgw_owner in RGWBucketInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: use rgw_owner for ACLOwner

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: use rgw_owner for topic ownership

allow topics to be owned by accounts instead of users. radosgw-admin
topic list can now filter by --account-id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use rgw_owner in RGWBucketEntryPoint

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: add rgwrados::buckets namespace abstraction for cls_user

move cls_user stuff out of RGWSI_User_RADOS into namespace
rgwrados::buckets that doesn't depend on rgw_user, so it can also be
used for account bucket owners

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: add account metadata handler

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: implement account metadata operations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: generalize RGWUID for accounts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: add .rgw.meta:accounts pool

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/quota: generalize quota to rgw_owners

use rgw_owner instead of rgw_user as a key in the quota cache stats.
only fetch_stats_from_storage() and init_refresh() need to differentiate
between user and account ids

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: Identity::is_owner/admin_of(rgw_owner)

is_owner_of() and is_admin_of() take rgw_owner instead of rgw_user so
that identities associated with an account share ownership of that
account's resources

LocalApplier is the only Identity type that supports accounts, based on
comparison with RGWUserInfo::account_id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: account users also match ACL grants to their account id

ACL grants can now specify an account id for the CanonicalUser to
grant access to the entire account. this is implemented only for
LocalApplier

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_owner variant with json encoding

existing buckets and objects use `struct rgw_user` for the owner. with
the addition of accounts, we need to be able to represent ownership by
the account rather than its users

add a `rgw_owner` variant that preserves the existing json encoding of
`rgw_user` while adding a new representation for account ids

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: enable 'user modify --account-id'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add /admin/account rest apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add account admin commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add account admin ops

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: add account interfaces to Driver

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add account_id to RGWUserInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add struct RGWAccountInfo

initial design and prototype by Abhishek

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: define account ids and names

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: add gen_rand_numeric()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: pass in ACLOwner for object writes

`ACLOwner` contains both the user id and display name. the bucket index
needs both values for:
```
struct rgw_bucket_dir_entry_meta {
  ...
  std::string owner;
  std::string owner_display_name;
```
`RGWRados::Bucket::UpdateIndex::complete()` relied on the parsing of
`RGW_ATTR_ACL` to get those values. but object write operations already
had that information earlier in the call stack, so we might as well pass
them in directly

for other operations like the copy/rewrite/transition of existing objects,
we decode the owner from the source object's `RGW_ATTR_ACL`

the existing `owner` param was confusing, as it represented the bucket
owner for quota stats updates. this get renamed to `bucket_owner` inside
of `RGWRados`, and hidden from the outside. bucket stats are attributed to
`RGWBucketInfo::owner`, not the `ACLOwner`, so we use that instead of
`s->bucket_owner`

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: use ACLOwners for create_default()

initialize RGWAccessControlPolicy with ACLOwners from the auth identity

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: initialize s->owner in Strategy::apply()

for a common location that applies to all rest handlers that implement
authorize()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: Identity::get_aclowner() for resources it creates

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: generalize Identity::get_role_tenant() as get_tenant()

all identities can return a tenant. rgw ops should consult the auth
identity for this instead of a rgw_user or rgw::sal::User

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: zero-pad milliseconds for role and oidc-provider CreateDate

make sure that tv_usec=41000 gets rendered as '.041Z' instead of '.41Z'

Fixes: https://tracker.ceph.com/issues/64381
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/cmake: rgw_common links openssl unconditionally

we had been relying on WITH_RADOSGW_AMQP_ENDPOINT for this dependency

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/d4n: object attrs shouldn't store/update user metadata

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56765 from cbodley/wip-65373

rgw/multisite: x-rgw-replicated-at uses dump_time_header()

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #55661 from shreyanshjain7174/tracker_63791

RGW: a subuser with no permission can still list buckets and create buckets

Reviewed-by: Jiffin Tony Thottan <jthottan@redhat.com>

Merge pull request #56789 from cbodley/wip-65334

qa/rgw: barbican uses branch stable/2023.1

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #56728 from pritha-srivastava/wip-rgw-d4n-compilation-errors

rgw/d4n: fix compilation issue when WITH_RADOSGW_D4N=OFF.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #56704 from cbodley/wip-rgw-default-http-error

rgw: don't map to EIO in rgw_http_error_to_errno()

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge PR #56629 into main

* refs/pull/56629/head:
mds: log at a lower level when stopping

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #55663 from yuvalif/wip-yuval-63909

test/rgw/notifications: do not delay http responses to notifications

Reviewed-by: Casey Bodley <cbodley@redhat.com>, Ali Marsawa <ali.masarwa@ibm.com>, Krunal Chheda <kchheda3@bloomberg.net>

Merge pull request #56774 from idryomov/wip-doc-releases-reef-latest

doc: reorder "releases" entries for reef to fix the diagram

Merge pull request #56796 from zdover23/wip-doc-2024-04-10-mgr-zabbix

doc/mgr: remove Zabbix 1 information

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/mgr: remove Zabbix 1 information

Remove information about the installation of the Zabbix module and link
to a discussion of the reasoning behind Ceph's refusal to support
Zabbix.

John Jasen developed a procedure explaining how to install "Zabbix 2".
This commit removes outdated procedures and explains why those
procedures were removed. Immediately following this explanation, the
text includes an explanation of how to install "Zabbix 2".

Signed-off-by: Zac Dover <zac.dover@proton.me>

test/rgw/notification: fix threading issue in test

test_ps_s3_persistent_topic_stats may delete boto client threads
before they are done.
also, no need to run the http server twice

Fixes: https://tracker.ceph.com/issues/63909
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

qa/rgw: barbican uses branch stable/2023.1

the stable/xena branch no longer exists. it was moved to
unmaintained/xena. use the same stable/2023.1 branch as keystone

Fixes: https://tracker.ceph.com/issues/65334
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56577 from nbalacha/wip-nbalacha-code-cleanup

rbd-mirror: rename on_stop_journal_replay() and some operator<< arguments

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #56754 from zdover23/wip-doc-2024-04-08-cephfs-client-auth-1-of-3

doc/cephfs: refine client-auth (1 of 3)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #56459 from ronen-fr/wip-rf-nohist3

osd/scrub: implement reservation state as a ReplicaActive flag

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #56641 from rhcs-dashboard/wip-65268-main

mgr/dashboard: update NVMeoF gRPC: "listener add" changes

Reviewed-by: gbregman <NOT@FOUND>

test/rgw/notification: use polling instead of sleep

when checking persistent stats queue length

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: cleanup leftover topics

this may happen when a test fails, and does not cleanup topics
it created. other tests that verify the number of topics may fail
because of that.
all tests that verify number of topics, should delete all topics at the
start of the test.

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: use simpler multithreaded http server

Fixes: https://tracker.ceph.com/issues/63909
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #56383 from xxhdx1985126/wip-seastore-btree-cleanup

crimson/os/seastore/btree: clean up `FixedKVLeafNode::get_logical_child()`

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

doc: reorder "releases" entries for reef to fix the diagram

The Gantt diagram currently shows "reef (latest 18.2.0)" instead of
18.2.2.  This is because ReleasesGantt expects releases array to be
sorted in reverse order:

    for code_name, info in releases.items():
        last_release = info['releases'][0]
        first_release = info['releases'][-1]

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd_mirror: rename method

Rename on_stop_journal_replay to on_stop_replay as the
function doesn't do anything journal specific.

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge pull request #56764 from ceph/wip-yuriw-archive-pacific-main

doc: moved pacific release to Archived Releases

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Josh Durgin <jdurgin@ibm.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56469 from asm0deuz/bz2270211_ganesha_nlm

mgr/cephadm: Allows enabling NFS Ganesha NLM

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #56637 from adk3798/node-proxy-service-expected-daemon-count

mgr/cephadm: fix node-proxy service size

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #56674 from adk3798/wrong-grafana-uid

mgr/cephadm: pass daemon's current image when reconfiguring

Reviewed-by: Laura Flores <lflores@ibm.com>

mgr/dashboard: update NVMeoF gRPC

To include latest changes to the listener add call.

Fixes: https://tracker.ceph.com/issues/65268
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #56705 from adk3798/no-osd-require-osd-release

mgr/cephadm: handle setting required osd release with no OSDs during upgrade

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #56713 from adk3798/client-keyring-ceph-conf

mgr/cephadm: make client-keyring deploying ceph.conf optional

Reviewed-by: John Mulligan <jmulligan@redhat.com>

doc: moved pacific release to Archived Releases

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

test/rgw/notifications: do not delay http responses to notifications

we also increase the wating time after rerunning the http server
to allow fo the persistent notifications retries to finish

Fixes: https://tracker.ceph.com/issues/63909
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #56760 from adk3798/cephadm-pin-pyfakefs-version

cephadm: pin pyfakefs version for tox tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56712 from kchheda3/wip-notif-reload-bkt

rgw/notify: fix crashes in lc due to reload of bucket.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw/multisite: x-rgw-replicated-at uses dump_time_header()

dump_time() encodes this timestamp as json/xml. use dump_time_header()
instead

Fixes: https://tracker.ceph.com/issues/65373
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56619 from ljflores/wip-tracker-64864

qa/suites: add "mon down" log variations to ignorelist

cephadm: pin pyfakefs version for tox tests

5.4.0 seems to have caused a number of permission issues
in the test. In order to unblock make check, let's pin it
to 5.3.5 for now.

Signed-off-by: Adam King <adking@redhat.com>

doc/cephfs: refine client-auth (1 of 3)

Refine the first third of doc/cephfs/client-auth.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56530 from rhcs-dashboard/enable-grpc-copr

install-deps: enable copr ceph/grpc

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #54955 from leonid-s-usov/always_on_config

MgrMonitor: consider the build config when defining the always-on modules

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #54335 from batrick/i63432

qa: test test_kill_mdstable for all mount types

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/btree: clean up `FixedKVLeafNode::get_logical_child()`

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #56627 from zhscn/wip-rewrite-snap-refcount

crimson/os/seastore: alloc mapping with refcount when rewriting logical extents

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>