rgw: verify_permission logs acl grants

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: adapt verify_user_permission() for account users

Signed-off-by: Casey Bodley <cbodley@redhat.com>

vstart/rgw: add default config for sts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/role: support Description for Create/Get/UpdateRole

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: add s3:Get/PutBucketOwnershipControls

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: remove load_account_role_by_name()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/role: role APIs support account users

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/role: separate dump_iam_role() for iam api

create a new dump_iam_role() for iam api responses that dumps the subset
of role information presented by the apis

RGWRoleInfo::dump() and decode_json() are used by metadata sync to
transfer role metadata between zones, so must contain all information
about the role

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: add pagination to ListRoles

rename sal::Driver::get_roles() to list_roles() and add pagination
support for the RGWListRoles op and 'radosgw-admin role list'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: enable Role apis against account users

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: role apis override init_processing/verify_permission

replace get_params() with init_processing() override which runs before
verify_permission(). use this to validate request parameters and load
the existing role if necessary. simplify verify_permission() by
forwarding to RGWRESTOp::verify_permission() which calls check_caps()

simplify inheritence hierarchy by taking cap perm and iam action as
constructor arguments

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: split RGWRestRole member variables

move member variables into the subclasses they're needed for

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: RGWUntagRole uses lower/upper bounds for iteration

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: RGWRestRole::parse_tags() as free function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: AccessKey apis call forward_iam_request_to_master()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: User apis call forward_iam_request_to_master()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move forward_iam_request_to_master() to rgw_rest_iam.*

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rest: enable iam UserPolicy apis against account users

when the authenticated user belongs to an account:
* operate only on that account's users
* match UserName to user's display_name instead of user_id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rest: simplify RGWRestUserPolicy hierarchy

base class constructor takes `uint64_t action` instead of overriding
the virtual `get_op()` on each subclass

constructor takes `uint32_t perm` instead of deriving separate base
classes RGWUserPolicyRead/Write for check_caps() permission

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rest: iam user policy api cleanup

make get_params() virtual and protected. base class always validates
UserName

add common init_processing() function that calls get_params() and loads
the user by UserName. this step happens before verify_permission()

set s->err.message in several error paths

add the xmlns="https://iam.amazonaws.com/doc/2010-05-08/" part to the
responses

return ERR_LIMIT_EXCEEDED instead of ERR_INVALID_REQUEST when
RGWPutUserPolicy exceeds the policy limit

where RGW_ATTR_USER_POLICY is missing, treat it the same way we treat an
empty map of policies. this avoids separate error paths

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: Identity matches account user principals

when a user belongs to an account, they match Principal ARNs by account
id instead of tenant name, and by user name instead user id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: Identity matches paths in user principals

when RGWUserInfo::path is present, use it when matching user principals

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: Identity::is_identity() takes one Principal

take a single Principal instead flat_set<Principal>, and iterate over
calls to is_identity() instead

why?
* it simplifies the logic of each is_identity() function because they
  can use early returns to avoid visiting all of the cases
* Statement::eval_principal() no longer has to allocate a flat_set
  with a single element when the Identity is a role
* rgw::auth::Identity no longer depends on rgw::iam's choice of
  container type

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: rename rgw::auth::Principal::Tenant to Account

just changes the name to match its use in AWS, without changing any
behavior in rgw policy parsing/evaluation

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: add interfaces for account roles

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: add rgwrados::roles namespace abstraction for cls_user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: init_quota() loads owner quota unconditionally

now that owners can be accounts, don't default to s->user when s->owner
matches s->bucket_owner

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: quota commands can set account quota

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: add IAM AccessKey apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/user: add 'create_date' to RGWAccessKey

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/user: expose functions to generate access/secret keys

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: add initial IAM User apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rest: wrap iam/sns/sts Error responses with ErrorResponse

all iam/sns/sts requests wrap the s3 <Error> xml response in another
<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">

without this, boto3 fails to fully parse error responses, leading to
generic Unknown ClientError exceptions of the form:
    botocore.exceptions.ClientError: An error occurred (Unknown) when calling the PutUserPolicy operation: Unknown

with the ErrorResponse part, boto3 throws more specific exceptions that
include the error Code and Message:
    botocore.errorfactory.NoSuchEntityException: An error occurred (NoSuchEntity) when calling the PutUserPolicy operation: No such UserName in the account

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: user metadata links to account index

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: add interfaces for account users

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: add rgwrados::users namespace abstraction for cls_user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: use switch for is_non_s3_op()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/op: separate IAM ops in enum RGWOpType

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: use enum for action values

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: add iam user metadata (path, create_date, tags)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/cls: add ceph_test_cls_user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/user: add interfaces to index user account resources

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: move WITH_RADOSGW checks up to test/CMakeLists.txt

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/multisite: RGWBucketMetadataHandler updates linkage on owner change

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: update bucket ownership when adding user to account

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: Bucket::chown() updates owner on bucket instance

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWIdentityType::TYPE_ROOT for account root user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: create accounts for random s3test users

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: create_policy_from_headers() can yield

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: s3 CanonicalUser grants support account ids

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: add load_aclowner_by_email()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: move list_buckets() to Driver

move User::list_buckets() to Driver and take rgw_owner to serve bucket
listings for account owners

also unifies the user/account stats interfaces around rgw_owner in
Driver

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use rgw_owner in RGWBucketInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: use rgw_owner for ACLOwner

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: use rgw_owner for topic ownership

allow topics to be owned by accounts instead of users. radosgw-admin
topic list can now filter by --account-id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use rgw_owner in RGWBucketEntryPoint

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: add rgwrados::buckets namespace abstraction for cls_user

move cls_user stuff out of RGWSI_User_RADOS into namespace
rgwrados::buckets that doesn't depend on rgw_user, so it can also be
used for account bucket owners

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: add account metadata handler

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: implement account metadata operations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: generalize RGWUID for accounts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: add .rgw.meta:accounts pool

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/quota: generalize quota to rgw_owners

use rgw_owner instead of rgw_user as a key in the quota cache stats.
only fetch_stats_from_storage() and init_refresh() need to differentiate
between user and account ids

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: Identity::is_owner/admin_of(rgw_owner)

is_owner_of() and is_admin_of() take rgw_owner instead of rgw_user so
that identities associated with an account share ownership of that
account's resources

LocalApplier is the only Identity type that supports accounts, based on
comparison with RGWUserInfo::account_id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: account users also match ACL grants to their account id

ACL grants can now specify an account id for the CanonicalUser to
grant access to the entire account. this is implemented only for
LocalApplier

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_owner variant with json encoding

existing buckets and objects use `struct rgw_user` for the owner. with
the addition of accounts, we need to be able to represent ownership by
the account rather than its users

add a `rgw_owner` variant that preserves the existing json encoding of
`rgw_user` while adding a new representation for account ids

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: enable 'user modify --account-id'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add /admin/account rest apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add account admin commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add account admin ops

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: add account interfaces to Driver

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add account_id to RGWUserInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add struct RGWAccountInfo

initial design and prototype by Abhishek

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: define account ids and names

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: add gen_rand_numeric()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: pass in ACLOwner for object writes

`ACLOwner` contains both the user id and display name. the bucket index
needs both values for:
```
struct rgw_bucket_dir_entry_meta {
  ...
  std::string owner;
  std::string owner_display_name;
```
`RGWRados::Bucket::UpdateIndex::complete()` relied on the parsing of
`RGW_ATTR_ACL` to get those values. but object write operations already
had that information earlier in the call stack, so we might as well pass
them in directly

for other operations like the copy/rewrite/transition of existing objects,
we decode the owner from the source object's `RGW_ATTR_ACL`

the existing `owner` param was confusing, as it represented the bucket
owner for quota stats updates. this get renamed to `bucket_owner` inside
of `RGWRados`, and hidden from the outside. bucket stats are attributed to
`RGWBucketInfo::owner`, not the `ACLOwner`, so we use that instead of
`s->bucket_owner`

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: use ACLOwners for create_default()

initialize RGWAccessControlPolicy with ACLOwners from the auth identity

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: initialize s->owner in Strategy::apply()

for a common location that applies to all rest handlers that implement
authorize()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: Identity::get_aclowner() for resources it creates

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: generalize Identity::get_role_tenant() as get_tenant()

all identities can return a tenant. rgw ops should consult the auth
identity for this instead of a rgw_user or rgw::sal::User

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: zero-pad milliseconds for role and oidc-provider CreateDate

make sure that tv_usec=41000 gets rendered as '.041Z' instead of '.41Z'

Fixes: https://tracker.ceph.com/issues/64381
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/cmake: rgw_common links openssl unconditionally

we had been relying on WITH_RADOSGW_AMQP_ENDPOINT for this dependency

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/d4n: object attrs shouldn't store/update user metadata

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56765 from cbodley/wip-65373

rgw/multisite: x-rgw-replicated-at uses dump_time_header()

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #55661 from shreyanshjain7174/tracker_63791

RGW: a subuser with no permission can still list buckets and create buckets

Reviewed-by: Jiffin Tony Thottan <jthottan@redhat.com>

Merge pull request #56789 from cbodley/wip-65334

qa/rgw: barbican uses branch stable/2023.1

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #56728 from pritha-srivastava/wip-rgw-d4n-compilation-errors

rgw/d4n: fix compilation issue when WITH_RADOSGW_D4N=OFF.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #56704 from cbodley/wip-rgw-default-http-error

rgw: don't map to EIO in rgw_http_error_to_errno()

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge PR #56629 into main

* refs/pull/56629/head:
mds: log at a lower level when stopping

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #55663 from yuvalif/wip-yuval-63909

test/rgw/notifications: do not delay http responses to notifications

Reviewed-by: Casey Bodley <cbodley@redhat.com>, Ali Marsawa <ali.masarwa@ibm.com>, Krunal Chheda <kchheda3@bloomberg.net>

Merge pull request #56774 from idryomov/wip-doc-releases-reef-latest

doc: reorder "releases" entries for reef to fix the diagram

Merge pull request #56796 from zdover23/wip-doc-2024-04-10-mgr-zabbix

doc/mgr: remove Zabbix 1 information

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/mgr: remove Zabbix 1 information

Remove information about the installation of the Zabbix module and link
to a discussion of the reasoning behind Ceph's refusal to support
Zabbix.

John Jasen developed a procedure explaining how to install "Zabbix 2".
This commit removes outdated procedures and explains why those
procedures were removed. Immediately following this explanation, the
text includes an explanation of how to install "Zabbix 2".

Signed-off-by: Zac Dover <zac.dover@proton.me>

test/rgw/notification: fix threading issue in test

test_ps_s3_persistent_topic_stats may delete boto client threads
before they are done.
also, no need to run the http server twice

Fixes: https://tracker.ceph.com/issues/63909
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

qa/rgw: barbican uses branch stable/2023.1

the stable/xena branch no longer exists. it was moved to
unmaintained/xena. use the same stable/2023.1 branch as keystone

Fixes: https://tracker.ceph.com/issues/65334
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56577 from nbalacha/wip-nbalacha-code-cleanup

rbd-mirror: rename on_stop_journal_replay() and some operator<< arguments

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #56754 from zdover23/wip-doc-2024-04-08-cephfs-client-auth-1-of-3

doc/cephfs: refine client-auth (1 of 3)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #56459 from ronen-fr/wip-rf-nohist3

osd/scrub: implement reservation state as a ReplicaActive flag

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #56641 from rhcs-dashboard/wip-65268-main

mgr/dashboard: update NVMeoF gRPC: "listener add" changes

Reviewed-by: gbregman <NOT@FOUND>

test/rgw/notification: use polling instead of sleep

when checking persistent stats queue length

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: cleanup leftover topics

this may happen when a test fails, and does not cleanup topics
it created. other tests that verify the number of topics may fail
because of that.
all tests that verify number of topics, should delete all topics at the
start of the test.

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>