Merge pull request #56561 from phlogistonjohn/jjm-issue65122-maint-cmd

cephadm: fix host-maintenance command always exiting with a failure

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #56716 from adk3798/test_cephadm_images

qa/cephadm: update images for test_cephadm workunit

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56481 from adk3798/test-cephadm-idmap-conf

cephadm: add idmap.conf to nfs sample file

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56791 from adk3798/nvmeof-mon-setting

mgr/cephadm: make enable_monitor_client configurable for nvmeof

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56613 from NitzanMordhai/wip-nitzan-osd-down-ignore-cephadm-suites

suites/rados/cephadm: adding OSD_DOWN to the log-ignorelist

Reviewed-by: Adam King <adking@redhat.com>

qa/rgw/s3tests: remove 'client.0' from bucket prefix

new sns test cases are using this for topic names, but the '.' is not
allowed there:

> api_params = {'Name': 'test-client.0-n3bdgre5el2jk8v-606'}
> botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the CreateTopic operation: Name must be made up of only uppercase and lowercase ASCII letters, numbers, underscores, and hyphens

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/notify: populate event userIdentity with account ids

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw/pubsub: test persistent notifications with account user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/notify: support cross-tenant and cross-account notifications

a bucket's notification configuration may refer to topics from several
different tenants or accounts. when publishing to a given topic, look in
the correct namespace for each topic instead of defaulting to the
requesting user's tenant namespace

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: forward requests as s->owner instead of s->user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/notify: publish functions use rgw_pubsub_dest::persistent_queue

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: warn about topics under account migration

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: use rgw_pubsub_dest::persistent_queue for queue oid

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: RGWPubSub::remove_topic() removes persistent queue

move the persistent queue removal into remove_topic() where we have
access to the topic metadata. avoid trying to remove the queue if it
isn't enabled

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: rgw_pubsub_dest stores persistent queue oid

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: add/remove_persistent_topic() takes topic queue, not name

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw/pubsub: topic policy doesn't deny access to owner

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: CreateTopic consults existing topic policy for overwrite

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: fix DeleteTopic permissions

non-account users now consult identity policies with
verify_user_permission() when the topic doesn't exist

account users now consult topic policy when it does exist

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: CreateTopic consults identity policies when topic doesn't exist

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: CreateTopic requires notification_v2 for accounts

the account's topic index is only updated by writes/deletes to v2 topic
metadata

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: avoid allocating hash set of strings for attr search

this unordered_set was not static, so we reinitialized it on every call

replace with a constexpr array of string_views so we can search through
sequential memory that's laid out at compile time

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: add ERR_AUTHORIZATION -> AuthorizationError

sns docs specify AuthorizationError as the 403 error code rather than
s3's AccessDenied:

    https://docs.aws.amazon.com/sns/latest/api/API_CreateTopic.html#API_CreateTopic_Errors

boto3 sns clients can catch this as AuthorizationErrorException

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: return 404 NotFound instead of NoSuchKey

repurpose the ERR_NOT_FOUND define which was otherwise unused to
customize the error response for sns apis, which return the NotFound
error code instead of NoSuchKey from s3:

    https://docs.aws.amazon.com/sns/latest/api/API_GetTopicAttributes.html#API_GetTopicAttributes_Errors

this allows boto3 sns clients to catch the NotFoundException as expected

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: notifications can refer to topics in other accounts/tenants

accounts can use topic policy to grant sns:Publish permissions to other
accounts. the PutBucketNotification op should expect TopicArns from
other accounts. the account name from each TopicArn should be used as
the 'tenant' argument for RGWPubSub's constructor so we look for the
topic in the right namespace

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: customize permissions for account users

for account users, CreateTopic and ListTopics permissions come from
identity policy alone, ignoring the ownership/policy of existing topics

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: when present, use account id instead of tenant

RGWPubSub provides topic namespace isolation for tenants by adding
prefixes to rados object names and topic metadata keys. accounts use
this the same way

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: add index for account topics

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: verify_topic_permission handles cross-account access

refactor verify_topic_owner_or_policy() to share the same interface
as similar functions like verify_user/bucket/object_permission()
from rgw_common.cc

in addition to the topic resource policy, this now also consults iam
identity policies like user, group, or role policy

for account users, this now implements cross-account policy evaluation.
this only comes into play for sns:Publish permissions though, because
the topics themselves are scoped to the account

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: rgw_common.h exposes evaluate_iam_policies()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: do init/validation in init_processing()

verify_permission() should do permission checks and nothing else!

admin/system users ignore errors from verify_permission() and go on to
call execute() regardless. that means that execute() can't rely on any
initialization that happened during verify_permission(), at risk of
crashing on admin/system requests. it also means that any permission
checks in execute() won't get overridden for admin/system users,
breaking their superuser access

by moving all parameter validation and initialization into
init_processing(), we can prepare all the state that verify_permission()
will need to do it's thing

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: replace log messages with error response

parameter validation errors should be returned to the client instead of
written to the rgw log

also raises the log level for lots of error messages. very few of them
should require admin attention

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: use existing s->bucket for notification ops

s->bucket is already initialized during rgw_build_bucket_policies(),
called from RGWHandler::do_init_permissions()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: make v2-specific calls private

Signed-off-by: Casey Bodley <cbodley@redhat.com>

PendingReleaseNotes: announce the rgw user account feature

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: reject user tenant that looks like an account id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: log each policy that returns Allow or Deny

makes it much easier to debug authorization issues when you can see
exactly which policies led to success/failure

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/admin: 'user modify' won't change existing account id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/admin: user list accepts --account-id or -name

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: link account root to account user index

account root users were not linked to the account's user index because
they're not visible to iam apis like ListUsers

but now that 'account rm' is prevented from deleting the account while
users are still present, we want account root users to prevent deletion
too

add root users back to the account user index, but filter them out of
the iam user apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: expose Identity::get_account()

now that all identities store an optional account, expose that to the
rest of rgw with get_account(). this cleans up lots of code that
otherwise has to deal with the rgw_owner variant

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: validate account user names

iam apis have specific requirements for the UserName field. enforce
these requirements for 'user create' and 'user modify' admin ops for
account users

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: add account_id and role_id to ops log

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: object ops use new verify_bucket_permission() overload

several object operations like PutObject, DeleteObject, etc were handling
policy evaluation manually instead of using the helper functions like
verify_user/bucket/object_permission(), so were missing the cross-policy
evaluation rules for account users

these now call the new 'custom arn' overload of verify_bucket_permission()
for equivalent functionality

the eval_identity_or_session_policies() function is no longer exposed by
rgw_common.h to prevent other ops from adding new logic that doesn't
handle cross-account access

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: add verify_bucket_permission() overload for custom arn

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: WebIdentityApplier doesn't create shadow users for account roles

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: pass user policies into identities

loading user policies in rgw_build_bucket_policies() doesn't work for
PostObj requests because we haven't authenticated yet at that point

instead, auth engines load/parse policies when they load the user info.
policies are passed into the auth identities and applied to req_state
via modify_request_state() similar to how RoleApplier handles role
policy

this also moves the load_iam_identity_policies() into rgw_auth.cc for
use by transform_old_authinfo()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: auth engines pass optional account info into identities

the auth identities need the RGWAccountInfo instead of just the account
id so they can fill in the correct ACLOwner::display_name

this also adds account ownership support to WebIdentityApplier for
AssumeRoleWithWebIdentity

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: load attrs with RadosUser

when auth looks up a user by key, that should also initialize the user's
attrs so we don't have to load them separately

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: make user email matching case-insensitive

handle user emails the same way we do account account emails. store
RGWUserInfo::user_email exactly as the user specified it, but convert
the object name to lower-case for case-insensitive matching

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: deny 'account rm' if not empty

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: configure sts for all suites that run s3tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: Policy takes optional tenant name

the iam policy parser takes a tenant string to reject Resource ARNs that
specify resources in other tenants, and prevent wildcards from applying
to other tenants

this is problematic for account users, because cross-account access requires
an identity policy that covers another account's resource. it's the
cross-policy evaluation rules that prevent that from granting access to
things it shouldn't. so for account users, pass a null tenant string to
allow all resource arns

for resource policies, this restriction is unnecessary in the first
place, because the resource policy can only match itself as the resource

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: replace uses of verify_bucket_owner_or_policy()

all of the s3 actions that we call verify_bucket_owner_or_policy() for
are already covered by rgw::IAM::op_to_perm(), which maps actions to
acl permissions like RGW_PERM_READ, RGW_PERM_WRITE_ACP etc

that means we can call verify_bucket_permission() as most other bucket
ops do, and rely on its call to verify_bucket_permission_no_policy() to
find the owner's acl grant

i also hadn't implemented the cross-account rules for
verify_bucket_owner_or_policy() yet, and didn't want to

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: RoleApplier matches paths in role arns

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: RoleApplier matches account principals

account principals of the form ``arn:aws:iam::123456789012:root``
or ``123456789012`` delegate authority to the account, which means that
it applies to all of the account's users and roles

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: always fill in DisplayName for account owners/grants

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: AssumeRole uses role account as owner

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: fix DeleteRolePolicyResponse

should only be written on success

was misspelled as DeleteRolePoliciesResponse which boto doesn't like:

botocore.parsers.ResponseParserError: Unable to parse response (junk after document element: line 1, column 159), invalid XML received. Further retries may su
cceed:
b'<DeleteRolePoliciesResponse><ResponseMetadata><RequestId>tx0000082c62511b240fd3d-0065d20f66-4129-a2</RequestId></ResponseMetadata></DeleteRolePoliciesResponse><?xml version="1.0" encoding="UTF-8"?><ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/"><Error><Code>NoSuchEntity</Code><Message>The requested PolicyName was not found</Message><RequestId>tx0000082c62511b240fd3d-0065d20f66-4129-a2</RequestId><HostId>4129-a2-a</HostId></Error></ErrorResponse>'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw/multisite: test sync of iam-related metadata

use boto3 for iam connection

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: forward_to_master() passes rgw_owner for effective uid header

when s3 requests get forwarded to the master zone in multisite, we sign
them as the multisite system user because we need to extend the s3
protocol. for example, CreateBucket requests issues by a system user
include an extra response body that encodes the RGWBucketInfo. this way,
the secondary zone can recreate exactly the same bucket that the master
zone did

these forwarded requests include a header like "rgwx-uid: myuserid" to
request that the system user impersonate the given uid. this isn't
necessary for authorization, because the system user overrides
permission checks already. but it's important for resource ownership -
the result of a forwarded CreateBucket request should be a bucket owned
by "myuserid", not the system user

because this "rgwx-uid" header is concerned with ownership, we pass the
string encoding of rgw_owner instead of rgw_user. on the receiving side,
we parse this header in SysReqApplier and override get_aclowner() to
expose it

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: RGWRoleMetadataHandler creates with exclusive=false

metadata sync needs to be able to overwrite existing role metadata

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: only RGWDeleteRole returns ERR_DELETE_CONFLICT

metadata sync calls RadosRole::delete_obj() after the role is deleted on
the metadata master zone. the role was verified to be empty there, so
metadata sync needs to delete the role anyway

only the iam DeleteRole api should require policies to be removed first

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: use retry_raced_role_write() for Role apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: UserPolicy apis use forward_iam_request_to_master()

fix signature mismatch errors when PutUserPolicy/DeleteUserPolicy are
forwarded in multisite

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: use retry_raced_user_write() for User/AccessKey apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/role: use CreateDate from forwarded CreateRole response

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: 'user stats' redirects to 'account stats'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: bucket list --uid redirects to account buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/radosgw: add awscli examples

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/radosgw: document iam managed policies

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/radosgw: start on iam/account docs

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: load and evaluate group policies

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename iam_user_policies to iam_identity_policies

identity policies can come from iam groups and roles too

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: add Group/GroupPolicy APIs

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: ListUserPolicies supports Marker/MaxItems

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: add backend interfaces for group metadata

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add struct RGWGroupInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: OpenIDConnectProvider apis support account users

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: remove virtual class RGWOIDCProvider

class RGWOIDCProvider was doing a lot of different things, so i've split
out its responsibilities:

* move data members and encoding into new struct RGWOIDCProviderInfo,
  and add ceph-dencoder hooks for regression testing
* remove RGWOIDCProvider class and add load/store/delete/list functions
  to the sal::Driver interface
* rgw_rest_oidc_provider.cc handles most of the parameter validation,
  ARN parsing, and json formatting

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: refactor OIDC ops

rearrange the RGWRESTOp subclasses so that the base RGWRestOIDCProvider
can provide a simple verify_permission() that works the same for all
derived ops

Signed-off-by: Casey Bodley <cbodley@redhat.com>

vstart/rgw: add account users for s3-tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add commands for managed policy

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: AttachRolePolicy adds managed role policy

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: AttachUserPolicy adds managed user policy

implement iam apis AttachUserPolicy, DetachUserPolicy, and
ListAttachedUserPolicies to manipulate managed user policy

the set of managed policy ARNs is stored in the user attr
RGW_ATTR_MANAGED_POLICY

for incoming requests, the policies from RGW_ATTR_MANAGED_POLICY are
added to s->iam_user_policies at the same time as RGW_ATTR_USER_POLICY

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: add get_managed_policy() factory function

add definitions for the following managed policy ARNs:

* arn:aws:iam::aws:policy/IAMFullAccess
* arn:aws:iam::aws:policy/IAMReadOnlyAccess
* arn:aws:iam::aws:policy/AmazonSNSFullAccess
* arn:aws:iam::aws:policy/AmazonSNSReadOnlyAccess
* arn:aws:iam::aws:policy/AmazonS3FullAccess
* arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

factory function get_managed_policy() returns a parsed Policy for the
requested ARN if available

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: add lots of actions needed for managed policies

in order to parse managed policies, we have to recognize all of the
actions and wildcards they use

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: Policy() takes string instead of bufferlist

the constructor immediately called bufferlist::to_str() to convert it
into a string; just take string so callers don't have to convert it

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: evaluate_iam_policies() handles account root user

> By default, all requests are implicitly denied with the exception of
> the AWS account root user, which has full access.

the account root user turns an implicit deny from identity policy into
an allow, though other policies can still deny explicitly

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/auth: account users match account arns

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add cross-account policy evaluation

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add generic evaluate_iam_policies()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: verify_permission logs acl grants

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: adapt verify_user_permission() for account users

Signed-off-by: Casey Bodley <cbodley@redhat.com>

vstart/rgw: add default config for sts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/role: support Description for Create/Get/UpdateRole

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: add s3:Get/PutBucketOwnershipControls

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: remove load_account_role_by_name()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/role: role APIs support account users

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/role: separate dump_iam_role() for iam api

create a new dump_iam_role() for iam api responses that dumps the subset
of role information presented by the apis

RGWRoleInfo::dump() and decode_json() are used by metadata sync to
transfer role metadata between zones, so must contain all information
about the role

Signed-off-by: Casey Bodley <cbodley@redhat.com>