qa/rgw: multisite checkpoints consider pubsub zone

pubsub zones are configured to only sync_from the master zone, so bucket
checkpoints should ignore its sync status with respect to other zones

Fixes: https://tracker.ceph.com/issues/43768
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31686 from yehudasa/wip-rgw-sync-bucket-6

rgw: bucket granularity sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: bucket sync: invalidate hints data if raced with another writer

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix compilation after rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't include empty buckets in potential related buckets

Being used by the sync hint system

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep sync_policy as optional member inside RGWBucketInfo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move a few basic types into rgw_basic_types.{h,cc}

To help with the fight against circular dependencies

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: initiate bucket meta sync if can't find sync handler

If bucket sync handler returns -ENOENT, try to get bucket info. This can
trigger bucket meta sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trim bilog: send bucket param instead of source-bucket param

For backward compatibility, so that older rgws could handle it appropriately.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: don't allow replication api on legacy config

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

PendingReleaseNotes: rgw, bucket granularity

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clarify api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: get_hint_entities(), call read_bucket_info once

once per bucket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: no create param in find directional/symmetrical

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: don't use optional for some members

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix a potential leak

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove return std::move()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch_remote_obj() create default acls if not found

Create default acls if the original object did not have acl header.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove superfluous alias

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: doc fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: find first key with longest matching prefix

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: fix bucket sync markers for non-trivial case

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket autotrim: fetch aggregated bilog info of all bucket targets

Add a new option to the bilog status request that allows returning the
min status for all buckets in a zone that sync from a specific bucket.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket trim: only fetch status from relevant zones

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: use list instead of vector

Keeping pointers into entries inside the container, vector might be
reallocated.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync policy doc

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_argparse: binary flag value can be separate arg

Resurrect old behavior where a binary flag could be detached from the
argument, e.g., "--rgw-cache-enabled false".

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: fix use of opt_storage_class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa: s3a-hadoop: fix user creation

This breaks with the new more strict radosgw-admin command line parsing

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: source zone, dest zone converted to zone id

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pipe map in pipe_set should be multimap

Can hold multiple entries for the same pipe id (if flow defines
more than a single entry).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: flatten binary encoding for rgw_sync_pipe_params

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: implement s3 get bucket replication api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: implement s3 delete bucket replication api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: put bucket replication extension

Now able to set specific source and destination zones

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: implement s3 put-bucket-replication api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix pipes zone wildcard match logic

and improve logging

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: bucket sync status related fixes and improvements

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use negative shard id when creating sync pair

negative shard id means the bucket has a single shard. This
is to be backward compatible.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unneeded #warnings

and other cleanups

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: only update data log if bucket exports data

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: user source bucket owner for event, not dest

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove a #warning with a comment

It's a nice to have, not crucial

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: object sync info oid depends on pipe

The object sync info object (used in cloud sync module) depends on pipe. Different
oid is crafted if source and destination buckets are not the same.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: guard against racing source object change

Source object might have changed since we tried to sync it,
and the decisions we made originally might be irrelevant.
After reading object header, check if the original dest params
are the same dest params we find now for this object, otherwise
need to retry.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: override dest object owner if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: check user permission in user level sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: auth: api to create auth identity without req_state

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api to check bucket and object permissions without need for req_state

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: user mode sync pipes fetch objects using effective uid

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: prepare for system and user mode changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: handle preconfition failed error

Handle precondition failed error. This would happen in case we cannot
sync the object due to either newer local changes, or if the policy
doesn't allow syncing the specific object. Handle this as success, but
log a message.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http client: propagate errors in client callbacks

If one of the callbacks return error, stop the processing and set that
as the request error.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: filter fetch remote obj after reading remote metadata

In the sync case: find the appropriate pipe params that match this
remote object (if matches tags / prefix), and adjust fetch if needed --
abort if doesn't match tags. Later will adjust acls as needed (if user
sync and not system).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: find pipe rules param for specific obj changes

 - reuse RGWObjTags::tag_map_t instead of vector of strings
 - fix the prefix search

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: add mode (system, user) as a pipe param

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add delimiter to tags parsing

Otherwise it will use '=' as a delimiter, and need to preserve it as
part of the tag initially.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: minor message fix

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync policy: reorganize a bit

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: json encoding filter

Can create and register a filter (that will be pointed at by the formatter
as an external feature), that could be then used to replace default
encoders for specific data types. This can be useful in cases where
we want to mutate the output (for example: translate zone ids to zone
names on the fly).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, radosgw-admin: explicit zone id type

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: resolve hints when initializing policy handler

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: modify zone_trace for bucket granularity info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: incremental prefixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: full sync checks for rules prefixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use pipe handlers instead of pipes off pipe_set

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: pipe rules handler

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate sync pipe params down the stack

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add option to configure sync pipe prefix, tags

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: extend data structures to extend bucket sync functionality

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: use hints when determining sync pipes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: resolve pipes by bucket sync hints

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync info shows and resolve bucket sync hints

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: manage bucket sync deps index

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: identify potential related (for sync) buckets on bucket update

Generate a diff off old and new bucket infos.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.zone: keep policy handler for all zones

So that we can use the correct policy handler when checking source bucket.
Also build list of source zones for data sync out of all potential zones,
and not just the ones that are enabled, because it can be enabled at the
bucket level.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: trigger per shard sync by source and target

and tie to data log sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: rework RGWGetBucketPeersCR

so that it returns pipes with bucket info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: more work towards new policy integration

RGWGetBucketPeersCR will be able to return correct buckets
that need to sync, either by a target bucket, and/or a source
zone and a source bucket (partially implemented).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: bucket sync manager adjustments for new system

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: encoding fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: modify sync policy handler api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: bucket sync status changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix, cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync flow handler: integrate with flow manager

Convert zone names to zone ids, adjust interfaces.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync flow: init zone level sync flow manager

Hold the zone level sync flow manager in the zone svc. Convert old
zonegroup sync config into new sync policy structure.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync policy mgr: more fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move code around

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync policy: cleanups and fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rgw_bucket operator fixes

Fix smaller-than operator, ostream operator.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: flow manager generates explicit pipes

Have multiple pipes instead of having a single pipe that can describe
 multiple different zones have multiple pipes, each point at a single
zone (or all zones).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use single entity in expanded sync determination

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rename some datastructures to signify plurality

pipe -> pipes, entity -> entities

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket flow manager initialization changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add 'sync info' command

Also move code into header so that flow manager could be used in radosgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: manage sync policy at the bucket level

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move code around

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: code cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync group pipe create and remove

also rename sync group flow add to sync group flow create.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>