rgw: list bucket instance shard entries for data sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read bucket sync status

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: per-bucket instance shard state

add 'radosgw-admin bucket sync init' command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial work on data sync run

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: implement data sync init

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: data sync status command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial data sync work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: skeleton for data sync commands

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: switch mdlog sync commands to metadata sync

radosgw-admin mdlog sync init -> metadata sync init
etc.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

build: add cls_timeindex_client to DENCODER_DEPS

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: propagate rgwx-bucket-instance for copying objects

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix time parsing for delete precondition

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add no-precondition-error http param for object delete

In case we don't want to get an error response when a precondition
failed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add mtime check to object removal if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: add an objclass method to check obj mtime

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add DELETE_IF_UNMODIFIED_SINCE header to obj delete

Will delete object only if not modified after the specified time. Still
need to close a race where object is modified after the check.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more api adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adjust calls

as part of  a rebase work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: identify racing writes when using copy-if-newer

When copying an object from a different zone, and copy-if-newer is
specified, if the final meta write is canceled check whether the
destinatioin that was created is actually newer than our mtime,
otherwise retry.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: translate http error 304 to appropriate status

So that we can return the appropriate response on a conditioinal multi
zone copy operation.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix if-modified-since boundary

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add rgwx-copy-if-newer http header

Similar to IF_MOD_SINCE, but does not take a time argument, and should
protected against races (not there yet).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: make rgw_user::empty() const

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: access zone.name, zonegroup_name through accessor

fix as part of a rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: dump internal object tag when system user lists bucket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: system user can list bucket shard

By setting the HTTP-Rgwx-Shard-Id header to the shard id.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: get_bucket_stats() returns a plain marker if shard specified

E.g, will return 00001234 vs. 2#00001234 if shard 2 specified.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bilog info can get shard id

We would like to be able to get bilog info by shard if needed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix uninitialized variable

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move more code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trivial fix

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use internal wait instead of special coroutine

Instead of using a special coroutine that spawns a separate thread
and waits on a condition, use the internal coroutine stack wait.
This is useful since it since we limit the number of IO threads
we spawn.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutine stack wait util

Instead of using a separate coroutine for waiting on
a condition.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move more code around

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move code around

move code from rgw_sync.cc to rgw_cr_rados.{h,cc}

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rearrange full metadata sync order

Users need to exist before their buckets, better have bucket instances
before bucket entry points

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trivial fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix shutdown

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http client req handling fixes

The error buffer for the async requests was on the stack, and not
on the request structure. Some other use after free and locking fixes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: wake up appropriate shard sync handler

When receiving notification about modified shard, wake up the appropriate handler

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a few sync thread init/finalization fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: run and stop sync thread

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: simple thread base class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: decode mdlog notification shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: send mdlog changes notifications to zones

Will probably need to rework this, as it is now zones may interfere
with each other which is not optimal.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add simple POST requests interfaces

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: add json encoder / decoder for std::set

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: periodical thread that tracks mdlog changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle mdlog post request

with 'notify' subresource. Will be used later for handling notifications
about modified mdlog shards.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep track of modified mdlog shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create the first period when creating a new realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add zone add command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: allow creating a zone without a zone group

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add master flag to zonegroup create command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: period pull supports --url

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: fixes for period push/pull

for send_to_remote_gateway() to work, the push and pull commands had to
be moved to the not-raw-storage section, because RGWRados::init_complete
is needed to set up the gateway connections

the period push command was changed to send the json-encoded period
data, so an in_data argument was added to send_to_remote_gateway()

the parameters for push and pull needed to remain separate from the
request_uri, because request_uri is used as input for the auth digest,
and the server side parses the parameters separately as well

Implements feature #11944

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRESTSimpleRequest::forward_request params

forward_request() uses new_info.build_from(info) to initialize the new
request info. but this doesn't copy the RGWHTTPArgs, so any parameters
associated with info are dropped

it later uses new_info.args.get_params() to generate the params_str,
but no code in between adds any parameters to new_info. so use
info.args.get_params() instead to keep any parameters from the original
request info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: expose Formatter::flush to derived classes

class Formatter declares both a 'virtual void flush(ostream&)' and a
'void flush(bufferlist&)'. however, because its subclasses only override
the former, c++'s function overloading rules cause the latter overload
to be hidden

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: POST /admin/realm/period reads in json data

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: delay allocation of RGWPeriod

RGWOp_Period_Get and _Post don't have a valid RGWRados pointer until
they get the init() call, so the constructors were passing NULL to
RGWPeriod

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add handler for /admin/realm/period

the handler returns the existing RGWOp_Period_Get/Post ops from
rgw_rest_config.h

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fixes for RGWZoneGroupMap::decode_json

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add realm set command

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add json output to create commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix period init and activate

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: update period predecessor when it is activated

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix realm list-periods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: realm id and epoch should be part of period oid

fixes period get and activate command
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add realm list-periods command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: Add master_zonegroup and master_zone to period prepare

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: Add period list command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: print new perion in period perpare command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix zone set command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove debug messages

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: default zone should not be master

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add zone create, delete and rename commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWZoneParams should inherit from RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use id for master zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add id to RGWZoneParams

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add RGWSystemMetaObj class

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_rados.cc

rgw-admin: adjustments to period get-current output

Make it output through the formatter

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: use default realm on realm get

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pass const string reference

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: fix realms list

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix typo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: do the actual incremental metadata sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: track mdlog marker

sync entries if the sync marker is < mdlog marker, otherwise
try to fetch more entries.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't force wait on blocked coroutines

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix busy wait in RGWHTTPManager

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync local mdlog if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: wake up wait coroutine thread on destruction

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Add zone create admin command - wip

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add id to RGWZone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: poll mdlog for incremental sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read mdlog entries for incremental sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: get remote mdlog status when initializing

Use that as the lowerbound for the meta marker when we start
the incremental sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: hash full sync keys to shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch to incremental sync when done full meta sync

each shard need to collect the coroutines it spawned before continuing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update global sync state after building meta indexes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update meta sync markers during full sync operation

update marker per shard, keep a window so that we don't update after each change

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>