rgw: add rgwx-copy-if-newer http header

Similar to IF_MOD_SINCE, but does not take a time argument, and should
protected against races (not there yet).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: make rgw_user::empty() const

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: access zone.name, zonegroup_name through accessor

fix as part of a rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: dump internal object tag when system user lists bucket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: system user can list bucket shard

By setting the HTTP-Rgwx-Shard-Id header to the shard id.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: get_bucket_stats() returns a plain marker if shard specified

E.g, will return 00001234 vs. 2#00001234 if shard 2 specified.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bilog info can get shard id

We would like to be able to get bilog info by shard if needed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix uninitialized variable

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move more code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trivial fix

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use internal wait instead of special coroutine

Instead of using a special coroutine that spawns a separate thread
and waits on a condition, use the internal coroutine stack wait.
This is useful since it since we limit the number of IO threads
we spawn.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutine stack wait util

Instead of using a separate coroutine for waiting on
a condition.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move more code around

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move code around

move code from rgw_sync.cc to rgw_cr_rados.{h,cc}

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rearrange full metadata sync order

Users need to exist before their buckets, better have bucket instances
before bucket entry points

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trivial fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix shutdown

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http client req handling fixes

The error buffer for the async requests was on the stack, and not
on the request structure. Some other use after free and locking fixes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: wake up appropriate shard sync handler

When receiving notification about modified shard, wake up the appropriate handler

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a few sync thread init/finalization fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: run and stop sync thread

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: simple thread base class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: decode mdlog notification shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: send mdlog changes notifications to zones

Will probably need to rework this, as it is now zones may interfere
with each other which is not optimal.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add simple POST requests interfaces

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: add json encoder / decoder for std::set

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: periodical thread that tracks mdlog changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle mdlog post request

with 'notify' subresource. Will be used later for handling notifications
about modified mdlog shards.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep track of modified mdlog shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create the first period when creating a new realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add zone add command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: allow creating a zone without a zone group

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add master flag to zonegroup create command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: period pull supports --url

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: fixes for period push/pull

for send_to_remote_gateway() to work, the push and pull commands had to
be moved to the not-raw-storage section, because RGWRados::init_complete
is needed to set up the gateway connections

the period push command was changed to send the json-encoded period
data, so an in_data argument was added to send_to_remote_gateway()

the parameters for push and pull needed to remain separate from the
request_uri, because request_uri is used as input for the auth digest,
and the server side parses the parameters separately as well

Implements feature #11944

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRESTSimpleRequest::forward_request params

forward_request() uses new_info.build_from(info) to initialize the new
request info. but this doesn't copy the RGWHTTPArgs, so any parameters
associated with info are dropped

it later uses new_info.args.get_params() to generate the params_str,
but no code in between adds any parameters to new_info. so use
info.args.get_params() instead to keep any parameters from the original
request info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: expose Formatter::flush to derived classes

class Formatter declares both a 'virtual void flush(ostream&)' and a
'void flush(bufferlist&)'. however, because its subclasses only override
the former, c++'s function overloading rules cause the latter overload
to be hidden

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: POST /admin/realm/period reads in json data

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: delay allocation of RGWPeriod

RGWOp_Period_Get and _Post don't have a valid RGWRados pointer until
they get the init() call, so the constructors were passing NULL to
RGWPeriod

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add handler for /admin/realm/period

the handler returns the existing RGWOp_Period_Get/Post ops from
rgw_rest_config.h

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fixes for RGWZoneGroupMap::decode_json

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add realm set command

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add json output to create commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix period init and activate

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: update period predecessor when it is activated

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix realm list-periods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: realm id and epoch should be part of period oid

fixes period get and activate command
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add realm list-periods command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: Add master_zonegroup and master_zone to period prepare

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: Add period list command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: print new perion in period perpare command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix zone set command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove debug messages

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: default zone should not be master

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add zone create, delete and rename commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWZoneParams should inherit from RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use id for master zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add id to RGWZoneParams

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add RGWSystemMetaObj class

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_rados.cc

rgw-admin: adjustments to period get-current output

Make it output through the formatter

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: use default realm on realm get

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pass const string reference

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: fix realms list

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix typo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: do the actual incremental metadata sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: track mdlog marker

sync entries if the sync marker is < mdlog marker, otherwise
try to fetch more entries.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't force wait on blocked coroutines

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix busy wait in RGWHTTPManager

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync local mdlog if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: wake up wait coroutine thread on destruction

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Add zone create admin command - wip

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add id to RGWZone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: poll mdlog for incremental sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read mdlog entries for incremental sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: get remote mdlog status when initializing

Use that as the lowerbound for the meta marker when we start
the incremental sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: hash full sync keys to shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch to incremental sync when done full meta sync

each shard need to collect the coroutines it spawned before continuing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update global sync state after building meta indexes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update meta sync markers during full sync operation

update marker per shard, keep a window so that we don't update after each change

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add zonegroup name or id to zonegroup get and set commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add zonegroup create, delete and rename admin commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add period activate admin command and fix get current command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add current_period to RGWRealm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWZoneGroup should inheirt from RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add old_format support to RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add id to zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove region

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: change RGWZoneGroup::get_pool_name to return string

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: convert old region configuration to zonegroups during init

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

fix ceph-dencoder link

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cmake: add rgw_sync.cc and rgw_coroutine.cc

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix async sleep and wakeup logic

lot's of cleanups and fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: full meta sync

need to handle errors, and update markers, but full meta sync is functional

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: naively fetch all meta objects

doing it one by one, later will spawn these concurrently

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read remote meta coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read full sync indexes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: modify sync state, sync run uses state

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutie for omap_get_vals()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep track of spawned coroutines under the current op

instead of keeping a per-stack list of spawned coroutines, keep it
on the op that spawned it. Otherwise, we might spawn a cr, call a second
one that will spawn some crs, and when waiting for these to complete
it will also wait on the first one.
The list of spawned coroutines is inherited. Once a coroutine finishes
its parent will take over all the spawned ones that weren't collected.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: replace omap_set coroutine code

No need to use the async rados worker.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>