rgw: Add zone add command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: allow creating a zone without a zone group

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add master flag to zonegroup create command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: period pull supports --url

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: fixes for period push/pull

for send_to_remote_gateway() to work, the push and pull commands had to
be moved to the not-raw-storage section, because RGWRados::init_complete
is needed to set up the gateway connections

the period push command was changed to send the json-encoded period
data, so an in_data argument was added to send_to_remote_gateway()

the parameters for push and pull needed to remain separate from the
request_uri, because request_uri is used as input for the auth digest,
and the server side parses the parameters separately as well

Implements feature #11944

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRESTSimpleRequest::forward_request params

forward_request() uses new_info.build_from(info) to initialize the new
request info. but this doesn't copy the RGWHTTPArgs, so any parameters
associated with info are dropped

it later uses new_info.args.get_params() to generate the params_str,
but no code in between adds any parameters to new_info. so use
info.args.get_params() instead to keep any parameters from the original
request info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: expose Formatter::flush to derived classes

class Formatter declares both a 'virtual void flush(ostream&)' and a
'void flush(bufferlist&)'. however, because its subclasses only override
the former, c++'s function overloading rules cause the latter overload
to be hidden

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: POST /admin/realm/period reads in json data

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: delay allocation of RGWPeriod

RGWOp_Period_Get and _Post don't have a valid RGWRados pointer until
they get the init() call, so the constructors were passing NULL to
RGWPeriod

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add handler for /admin/realm/period

the handler returns the existing RGWOp_Period_Get/Post ops from
rgw_rest_config.h

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fixes for RGWZoneGroupMap::decode_json

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add realm set command

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add json output to create commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix period init and activate

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: update period predecessor when it is activated

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix realm list-periods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: realm id and epoch should be part of period oid

fixes period get and activate command
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add realm list-periods command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: Add master_zonegroup and master_zone to period prepare

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: Add period list command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: print new perion in period perpare command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix zone set command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove debug messages

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: default zone should not be master

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add zone create, delete and rename commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWZoneParams should inherit from RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use id for master zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add id to RGWZoneParams

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add RGWSystemMetaObj class

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_rados.cc

rgw-admin: adjustments to period get-current output

Make it output through the formatter

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: use default realm on realm get

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pass const string reference

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: fix realms list

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix typo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: do the actual incremental metadata sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: track mdlog marker

sync entries if the sync marker is < mdlog marker, otherwise
try to fetch more entries.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't force wait on blocked coroutines

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix busy wait in RGWHTTPManager

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync local mdlog if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: wake up wait coroutine thread on destruction

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Add zone create admin command - wip

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add id to RGWZone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: poll mdlog for incremental sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read mdlog entries for incremental sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: get remote mdlog status when initializing

Use that as the lowerbound for the meta marker when we start
the incremental sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: hash full sync keys to shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch to incremental sync when done full meta sync

each shard need to collect the coroutines it spawned before continuing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update global sync state after building meta indexes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update meta sync markers during full sync operation

update marker per shard, keep a window so that we don't update after each change

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add zonegroup name or id to zonegroup get and set commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add zonegroup create, delete and rename admin commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add period activate admin command and fix get current command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add current_period to RGWRealm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWZoneGroup should inheirt from RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add old_format support to RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add id to zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove region

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: change RGWZoneGroup::get_pool_name to return string

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: convert old region configuration to zonegroups during init

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

fix ceph-dencoder link

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cmake: add rgw_sync.cc and rgw_coroutine.cc

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix async sleep and wakeup logic

lot's of cleanups and fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: full meta sync

need to handle errors, and update markers, but full meta sync is functional

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: naively fetch all meta objects

doing it one by one, later will spawn these concurrently

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read remote meta coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read full sync indexes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: modify sync state, sync run uses state

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutie for omap_get_vals()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep track of spawned coroutines under the current op

instead of keeping a per-stack list of spawned coroutines, keep it
on the op that spawned it. Otherwise, we might spawn a cr, call a second
one that will spawn some crs, and when waiting for these to complete
it will also wait on the first one.
The list of spawned coroutines is inherited. Once a coroutine finishes
its parent will take over all the spawned ones that weren't collected.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: replace omap_set coroutine code

No need to use the async rados worker.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix the simple coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix async sleep and wakeup logic

lot's of cleanups and fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutines producer consumers

Create a new producer-consumer(s) abstraction for the coroutines
infrastructure.
Move environment info from the op into the stack. Stack can be
sent to sleep and awaken explicitly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: build index of all master zone meta keys

still wip, but getting there

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create a coroutine for fetching REST api objects

this simplifies the code that yanks all the metadata from the master

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch all metadata entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta sync cleanups

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove calls for synchronous rados ops

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: dump shard status on mdlog sync status

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initialize all shard markers

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: return error if any spawned op errored

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: iterate through all mdlog sync status shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unused code, cleanups

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adjust mdlog fetch to use new interfaces

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix init mdlog sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mdlog sync init

initialize mdlog sync state, uses the implicit yield scheme (from boost)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use boost for stackless coroutine flow

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add cookie to lock coroutines

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more lock/unlock coroutines work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: prepare rados lock operation to be used with coroutines

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: simple rgw metadata write coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move coroutines code

move the generic code to rgw_coroutine.{h,cc}

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: return retcode of failing coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rename async ops to coroutines

which they basically are

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fail if master zone tries to use meta sync

We'll change it later, but as it is now that's the way to handle it
gracefully.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: Fix init_complete

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: support old region configurations

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Remove region admin commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add zone group admin commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: zonegroup json encoding/decoding with backward compatability

Signed-off-by: Orit Wasserman <owasserm@redhat.com>