test/rgw: wait for metadata sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add mdlog status command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: basic meta sync status test util

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: initial work on multi site test framework

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: option to set realm, zonegroup, zone as default

when modifying / creating the entity. This is in addition to the explicit
command.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix period pull of first epoch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http client lock related fix

need to call _finish_request() under the lock.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more log info on realm reloading

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync, don't drain_all() before shutting down lease cr

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch to local zonegroup config if zone is not found

We might be just creating a zone that is not in the period config yet,
if we can't find it in period config switch to using the local zonegroup
config.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: make realm pull a raw storage operation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adding more reflection where needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rearrange zonegroup and zone initialization

move into helpers, don't keep distinct structures for zonegroup local and period
zonegroup, we end up using the one we configured anyway.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: report error

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reflect period when setting as current period

reflect period into local objects when setting it as current.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: realm pull also pulls period

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data shard sync locking

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: take a lease before syncing bucket shard

This is needed for having multiple rgws running on the same zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: merge related fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update bucket index marker before skipping entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix data sync notification

should send zone id, not zone name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove incorrect conn assignment

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: incremental sync, skip non-complete entries early

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: forward bucket creation removal if it's not meta master

We can be in the master zonegroup, but not the meta master -- in which case
we still need to forward the request to the master.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, don't yield when spawning

this enables us to filter out duplicate keys without worrying about races, since
spawned crs will only be executed once we yield.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep track of how busy a cr stack is

this can give indication about crs that aren't structured correctly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: avoid unneeded yield

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix setting of rgw_zonegroup from rgw_region

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: clean backward support for region in the cli

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use zonegroup id for location constrain

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone set add a check for zone name in json

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: create a unique zone pool names

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: don't spawn multiple concurrent object sync for same object

Either mark the current running operation to retry, or wait for it to finish
(if it's not doing the same operation).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix call to index_key_to_marker()

was passing variables in the wrong order

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cancel index operation if delete got ENOENT

Should mark the operation as canceled, so that sync followers don't try to
apply it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework set_description(), set_status()

make it easier to construct description, status info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, only use complete log entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more coroutines annotation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start annotating coroutines status and description

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove sync reporting

will use the cr reporting framework instead

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep status and history in coroutine

will replace the report container stuff

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove run context when finished running

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: track all stacks, not just scheduled

And don't expand spawned stacks out of existing cr. This flattens the
admin socket output.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more on fix for g_ceph_context shutdown

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: handle the case of non exisiting zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: allow region use for backward compatability

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: admin socket related fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: hook cr managers to admin socket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add locking to coroutines reporting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start moving report into coroutines

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more metadata sync reporting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start hooking sync report

also multiple fixes around the report tool

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: framework for per-rgw sync status reporting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: add deque encode/decode

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: replace region with zonegroup in the help file

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix assert on shutdown by keeping g_ceph_context around longer

RGWRealmWatcher and friends are on the stack in main(), so we can't drop
the last ref on g_ceph_context until their destructors have fired. use a
boost::intrusive_ptr to claim g_ceph_context's initial ref so that its
destructor fires last

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove epoch and duplicates from list_periods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix compilation error in RGWPeriodPusher::handle_notify

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix compilation error on i386

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: call drain_all() before returning from some crs

spawned crs are still using marker_tracker, need to drain_all() as
we destroy it in the destructor.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: allocate a larger buffer for full sync omap oid

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: register RGWPeriodPusher with RGWRealmWatcher

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWPeriodPusher shares periods between zones/groups

RGWPeriodPusher implements the RGWRealmWatcher interface to get
notifications for new periods.  when it discovers that it needs to push
a period to other zones, it spawns a thread to send them (and keep
retrying) until all of those zones successfully acknowledge the period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRealmWatcher supports multiple watch types

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename RGWRealmReloader::FrontendPauser to Pauser

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period push succeeds even if it isn't a new period

when pushing periods between zones, we need a successful result to
signal that we can stop sending the period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: refactor reconfigure out of RGWRealmWatcher

add a RGWRealmReloader class to handle reconfiguration, and use a
Watcher interface to get notifications from RGWRealmWatcher

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: keep Context for reconfig timer cancellation

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWRealm::notify_new_period()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix typo in RGWPeriod::update

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix for decode of ZoneGroup id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: cr call() doesn't return a value

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync marker tracker, don't assert

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: we need to rename the zone inside the zonegroup too

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix RGWSystemMetaObj::rename

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add --zone-new-name to usage

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: init num shards if sync status did not exist

in data sync process

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use correct connection for certain data sync operations

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: translate source_zone param into source zone id

operations are done using the zone id, not the zone name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start marker tracker sync entry can fail

We might hit duplicate entry, just skip it, don't assert.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clarify log message

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch bucket instance info if doesn't exist

when doing data sync. This handles a race in which we try to sync data
of bucket instance before fetched its metadata

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't force pool names to start with periods

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more backoff related changes and fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move code around

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: backoff mechanism for data sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync CR retry after specific errors

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initialize data sync if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: full data sync, pass correct op

Need to specify appropriate op (depending on whether object is
versioned or not versioned).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: special handling of 'null' object version

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initialize data member

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep timestamp stats in incremental sync position data

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: init REST connection by zone name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutine manager returns ECANCELED when going down

RGWRemoteMetaLog::run_sync() was getting stuck in an infinite loop after
a call to RGWCoroutinesManager::stop().  instead of returning success
when we see 'going_down', fail the coroutine with ECANCELED

the other RGWCoroutinesManager::run() overload also needed modification
to prevent us from overwriting this error code

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix compilation with older boost versions that don't have asio coroutines

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove include to <boost/asio/coroutine.h>

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove include to <boost/asio/coroutine.h>

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: show data full sync stats

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep sync progress info for full sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>