rgw: update bucket index marker before skipping entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix data sync notification

should send zone id, not zone name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove incorrect conn assignment

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: incremental sync, skip non-complete entries early

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: forward bucket creation removal if it's not meta master

We can be in the master zonegroup, but not the meta master -- in which case
we still need to forward the request to the master.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, don't yield when spawning

this enables us to filter out duplicate keys without worrying about races, since
spawned crs will only be executed once we yield.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep track of how busy a cr stack is

this can give indication about crs that aren't structured correctly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: avoid unneeded yield

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix setting of rgw_zonegroup from rgw_region

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: clean backward support for region in the cli

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use zonegroup id for location constrain

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone set add a check for zone name in json

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: create a unique zone pool names

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: don't spawn multiple concurrent object sync for same object

Either mark the current running operation to retry, or wait for it to finish
(if it's not doing the same operation).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix call to index_key_to_marker()

was passing variables in the wrong order

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cancel index operation if delete got ENOENT

Should mark the operation as canceled, so that sync followers don't try to
apply it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework set_description(), set_status()

make it easier to construct description, status info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, only use complete log entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more coroutines annotation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start annotating coroutines status and description

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove sync reporting

will use the cr reporting framework instead

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep status and history in coroutine

will replace the report container stuff

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove run context when finished running

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: track all stacks, not just scheduled

And don't expand spawned stacks out of existing cr. This flattens the
admin socket output.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more on fix for g_ceph_context shutdown

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: handle the case of non exisiting zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: allow region use for backward compatability

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: admin socket related fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: hook cr managers to admin socket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add locking to coroutines reporting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start moving report into coroutines

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more metadata sync reporting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start hooking sync report

also multiple fixes around the report tool

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: framework for per-rgw sync status reporting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: add deque encode/decode

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: replace region with zonegroup in the help file

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix assert on shutdown by keeping g_ceph_context around longer

RGWRealmWatcher and friends are on the stack in main(), so we can't drop
the last ref on g_ceph_context until their destructors have fired. use a
boost::intrusive_ptr to claim g_ceph_context's initial ref so that its
destructor fires last

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove epoch and duplicates from list_periods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix compilation error in RGWPeriodPusher::handle_notify

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix compilation error on i386

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: call drain_all() before returning from some crs

spawned crs are still using marker_tracker, need to drain_all() as
we destroy it in the destructor.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: allocate a larger buffer for full sync omap oid

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: register RGWPeriodPusher with RGWRealmWatcher

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWPeriodPusher shares periods between zones/groups

RGWPeriodPusher implements the RGWRealmWatcher interface to get
notifications for new periods.  when it discovers that it needs to push
a period to other zones, it spawns a thread to send them (and keep
retrying) until all of those zones successfully acknowledge the period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRealmWatcher supports multiple watch types

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename RGWRealmReloader::FrontendPauser to Pauser

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period push succeeds even if it isn't a new period

when pushing periods between zones, we need a successful result to
signal that we can stop sending the period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: refactor reconfigure out of RGWRealmWatcher

add a RGWRealmReloader class to handle reconfiguration, and use a
Watcher interface to get notifications from RGWRealmWatcher

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: keep Context for reconfig timer cancellation

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWRealm::notify_new_period()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix typo in RGWPeriod::update

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix for decode of ZoneGroup id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: cr call() doesn't return a value

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync marker tracker, don't assert

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: we need to rename the zone inside the zonegroup too

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix RGWSystemMetaObj::rename

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add --zone-new-name to usage

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: init num shards if sync status did not exist

in data sync process

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use correct connection for certain data sync operations

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: translate source_zone param into source zone id

operations are done using the zone id, not the zone name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start marker tracker sync entry can fail

We might hit duplicate entry, just skip it, don't assert.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clarify log message

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch bucket instance info if doesn't exist

when doing data sync. This handles a race in which we try to sync data
of bucket instance before fetched its metadata

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't force pool names to start with periods

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more backoff related changes and fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move code around

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: backoff mechanism for data sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync CR retry after specific errors

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initialize data sync if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: full data sync, pass correct op

Need to specify appropriate op (depending on whether object is
versioned or not versioned).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: special handling of 'null' object version

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initialize data member

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep timestamp stats in incremental sync position data

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: init REST connection by zone name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutine manager returns ECANCELED when going down

RGWRemoteMetaLog::run_sync() was getting stuck in an infinite loop after
a call to RGWCoroutinesManager::stop().  instead of returning success
when we see 'going_down', fail the coroutine with ECANCELED

the other RGWCoroutinesManager::run() overload also needed modification
to prevent us from overwriting this error code

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix compilation with older boost versions that don't have asio coroutines

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove include to <boost/asio/coroutine.h>

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove include to <boost/asio/coroutine.h>

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: show data full sync stats

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep sync progress info for full sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update the correct shards when doing full sync

The shard we use for each entry (either meta, or data) needs to be selected
correctly to match the different logs we keep. Otherwise async notifications
don't work correctly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: shutdown lease CR on mdlog inc sync failure

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use boost::intrusive_ptr for http_op ref counting

this fixes a segfault in the RGWPostRESTResourceCR::request_complete()
error path, which accessed http_op after calling put()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: apply RGWRESTReadResource::wait_bl() fix to PostResource as well

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRESTPostResource takes param_list_t for RGWPostRESTResourceCR

RGWPostRESTResourceCR needs to make a copy of the parameters given to
its constructor, because it doesn't use them until send_request(). this
results in a segfault when the given parameter list is on the stack

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add make_param_list() helper

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add typedef for rest parameter lists

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use insert to copy containers in rgw_rest_conn

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move RGWPostRESTResourceCR to rgw_cr_rest.h

for use by RGWRealmPusher

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRestConn uses vector instead of map

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: workaround to clear location constrain when using default zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: bucket ops, obj create and obj delete should check to see if bucket exists

rgw_build_policies set the bucket_exists flag

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: realm watcher locks mutex for SafeTimer::shutdown()

this fixes an assert() on shutdown

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix bucket creation and deletion when there is no realm or period configured

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use default zonegroup and zone in init_complete

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: allow creating a bucket without current period

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix data sync initialization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move data sync init into the processing thread

that way we don't depend on remote peer to be available

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework incremental md sync error handling

similar to what we do with the full md sync. Identify transient
errors, and if so return -EAGAIN so that caller would retry.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>