rgw/test_multi.py: meta checkpoint uses zones, not clusters

The unit we need to care about is the zone, not the cluster

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multy.py: improve test_object_sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clear 'null' object instance when fetching remote obj

otherwise the objclass code will handle it as a versioned object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi: buckets and objects comparison

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi: test object sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add datalog status command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta sync can remove entries

and also skip non complete entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clone mdlog updates marker in any case

Need to set the new_marker even if no new entries fetched.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi: add more bucket related tests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta sync, handle ENOENT, ECANCELED

retry when canceled, continue when entry does not exist.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: test_multi: start radosgw with debug logs, asok

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

tests_multi.py: test bucket creation, fix meta checkpoint

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: can run through python nose

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: generate boto connection per zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: create a test user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_python.py: call radosgw-admin with cache disabled where applicable

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: move meta checkpoint inside, add configurables

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: generate random key

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: wait for meta sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: finer control for bringing up system

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: more test work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: fix help.t

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove error message from read_default

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone list should print default zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: default zone per realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: default zonegroup per realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: create_default cannot set as default

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: cleanup remove realm unused master_zonegroup and zonegroups

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: use realm current period master_zonegroup as remote for period pull

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

test/rgw: wait for metadata sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add mdlog status command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: basic meta sync status test util

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: initial work on multi site test framework

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: option to set realm, zonegroup, zone as default

when modifying / creating the entity. This is in addition to the explicit
command.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix period pull of first epoch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http client lock related fix

need to call _finish_request() under the lock.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more log info on realm reloading

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync, don't drain_all() before shutting down lease cr

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch to local zonegroup config if zone is not found

We might be just creating a zone that is not in the period config yet,
if we can't find it in period config switch to using the local zonegroup
config.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: make realm pull a raw storage operation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adding more reflection where needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rearrange zonegroup and zone initialization

move into helpers, don't keep distinct structures for zonegroup local and period
zonegroup, we end up using the one we configured anyway.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: report error

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reflect period when setting as current period

reflect period into local objects when setting it as current.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: realm pull also pulls period

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data shard sync locking

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: take a lease before syncing bucket shard

This is needed for having multiple rgws running on the same zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: merge related fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update bucket index marker before skipping entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix data sync notification

should send zone id, not zone name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove incorrect conn assignment

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: incremental sync, skip non-complete entries early

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: forward bucket creation removal if it's not meta master

We can be in the master zonegroup, but not the meta master -- in which case
we still need to forward the request to the master.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, don't yield when spawning

this enables us to filter out duplicate keys without worrying about races, since
spawned crs will only be executed once we yield.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep track of how busy a cr stack is

this can give indication about crs that aren't structured correctly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: avoid unneeded yield

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix setting of rgw_zonegroup from rgw_region

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: clean backward support for region in the cli

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use zonegroup id for location constrain

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone set add a check for zone name in json

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: create a unique zone pool names

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: don't spawn multiple concurrent object sync for same object

Either mark the current running operation to retry, or wait for it to finish
(if it's not doing the same operation).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix call to index_key_to_marker()

was passing variables in the wrong order

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cancel index operation if delete got ENOENT

Should mark the operation as canceled, so that sync followers don't try to
apply it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework set_description(), set_status()

make it easier to construct description, status info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, only use complete log entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more coroutines annotation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start annotating coroutines status and description

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove sync reporting

will use the cr reporting framework instead

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep status and history in coroutine

will replace the report container stuff

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove run context when finished running

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: track all stacks, not just scheduled

And don't expand spawned stacks out of existing cr. This flattens the
admin socket output.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more on fix for g_ceph_context shutdown

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: handle the case of non exisiting zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: allow region use for backward compatability

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: admin socket related fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: hook cr managers to admin socket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add locking to coroutines reporting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start moving report into coroutines

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more metadata sync reporting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start hooking sync report

also multiple fixes around the report tool

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: framework for per-rgw sync status reporting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: add deque encode/decode

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: replace region with zonegroup in the help file

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix assert on shutdown by keeping g_ceph_context around longer

RGWRealmWatcher and friends are on the stack in main(), so we can't drop
the last ref on g_ceph_context until their destructors have fired. use a
boost::intrusive_ptr to claim g_ceph_context's initial ref so that its
destructor fires last

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove epoch and duplicates from list_periods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix compilation error in RGWPeriodPusher::handle_notify

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix compilation error on i386

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: call drain_all() before returning from some crs

spawned crs are still using marker_tracker, need to drain_all() as
we destroy it in the destructor.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: allocate a larger buffer for full sync omap oid

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: register RGWPeriodPusher with RGWRealmWatcher

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWPeriodPusher shares periods between zones/groups

RGWPeriodPusher implements the RGWRealmWatcher interface to get
notifications for new periods.  when it discovers that it needs to push
a period to other zones, it spawns a thread to send them (and keep
retrying) until all of those zones successfully acknowledge the period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRealmWatcher supports multiple watch types

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename RGWRealmReloader::FrontendPauser to Pauser

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period push succeeds even if it isn't a new period

when pushing periods between zones, we need a successful result to
signal that we can stop sending the period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: refactor reconfigure out of RGWRealmWatcher

add a RGWRealmReloader class to handle reconfiguration, and use a
Watcher interface to get notifications from RGWRealmWatcher

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: keep Context for reconfig timer cancellation

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWRealm::notify_new_period()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix typo in RGWPeriod::update

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix for decode of ZoneGroup id

Signed-off-by: Casey Bodley <cbodley@redhat.com>