rgw: keep pg ver on obj state, and send it in HEAD request

send it in HEAD request if it was done by the system user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: store obj pg version in the object's head

this is done atomically when object is written, will allow us
to determine whether object is newer or not.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test-rgw-call.sh: add missing file

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, update high marker when skipping duplicate sync entries

Need to update the high marker when skipping an entry, so that when the processing
entry finishes we set the markers correctly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multi.py: add log level for saner output

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multi.py: meta checkpoint uses zones, not clusters

The unit we need to care about is the zone, not the cluster

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multy.py: improve test_object_sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clear 'null' object instance when fetching remote obj

otherwise the objclass code will handle it as a versioned object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi: buckets and objects comparison

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi: test object sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add datalog status command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta sync can remove entries

and also skip non complete entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clone mdlog updates marker in any case

Need to set the new_marker even if no new entries fetched.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi: add more bucket related tests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta sync, handle ENOENT, ECANCELED

retry when canceled, continue when entry does not exist.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: test_multi: start radosgw with debug logs, asok

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

tests_multi.py: test bucket creation, fix meta checkpoint

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: can run through python nose

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: generate boto connection per zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: create a test user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_python.py: call radosgw-admin with cache disabled where applicable

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: move meta checkpoint inside, add configurables

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: generate random key

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: wait for meta sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: finer control for bringing up system

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: more test work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: fix help.t

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove error message from read_default

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone list should print default zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: default zone per realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: default zonegroup per realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: create_default cannot set as default

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: cleanup remove realm unused master_zonegroup and zonegroups

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: use realm current period master_zonegroup as remote for period pull

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

test/rgw: wait for metadata sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add mdlog status command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: basic meta sync status test util

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: initial work on multi site test framework

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: option to set realm, zonegroup, zone as default

when modifying / creating the entity. This is in addition to the explicit
command.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix period pull of first epoch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http client lock related fix

need to call _finish_request() under the lock.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more log info on realm reloading

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync, don't drain_all() before shutting down lease cr

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch to local zonegroup config if zone is not found

We might be just creating a zone that is not in the period config yet,
if we can't find it in period config switch to using the local zonegroup
config.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: make realm pull a raw storage operation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adding more reflection where needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rearrange zonegroup and zone initialization

move into helpers, don't keep distinct structures for zonegroup local and period
zonegroup, we end up using the one we configured anyway.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: report error

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reflect period when setting as current period

reflect period into local objects when setting it as current.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: realm pull also pulls period

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data shard sync locking

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: take a lease before syncing bucket shard

This is needed for having multiple rgws running on the same zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: merge related fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update bucket index marker before skipping entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix data sync notification

should send zone id, not zone name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove incorrect conn assignment

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: incremental sync, skip non-complete entries early

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: forward bucket creation removal if it's not meta master

We can be in the master zonegroup, but not the meta master -- in which case
we still need to forward the request to the master.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, don't yield when spawning

this enables us to filter out duplicate keys without worrying about races, since
spawned crs will only be executed once we yield.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep track of how busy a cr stack is

this can give indication about crs that aren't structured correctly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: avoid unneeded yield

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix setting of rgw_zonegroup from rgw_region

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: clean backward support for region in the cli

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use zonegroup id for location constrain

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone set add a check for zone name in json

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: create a unique zone pool names

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: don't spawn multiple concurrent object sync for same object

Either mark the current running operation to retry, or wait for it to finish
(if it's not doing the same operation).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix call to index_key_to_marker()

was passing variables in the wrong order

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cancel index operation if delete got ENOENT

Should mark the operation as canceled, so that sync followers don't try to
apply it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework set_description(), set_status()

make it easier to construct description, status info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, only use complete log entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more coroutines annotation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start annotating coroutines status and description

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove sync reporting

will use the cr reporting framework instead

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep status and history in coroutine

will replace the report container stuff

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove run context when finished running

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: track all stacks, not just scheduled

And don't expand spawned stacks out of existing cr. This flattens the
admin socket output.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more on fix for g_ceph_context shutdown

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: handle the case of non exisiting zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: allow region use for backward compatability

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: admin socket related fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: hook cr managers to admin socket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add locking to coroutines reporting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start moving report into coroutines

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more metadata sync reporting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start hooking sync report

also multiple fixes around the report tool

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: framework for per-rgw sync status reporting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: add deque encode/decode

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: replace region with zonegroup in the help file

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix assert on shutdown by keeping g_ceph_context around longer

RGWRealmWatcher and friends are on the stack in main(), so we can't drop
the last ref on g_ceph_context until their destructors have fired. use a
boost::intrusive_ptr to claim g_ceph_context's initial ref so that its
destructor fires last

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove epoch and duplicates from list_periods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix compilation error in RGWPeriodPusher::handle_notify

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix compilation error on i386

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: call drain_all() before returning from some crs

spawned crs are still using marker_tracker, need to drain_all() as
we destroy it in the destructor.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: allocate a larger buffer for full sync omap oid

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: register RGWPeriodPusher with RGWRealmWatcher

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWPeriodPusher shares periods between zones/groups

RGWPeriodPusher implements the RGWRealmWatcher interface to get
notifications for new periods.  when it discovers that it needs to push
a period to other zones, it spawns a thread to send them (and keep
retrying) until all of those zones successfully acknowledge the period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRealmWatcher supports multiple watch types

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename RGWRealmReloader::FrontendPauser to Pauser

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period push succeeds even if it isn't a new period

when pushing periods between zones, we need a successful result to
signal that we can stop sending the period

Signed-off-by: Casey Bodley <cbodley@redhat.com>