rgw: add realm epoch to RGWRealm and RGWPeriod

RGWRealm::epoch is incremented on each new period, and
RGWPeriod::realm_epoch tracks the realm epoch when it was created

this makes it easy to determine whether a given period comes before or
after another period, which is needed to build a view of the period
history.  it also makes it easy to detect forks in the period history,
where two different periods share the same realm_epoch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRealm::set_current_period takes RGWPeriod

set_current_period() doesn't need to set the predecessor;
RGWPeriod::commit() takes care of that

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: start pushing current period on startup

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period commit updates sync status before writing the object

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: 'realm pull' uses realm's current_period for do_period_pull

instead of using period_id/period_epoch, which are radosgw-admin
arguments that are unused by 'realm pull', use the period id from
realm.get_current_period() and epoch=0 to get its latest epoch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use refrence in read_clear_modified

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zonegroup set and zone set should set the realm id

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix RGWZoneGroup ctor and check period update return code

Add more debug messages

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: init_zg_from_period() fixes

Reported-by: fangchen sun <sunspot0105@gmail.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, handle delete marker removal

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: some debug info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: guard versioned object removal from circular sync

check if marker instance already exists, only apply (and update bi log)
if does not exist. Propagate the versioning flag so that we apply the
appropriate operation on the object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle delete marker when doing incremental data sync

Still need to handle the full sync, and to make sure we apply it
only if-newer (as with the other data sync operations) to avoid
circular sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: log owner, display name when creating delete marker

In the case of delete marker, we need to log the owner and the owner
display name so that we can create it correctly when syncing to a
different zone. We don't need it in other cases so we don't store it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mrgw.sh: modify log file name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update datalog when creating delete marker

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: set_date_header should use header_name

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add mdlog and datalog status to help.t

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fetch remote obj depends on zone id and pg ver

This provides a consistent objects view on zones when changes happen on
the same objects at the same time. The zone id and pg ver are used for break
even decisions when mtime is equal.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep source zone attr on head object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create a short (32bit) zone id, keep it on the period map

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep pg ver on obj state, and send it in HEAD request

send it in HEAD request if it was done by the system user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: store obj pg version in the object's head

this is done atomically when object is written, will allow us
to determine whether object is newer or not.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test-rgw-call.sh: add missing file

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, update high marker when skipping duplicate sync entries

Need to update the high marker when skipping an entry, so that when the processing
entry finishes we set the markers correctly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multi.py: add log level for saner output

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multi.py: meta checkpoint uses zones, not clusters

The unit we need to care about is the zone, not the cluster

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multy.py: improve test_object_sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clear 'null' object instance when fetching remote obj

otherwise the objclass code will handle it as a versioned object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi: buckets and objects comparison

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi: test object sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add datalog status command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta sync can remove entries

and also skip non complete entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clone mdlog updates marker in any case

Need to set the new_marker even if no new entries fetched.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi: add more bucket related tests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta sync, handle ENOENT, ECANCELED

retry when canceled, continue when entry does not exist.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: test_multi: start radosgw with debug logs, asok

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

tests_multi.py: test bucket creation, fix meta checkpoint

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: can run through python nose

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: generate boto connection per zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: create a test user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_python.py: call radosgw-admin with cache disabled where applicable

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: move meta checkpoint inside, add configurables

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: generate random key

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: wait for meta sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: finer control for bringing up system

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: more test work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: fix help.t

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove error message from read_default

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone list should print default zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: default zone per realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: default zonegroup per realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: create_default cannot set as default

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: cleanup remove realm unused master_zonegroup and zonegroups

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: use realm current period master_zonegroup as remote for period pull

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

test/rgw: wait for metadata sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add mdlog status command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: basic meta sync status test util

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: initial work on multi site test framework

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: option to set realm, zonegroup, zone as default

when modifying / creating the entity. This is in addition to the explicit
command.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix period pull of first epoch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http client lock related fix

need to call _finish_request() under the lock.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more log info on realm reloading

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync, don't drain_all() before shutting down lease cr

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch to local zonegroup config if zone is not found

We might be just creating a zone that is not in the period config yet,
if we can't find it in period config switch to using the local zonegroup
config.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: make realm pull a raw storage operation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adding more reflection where needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rearrange zonegroup and zone initialization

move into helpers, don't keep distinct structures for zonegroup local and period
zonegroup, we end up using the one we configured anyway.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: report error

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reflect period when setting as current period

reflect period into local objects when setting it as current.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: realm pull also pulls period

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data shard sync locking

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: take a lease before syncing bucket shard

This is needed for having multiple rgws running on the same zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: merge related fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update bucket index marker before skipping entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix data sync notification

should send zone id, not zone name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove incorrect conn assignment

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: incremental sync, skip non-complete entries early

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: forward bucket creation removal if it's not meta master

We can be in the master zonegroup, but not the meta master -- in which case
we still need to forward the request to the master.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, don't yield when spawning

this enables us to filter out duplicate keys without worrying about races, since
spawned crs will only be executed once we yield.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep track of how busy a cr stack is

this can give indication about crs that aren't structured correctly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: avoid unneeded yield

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix setting of rgw_zonegroup from rgw_region

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: clean backward support for region in the cli

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use zonegroup id for location constrain

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone set add a check for zone name in json

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: create a unique zone pool names

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: don't spawn multiple concurrent object sync for same object

Either mark the current running operation to retry, or wait for it to finish
(if it's not doing the same operation).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix call to index_key_to_marker()

was passing variables in the wrong order

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cancel index operation if delete got ENOENT

Should mark the operation as canceled, so that sync followers don't try to
apply it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework set_description(), set_status()

make it easier to construct description, status info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, only use complete log entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more coroutines annotation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start annotating coroutines status and description

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove sync reporting

will use the cr reporting framework instead

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep status and history in coroutine

will replace the report container stuff

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove run context when finished running

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: track all stacks, not just scheduled

And don't expand spawned stacks out of existing cr. This flattens the
admin socket output.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more on fix for g_ceph_context shutdown

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: handle the case of non exisiting zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>