rgw: add drain_all() after collect() in sync crs

this fixes a segfault caused by parent stacks completing while there are
child stacks outstanding

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: adjust interfaces following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more error handling

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fail when trying to commit period if no master zone configured

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add error_logger to sync env

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync env cleanup

pass RGWDataSyncEnv container object instead of passing around multiple
params. (incomplete)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add a timelog add operation cr

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: replace #warning with a comment

not going to fix it now

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix do_curl_wait() if curl_multi_wait() not defined

needed for older systems

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix create_default to handle EEXIST

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: metadata and data sync share RGWAsyncRadosProcessor

each RGWAsyncRadosProcessor creates 32 worker threads by default, so we
should only create one instance and share it between callers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix RGWSystemMetaObj::init to handle perdefined name correctly

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add error messages to realm::create

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: realm pull should not fail if the realm exists

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use exclusive flag for store_name in RGWSystemMetaObj::create

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: hold reference to req_completion while calling aio_operate()

This is a workaround to fix issue with aio_operate() racing with the
request completion. This way we make sure completion->release() is
called after aio_operate() finishes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: link_pending_requests() fixes

set aside the list of reqs we need to finish, and only then iterate
through that list and finish. Otherwise we corrupt the reqs map. Also
clear reqs map on shutdown so that we don't end up clearing it twice
in certain cases.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more of dropping async ops reference

instead of deleting. This was missing from a previous commit.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix a typo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: in create_default read zone_params in case we raced with another

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add missing break in key rm command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix a few more leaks

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: safe disposal of async processors

the async processor might out its container cr, need to refcount.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: shorten name of RGWAsyncRadosProcessor thread pool

threads now call pthread_setname_np(), and have to enforce a
16-character limit on the names. this fixes the assertion:

ceph/src/common/Thread.cc: 156: FAILED assert(strlen(name) < 16)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: stop meta sync handler when going down

it needed to check internally whether it was going down

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more leak fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: minor cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reorder RGWRados shutdown

dependency order fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clear stack ops when unwinding last op

So that we don't drop the last one's reference twice

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more leak fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix a few leaks

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: forward meta changes to master

(initial work)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update datalog also when unlinking object instance

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rocksdb: set to master

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: 'period push' handler uses RGWPeriodHistory

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add period history and puller to RGWRados

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWPeriodPuller for RGWPeriodHistory

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add exclusive param to RGWPeriod::set_latest_epoch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: split RGWPeriod::reflect out of set_latest_epoch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWPeriodHistory to track period history

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix documentation of RGWRealmWatcher

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use realm epoch to choose most recent period to push

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add realm epoch to RGWRealm and RGWPeriod

RGWRealm::epoch is incremented on each new period, and
RGWPeriod::realm_epoch tracks the realm epoch when it was created

this makes it easy to determine whether a given period comes before or
after another period, which is needed to build a view of the period
history.  it also makes it easy to detect forks in the period history,
where two different periods share the same realm_epoch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRealm::set_current_period takes RGWPeriod

set_current_period() doesn't need to set the predecessor;
RGWPeriod::commit() takes care of that

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: start pushing current period on startup

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period commit updates sync status before writing the object

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: 'realm pull' uses realm's current_period for do_period_pull

instead of using period_id/period_epoch, which are radosgw-admin
arguments that are unused by 'realm pull', use the period id from
realm.get_current_period() and epoch=0 to get its latest epoch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use refrence in read_clear_modified

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zonegroup set and zone set should set the realm id

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix RGWZoneGroup ctor and check period update return code

Add more debug messages

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: init_zg_from_period() fixes

Reported-by: fangchen sun <sunspot0105@gmail.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, handle delete marker removal

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: some debug info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: guard versioned object removal from circular sync

check if marker instance already exists, only apply (and update bi log)
if does not exist. Propagate the versioning flag so that we apply the
appropriate operation on the object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle delete marker when doing incremental data sync

Still need to handle the full sync, and to make sure we apply it
only if-newer (as with the other data sync operations) to avoid
circular sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: log owner, display name when creating delete marker

In the case of delete marker, we need to log the owner and the owner
display name so that we can create it correctly when syncing to a
different zone. We don't need it in other cases so we don't store it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mrgw.sh: modify log file name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update datalog when creating delete marker

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: set_date_header should use header_name

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add mdlog and datalog status to help.t

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fetch remote obj depends on zone id and pg ver

This provides a consistent objects view on zones when changes happen on
the same objects at the same time. The zone id and pg ver are used for break
even decisions when mtime is equal.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep source zone attr on head object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create a short (32bit) zone id, keep it on the period map

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep pg ver on obj state, and send it in HEAD request

send it in HEAD request if it was done by the system user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: store obj pg version in the object's head

this is done atomically when object is written, will allow us
to determine whether object is newer or not.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test-rgw-call.sh: add missing file

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, update high marker when skipping duplicate sync entries

Need to update the high marker when skipping an entry, so that when the processing
entry finishes we set the markers correctly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multi.py: add log level for saner output

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multi.py: meta checkpoint uses zones, not clusters

The unit we need to care about is the zone, not the cluster

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/test_multy.py: improve test_object_sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clear 'null' object instance when fetching remote obj

otherwise the objclass code will handle it as a versioned object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi: buckets and objects comparison

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi: test object sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add datalog status command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta sync can remove entries

and also skip non complete entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clone mdlog updates marker in any case

Need to set the new_marker even if no new entries fetched.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi: add more bucket related tests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta sync, handle ENOENT, ECANCELED

retry when canceled, continue when entry does not exist.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: test_multi: start radosgw with debug logs, asok

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

tests_multi.py: test bucket creation, fix meta checkpoint

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: can run through python nose

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: generate boto connection per zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: create a test user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_python.py: call radosgw-admin with cache disabled where applicable

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: move meta checkpoint inside, add configurables

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: generate random key

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: wait for meta sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_multi.py: finer control for bringing up system

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: more test work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: fix help.t

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove error message from read_default

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone list should print default zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: default zone per realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: default zonegroup per realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: create_default cannot set as default

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: cleanup remove realm unused master_zonegroup and zonegroups

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: use realm current period master_zonegroup as remote for period pull

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

test/rgw: wait for metadata sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add mdlog status command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: basic meta sync status test util

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>