rgw: update the correct shards when doing full sync

The shard we use for each entry (either meta, or data) needs to be selected
correctly to match the different logs we keep. Otherwise async notifications
don't work correctly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: shutdown lease CR on mdlog inc sync failure

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use boost::intrusive_ptr for http_op ref counting

this fixes a segfault in the RGWPostRESTResourceCR::request_complete()
error path, which accessed http_op after calling put()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: apply RGWRESTReadResource::wait_bl() fix to PostResource as well

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRESTPostResource takes param_list_t for RGWPostRESTResourceCR

RGWPostRESTResourceCR needs to make a copy of the parameters given to
its constructor, because it doesn't use them until send_request(). this
results in a segfault when the given parameter list is on the stack

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add make_param_list() helper

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add typedef for rest parameter lists

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use insert to copy containers in rgw_rest_conn

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move RGWPostRESTResourceCR to rgw_cr_rest.h

for use by RGWRealmPusher

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRestConn uses vector instead of map

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: workaround to clear location constrain when using default zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: bucket ops, obj create and obj delete should check to see if bucket exists

rgw_build_policies set the bucket_exists flag

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: realm watcher locks mutex for SafeTimer::shutdown()

this fixes an assert() on shutdown

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix bucket creation and deletion when there is no realm or period configured

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use default zonegroup and zone in init_complete

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: allow creating a bucket without current period

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix data sync initialization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move data sync init into the processing thread

that way we don't depend on remote peer to be available

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework incremental md sync error handling

similar to what we do with the full md sync. Identify transient
errors, and if so return -EAGAIN so that caller would retry.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move children collection to other method

and call it again instead of just draining at the end

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle transient error in md full sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutine drain fixes

Init drain_cr() before draining, otherwise we wouldn't be able to
call it a second time

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWCoroutine::spawn() returns stack

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a coroutine util function to collect a single stack

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework error handling when syncing a single md entry

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: get some errors from req_data

When reading request status, we need to check with req_data first,
since it might have failed before we even got the chance to set our
own status.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: multiple sync initialization and error handling fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: return -EAGAIN on a connection related http failure

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: error out if failed to build md sync index

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: init sync env

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mdlog full sync, yield for each entry

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move sync_oid() and shard_object_name() to sync env

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create RGWMetaSyncEnv, use it as container to sync env

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWPeriod stores sync_status as vector of strings

rgw_meta_sync_status keeps the shards in a std::map, but a std::vector
is more appropriate when all keys are present. the period also doesn't
need to store the full/incremental status in rgw_meta_sync_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: store metadata sync status on period commit

store the new master zone's sync status with each new period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

make: fix link of ceph_test_cls_rgw_meta

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add rgw_rest_realm.h to the Makefile

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add missing rgw_meta_sync_status.h to the Makefile

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: account total number of metadata entries for sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add backoff and restart to meta sync

also fix lease cr lifecycle

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix the mdlog sync lock control flow

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use set_cr_error() and set_cr_done() instead of set_state()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: md full sync backoff

Still need to work this out correctly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create a realm in case of pervious regions configuration

The realm name is concatanation of master region and master zone and its
id is the mds sume of the name

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add support for region format in RGWOp_ZoneGroupMap_Get

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: update current_period after adding converted zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add RGWRegionMap for backward compatability

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: update realm_id when converting a region to zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add converted regions into the current period

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWRealmWatcher to reload RGWRados on changes

adds RGWRealmWatcher, which watches the RGWRealm control object for
changes.  in a separate thread, all frontends are paused while the
RGWRados object is replaced with a new one

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period commit calls notify_zone

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add realm control object for watch/notify

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add pause/unpause to RGWFrontend interface

RGWProcessFrontend implements pause/unpause through the ThreadPool
interface. RGWMongooseFrontend implements them with a reader/writer
lock that blocks the civetweb_callback() until unpause

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rwlock: add prioritize_write flag to prevent write starvation

RWLock's constructor takes an additional 'prioritize_write' flag, which
defaults to false.  when enabled, this adds the rwlock attribute
PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP, which will block calls to
pthread_rwlock_rdlock() if there are any writers waiting.  this avoids
starvation of pthread_rwlock_wrlock() in read-mostly workloads

also made the destructor non-virtual; this was adding a virtual function
table to the class, even though there are no other virtual functions.
added the 'final' keyword to document that it's not meant to be derived
from

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWHTTPManager dtor stops background thread

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: remove period activate in favor of commit

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add --commit flag for period update

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add 'period commit' command

on the master zone, store the new period locally and notify the zone for
restart.  on other zones, send a 'period push' to the master zone with
an empty period id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period push handler supports commit

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWPeriod::commit()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period push handler updates current period/epoch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWPeriod accessors are const

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add realm pull command

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add send_to_remote_or_url() helper

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add op for GET /admin/realm

this is for the radosgw-admin 'realm pull' command, which gets the realm
object by name or id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add realm params to period pull

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move period Ops to rgw_rest_realm.cc

this removes the handling of /admin/config?type=period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period allows setting id, epoch, latest_epoch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: omit epoch in oid for staging period

there should only be one staging period object, so the epoch is
an unnecessary part of its oid. and for period commit, we need to
remember the epoch of the predecessor period to make sure we're not
based on an old epoch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add 'period update' to usage

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRemoteMetaLog ctor sets async_rados=nullptr

RGWRemoteMetaLog::init() is not called in all error paths, so we need to
start with nullptr

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix comparison typo in RGWRados::init_complete

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWHandler_Config should assume empty type is zonegroupmap

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: guard metadata full/incremental sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: guard metadata sync initialization step

Guard the fetch-meta stage

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a bit better coroutine related logging

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initialize rest connection if no period

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: multiple init metadata sync locking fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: backward compatability update_name for existing zones

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: update new name after converting region to zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: fix zone get command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use default zone in case there is no period or zone name

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWRados zonegroup and zone_public_config should not be accessed directly

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWRados internal zone should be protected and not accessed directly

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix RGWSystemMetaObj::init by passing old_format flag

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: convert regionmap

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add support to zonegroupmap set

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zonegroupmap get should no error if no realm or period

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: no need to store or update zonegroupmap

Structure and command will be kept for backward compatability

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove zonegroup_map from RGWRados

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cmake: move link dependencies to rgw_a

Signed-off-by: Casey Bodley <cbodley@redhat.com>

CMake - add check for libcurl

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

rgw: don't read/init sync status if meta master

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix log message

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: can set zone read-only, can set zonegroup id

and other fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update rest connections from the current period

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: read local zonegroup and zone only when needed

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use current_period zonegroup and zone when exists

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: read current period during init

Signed-off-by: Orit Wasserman <owasserm@redhat.com>