radosgw-admin: update zonegroupmap when adding a zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use encode_map for zonegroups

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix zonegroup create

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix realm create

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix init_complete when there is no realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: fix error messages for zonegroup create

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix zonegroup update

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: update zonegroupmap when creating new realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add period_map to encode/decode

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add zonegroup_map period update

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add zonegroupmap realm update

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix encoding and decoding of realms and periods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add realms to zonegroupmap

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: zonegroup map contains a map of periods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: move zonegroup data into rgwperiodmap

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: cleanup period init

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use RGWZoneGroupMap in RGWPeriod

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add use_next_epoch method

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add zone create, delete and rename commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWZoneParams should inherit from RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWZoneGroup should inheirt from RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cmake: fix erasure-code

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Rename Region to ZoneGroup only internal structures and class

User visuable are not changed: admin commands, jsons and rados objects names

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_admin.cc
src/rgw/rgw_rados.cc

rgw: Add period admin commands and rest api to handle period

the new commands are prepare,delete, get, activate, pull and push

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_admin.cc

rgw: Add RGWSystemMetaObj class

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_rados.cc

rgw: Add RGWNameToId and RGWDefaultSystemMetaObjInfo

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: async rados requests through a work queue

Tie in the async rados requests to the async framework.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add new class to create resource request

RGWRESTReadResource holds the request input and output. Previously
we were using stack variables that couldn't be used in the async
path.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't require system pools to start with a period

Make sure cache operations are being done on system objects.
Separate calls where needed to make sure that separate pool/bucket
handling is done correctly. When separating calls, don't use objv_tracker
for data objects, don't use data constructs for system objects.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: rgw_bi_log_entry::decode_json(), decode ver

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate olh_epoch for obj sync correctly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: reset dirent flag appropriately

Need to mark dirent as versioned if it has instance.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch_remote_obj() uses correct version_id

This fixes one aspect of the versioned objects sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle object removal in incremental data sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: tie incremental data sync

and other fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync locking fixes

and other fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: minor cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't create connection object for zone without endpoints

warn about it though

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: async data changes notifications

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: work towards datalog sync notification

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: eliminate window for data sync marker

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: only account actual io for coroutine io window

stacks that are put in interval_wait shouldn't be accounted.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: incremental data sync adjustments

update marker, close a race

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: incremental data sync

still needs some more work, but basic structure is there.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix data sync init

Need to wait for shards' status.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add rwlock to RGWObjectCtx

It can be used from multiple threads now.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reduce the use of obj_ctx

Should really get rid of all the obj_ctx use, unless it's really needed. It's
not thread safe.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: revive io window

and several other fixes. Don't send unbounded number of IO requests,
send back http related errors (that didn't have a specific http status).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, spawn sync on all shards

full sync should work now

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move top level data sync function into coroutine

So that it can be called from a coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync work

Instead of just keeping map of bucket instances, keep map of all the shards.
Also prepare the CRs that will call into the bucket sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: multiple fixes related to coroutines draining

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Revert "rgw: simplify unblocked by stack state change"

This reverts commit 8509ef5034818e163bebc8fd2de039a96b4a8f1a.

rgw: clean up child draining a bit

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: simplify unblocked by stack state change

If blocked by another stack that is finished, just schedule, not waiting
on IO.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: finish execution when done with bucket sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add RGWCoroutine::wait_for_child()

Will yield until one of the spawned stacks finished execution.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: no need to yield while collecting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix pending operations accounting

create RGWCoroutine::num_pending() to get the number of
pending spawned coroutines.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: collect spawned requests before completing coroutine

Need to finish all processing before we return

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: url_encode http params in requests that we send

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: skeleton for bucket index incremental sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add rgw_bi_log_entry::decode_json

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: init shard_id param for bucket listing

shard_id can now be specified in the bucket instance http param

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix shard_id use

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move bucket shard full sync to its own coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch sync state when done with full bucket sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix iteration over bucket entries in full sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: silence some verbose logs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: bucket sync commands don't require bucket id

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep bucket sync markers in object attrs

This allows to set the status separately for each of the
different stages (full, incremental), while allowing also
to set the attributes atomically.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: can read attrs when reading system obj

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: window of full sync operations per bucket

also store marker periodically, reuse marker logic from md sync.
Marker still needs some work.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api adjustment following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix identification of canceled operation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync fetches remote objects

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync, send marker when listing bucket shard

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: extend list bucket versions api

System user can now provide 'objs-container=true' in conjuction
with 'format=json' so that response is json and entries are
encoded correctly as array.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix signing for generated requests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: list bucket instance shard entries for data sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read bucket sync status

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: per-bucket instance shard state

add 'radosgw-admin bucket sync init' command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial work on data sync run

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: implement data sync init

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: data sync status command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial data sync work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: skeleton for data sync commands

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: switch mdlog sync commands to metadata sync

radosgw-admin mdlog sync init -> metadata sync init
etc.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

build: add cls_timeindex_client to DENCODER_DEPS

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: propagate rgwx-bucket-instance for copying objects

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix time parsing for delete precondition

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add no-precondition-error http param for object delete

In case we don't want to get an error response when a precondition
failed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add mtime check to object removal if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: add an objclass method to check obj mtime

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add DELETE_IF_UNMODIFIED_SINCE header to obj delete

Will delete object only if not modified after the specified time. Still
need to close a race where object is modified after the check.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more api adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adjust calls

as part of  a rebase work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: identify racing writes when using copy-if-newer

When copying an object from a different zone, and copy-if-newer is
specified, if the final meta write is canceled check whether the
destinatioin that was created is actually newer than our mtime,
otherwise retry.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: translate http error 304 to appropriate status

So that we can return the appropriate response on a conditioinal multi
zone copy operation.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix if-modified-since boundary

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>