rgw: add period_map to period dump and encode_json

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix whitespace and remove debug messages

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add backward support for old regionmap format

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add RGWRegionMap for backward support

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove debug message from decode_zones

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: remove extra print from zonegroup set command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: rgwzone decode_json should handle empty id for backward compatabiltiy

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: systemmetaobj::create don't generate random id if id is provided

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone set should handle old format json (without name or id)

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add id to RGWZone dump and decode_json functions

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone create should update zonegroupmap for non standalone zones

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: fix zonegroup default

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add --rgw-zonegroup for backward compatability

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone set command should update zonegroupmap

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: realm set should update zonegroup map

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zonegroup set needs to update zonegroupmap

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: fix zonegroupmap update

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: backward compatabilty for zone set command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add --rgw-zone for backward compatability

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: update zonegroupmap when activating a period

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix zonegroupmap initialization in init_complete

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use realm provided in the command line

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: use the correct realm id when creating a new zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: use zonegroup realm id in zone add command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add realm_id to zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: update zonegroupmap when adding a zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use encode_map for zonegroups

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix zonegroup create

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix realm create

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix init_complete when there is no realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: fix error messages for zonegroup create

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix zonegroup update

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: update zonegroupmap when creating new realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add period_map to encode/decode

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add zonegroup_map period update

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add zonegroupmap realm update

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix encoding and decoding of realms and periods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add realms to zonegroupmap

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: zonegroup map contains a map of periods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: move zonegroup data into rgwperiodmap

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: cleanup period init

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use RGWZoneGroupMap in RGWPeriod

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add use_next_epoch method

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add zone create, delete and rename commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWZoneParams should inherit from RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWZoneGroup should inheirt from RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cmake: fix erasure-code

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Rename Region to ZoneGroup only internal structures and class

User visuable are not changed: admin commands, jsons and rados objects names

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_admin.cc
src/rgw/rgw_rados.cc

rgw: Add period admin commands and rest api to handle period

the new commands are prepare,delete, get, activate, pull and push

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_admin.cc

rgw: Add RGWSystemMetaObj class

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_rados.cc

rgw: Add RGWNameToId and RGWDefaultSystemMetaObjInfo

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: async rados requests through a work queue

Tie in the async rados requests to the async framework.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add new class to create resource request

RGWRESTReadResource holds the request input and output. Previously
we were using stack variables that couldn't be used in the async
path.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't require system pools to start with a period

Make sure cache operations are being done on system objects.
Separate calls where needed to make sure that separate pool/bucket
handling is done correctly. When separating calls, don't use objv_tracker
for data objects, don't use data constructs for system objects.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: rgw_bi_log_entry::decode_json(), decode ver

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate olh_epoch for obj sync correctly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: reset dirent flag appropriately

Need to mark dirent as versioned if it has instance.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch_remote_obj() uses correct version_id

This fixes one aspect of the versioned objects sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle object removal in incremental data sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: tie incremental data sync

and other fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync locking fixes

and other fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: minor cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't create connection object for zone without endpoints

warn about it though

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: async data changes notifications

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: work towards datalog sync notification

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: eliminate window for data sync marker

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: only account actual io for coroutine io window

stacks that are put in interval_wait shouldn't be accounted.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: incremental data sync adjustments

update marker, close a race

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: incremental data sync

still needs some more work, but basic structure is there.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix data sync init

Need to wait for shards' status.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add rwlock to RGWObjectCtx

It can be used from multiple threads now.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reduce the use of obj_ctx

Should really get rid of all the obj_ctx use, unless it's really needed. It's
not thread safe.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: revive io window

and several other fixes. Don't send unbounded number of IO requests,
send back http related errors (that didn't have a specific http status).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, spawn sync on all shards

full sync should work now

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move top level data sync function into coroutine

So that it can be called from a coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync work

Instead of just keeping map of bucket instances, keep map of all the shards.
Also prepare the CRs that will call into the bucket sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: multiple fixes related to coroutines draining

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Revert "rgw: simplify unblocked by stack state change"

This reverts commit 8509ef5034818e163bebc8fd2de039a96b4a8f1a.

rgw: clean up child draining a bit

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: simplify unblocked by stack state change

If blocked by another stack that is finished, just schedule, not waiting
on IO.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: finish execution when done with bucket sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add RGWCoroutine::wait_for_child()

Will yield until one of the spawned stacks finished execution.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: no need to yield while collecting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix pending operations accounting

create RGWCoroutine::num_pending() to get the number of
pending spawned coroutines.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: collect spawned requests before completing coroutine

Need to finish all processing before we return

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: url_encode http params in requests that we send

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: skeleton for bucket index incremental sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add rgw_bi_log_entry::decode_json

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: init shard_id param for bucket listing

shard_id can now be specified in the bucket instance http param

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix shard_id use

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move bucket shard full sync to its own coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch sync state when done with full bucket sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix iteration over bucket entries in full sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: silence some verbose logs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: bucket sync commands don't require bucket id

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep bucket sync markers in object attrs

This allows to set the status separately for each of the
different stages (full, incremental), while allowing also
to set the attributes atomically.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: can read attrs when reading system obj

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: window of full sync operations per bucket

also store marker periodically, reuse marker logic from md sync.
Marker still needs some work.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api adjustment following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix identification of canceled operation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>