rgw: add zonegroup modify command to allow setting/clearing zonegroup master status

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: fix zonegroup and master usage text

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove is_master flag from zoneparams

make it easier to change master zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: print zone name and id in the begining of the dump

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: fix zone create to check for zonegroup name

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: store latest sync status when activating a period

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add decode_json method to rgw_meta_sync_status structures

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: move rgw_meta_sync_status, rgw_meta_sync_marker and rgw_meta_sync_info into a sperate h file

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: move dump functions into rgw_json_enc.cc

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: zone config related changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix compilation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't try to unregister request on destructor

We already unregister the request earlier, and the locking here
was weird and wrong anyway.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add bucket_quota to RGWRegionMap::encode_json

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add period_map to period dump and encode_json

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix whitespace and remove debug messages

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add backward support for old regionmap format

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add RGWRegionMap for backward support

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove debug message from decode_zones

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: remove extra print from zonegroup set command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: rgwzone decode_json should handle empty id for backward compatabiltiy

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: systemmetaobj::create don't generate random id if id is provided

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone set should handle old format json (without name or id)

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add id to RGWZone dump and decode_json functions

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone create should update zonegroupmap for non standalone zones

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: fix zonegroup default

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add --rgw-zonegroup for backward compatability

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zone set command should update zonegroupmap

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: realm set should update zonegroup map

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: zonegroup set needs to update zonegroupmap

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: fix zonegroupmap update

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: backward compatabilty for zone set command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add --rgw-zone for backward compatability

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: update zonegroupmap when activating a period

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix zonegroupmap initialization in init_complete

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use realm provided in the command line

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: use the correct realm id when creating a new zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: use zonegroup realm id in zone add command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add realm_id to zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: update zonegroupmap when adding a zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use encode_map for zonegroups

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix zonegroup create

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix realm create

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix init_complete when there is no realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: fix error messages for zonegroup create

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix zonegroup update

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: update zonegroupmap when creating new realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add period_map to encode/decode

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add zonegroup_map period update

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add zonegroupmap realm update

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix encoding and decoding of realms and periods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add realms to zonegroupmap

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: zonegroup map contains a map of periods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: move zonegroup data into rgwperiodmap

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: cleanup period init

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use RGWZoneGroupMap in RGWPeriod

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add use_next_epoch method

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add zone create, delete and rename commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWZoneParams should inherit from RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWZoneGroup should inheirt from RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cmake: fix erasure-code

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Rename Region to ZoneGroup only internal structures and class

User visuable are not changed: admin commands, jsons and rados objects names

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_admin.cc
src/rgw/rgw_rados.cc

rgw: Add period admin commands and rest api to handle period

the new commands are prepare,delete, get, activate, pull and push

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_admin.cc

rgw: Add RGWSystemMetaObj class

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_rados.cc

rgw: Add RGWNameToId and RGWDefaultSystemMetaObjInfo

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: async rados requests through a work queue

Tie in the async rados requests to the async framework.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add new class to create resource request

RGWRESTReadResource holds the request input and output. Previously
we were using stack variables that couldn't be used in the async
path.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't require system pools to start with a period

Make sure cache operations are being done on system objects.
Separate calls where needed to make sure that separate pool/bucket
handling is done correctly. When separating calls, don't use objv_tracker
for data objects, don't use data constructs for system objects.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: rgw_bi_log_entry::decode_json(), decode ver

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate olh_epoch for obj sync correctly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: reset dirent flag appropriately

Need to mark dirent as versioned if it has instance.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch_remote_obj() uses correct version_id

This fixes one aspect of the versioned objects sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle object removal in incremental data sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: tie incremental data sync

and other fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync locking fixes

and other fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: minor cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't create connection object for zone without endpoints

warn about it though

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: async data changes notifications

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: work towards datalog sync notification

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: eliminate window for data sync marker

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: only account actual io for coroutine io window

stacks that are put in interval_wait shouldn't be accounted.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: incremental data sync adjustments

update marker, close a race

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: incremental data sync

still needs some more work, but basic structure is there.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix data sync init

Need to wait for shards' status.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add rwlock to RGWObjectCtx

It can be used from multiple threads now.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reduce the use of obj_ctx

Should really get rid of all the obj_ctx use, unless it's really needed. It's
not thread safe.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: revive io window

and several other fixes. Don't send unbounded number of IO requests,
send back http related errors (that didn't have a specific http status).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, spawn sync on all shards

full sync should work now

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move top level data sync function into coroutine

So that it can be called from a coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync work

Instead of just keeping map of bucket instances, keep map of all the shards.
Also prepare the CRs that will call into the bucket sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: multiple fixes related to coroutines draining

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Revert "rgw: simplify unblocked by stack state change"

This reverts commit 8509ef5034818e163bebc8fd2de039a96b4a8f1a.

rgw: clean up child draining a bit

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: simplify unblocked by stack state change

If blocked by another stack that is finished, just schedule, not waiting
on IO.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: finish execution when done with bucket sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add RGWCoroutine::wait_for_child()

Will yield until one of the spawned stacks finished execution.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: no need to yield while collecting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix pending operations accounting

create RGWCoroutine::num_pending() to get the number of
pending spawned coroutines.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: collect spawned requests before completing coroutine

Need to finish all processing before we return

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: url_encode http params in requests that we send

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: skeleton for bucket index incremental sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>