radosgw-admin: add realm set command

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add json output to create commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix period init and activate

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: update period predecessor when it is activated

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix realm list-periods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: realm id and epoch should be part of period oid

fixes period get and activate command
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add realm list-periods command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: Add master_zonegroup and master_zone to period prepare

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: Add period list command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: print new perion in period perpare command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix zone set command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove debug messages

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: default zone should not be master

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: add zone create, delete and rename commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWZoneParams should inherit from RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use id for master zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add id to RGWZoneParams

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add RGWSystemMetaObj class

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_rados.cc

rgw-admin: adjustments to period get-current output

Make it output through the formatter

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: use default realm on realm get

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pass const string reference

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: fix realms list

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix typo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: do the actual incremental metadata sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: track mdlog marker

sync entries if the sync marker is < mdlog marker, otherwise
try to fetch more entries.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't force wait on blocked coroutines

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix busy wait in RGWHTTPManager

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync local mdlog if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: wake up wait coroutine thread on destruction

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Add zone create admin command - wip

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add id to RGWZone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: poll mdlog for incremental sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read mdlog entries for incremental sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: get remote mdlog status when initializing

Use that as the lowerbound for the meta marker when we start
the incremental sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: hash full sync keys to shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch to incremental sync when done full meta sync

each shard need to collect the coroutines it spawned before continuing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update global sync state after building meta indexes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update meta sync markers during full sync operation

update marker per shard, keep a window so that we don't update after each change

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add zonegroup name or id to zonegroup get and set commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add zonegroup create, delete and rename admin commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add period activate admin command and fix get current command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add current_period to RGWRealm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: RGWZoneGroup should inheirt from RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add old_format support to RGWSystemMetaObj

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add id to zonegroup

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove region

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: change RGWZoneGroup::get_pool_name to return string

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: convert old region configuration to zonegroups during init

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

fix ceph-dencoder link

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cmake: add rgw_sync.cc and rgw_coroutine.cc

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix async sleep and wakeup logic

lot's of cleanups and fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: full meta sync

need to handle errors, and update markers, but full meta sync is functional

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: naively fetch all meta objects

doing it one by one, later will spawn these concurrently

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read remote meta coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read full sync indexes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: modify sync state, sync run uses state

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutie for omap_get_vals()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep track of spawned coroutines under the current op

instead of keeping a per-stack list of spawned coroutines, keep it
on the op that spawned it. Otherwise, we might spawn a cr, call a second
one that will spawn some crs, and when waiting for these to complete
it will also wait on the first one.
The list of spawned coroutines is inherited. Once a coroutine finishes
its parent will take over all the spawned ones that weren't collected.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: replace omap_set coroutine code

No need to use the async rados worker.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix the simple coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix async sleep and wakeup logic

lot's of cleanups and fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutines producer consumers

Create a new producer-consumer(s) abstraction for the coroutines
infrastructure.
Move environment info from the op into the stack. Stack can be
sent to sleep and awaken explicitly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: build index of all master zone meta keys

still wip, but getting there

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create a coroutine for fetching REST api objects

this simplifies the code that yanks all the metadata from the master

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch all metadata entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta sync cleanups

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove calls for synchronous rados ops

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: dump shard status on mdlog sync status

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initialize all shard markers

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: return error if any spawned op errored

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: iterate through all mdlog sync status shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unused code, cleanups

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adjust mdlog fetch to use new interfaces

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix init mdlog sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mdlog sync init

initialize mdlog sync state, uses the implicit yield scheme (from boost)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use boost for stackless coroutine flow

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add cookie to lock coroutines

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more lock/unlock coroutines work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: prepare rados lock operation to be used with coroutines

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: simple rgw metadata write coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move coroutines code

move the generic code to rgw_coroutine.{h,cc}

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: return retcode of failing coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rename async ops to coroutines

which they basically are

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fail if master zone tries to use meta sync

We'll change it later, but as it is now that's the way to handle it
gracefully.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: Fix init_complete

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: support old region configurations

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Remove region admin commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add zone group admin commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: zonegroup json encoding/decoding with backward compatability

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Rename Region to ZoneGroup only internal structures and class

User visuable are not changed: admin commands, jsons and rados objects names

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_admin.cc
src/rgw/rgw_rados.cc

rgw: Add period admin commands and rest api to handle period

the new commands are prepare,delete, get, activate, pull and push

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_admin.cc

rgw: Add RGWPeriod

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add RGWRealm class and admin commands to handle it

admin realm commands: create, get, delete, rename, set-default and get-default

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Use switch for raw storage opcodes

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use string for default realm

Conflicts:
src/rgw/rgw_rados.cc

rgw: Add RGWSystemMetaObj class

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_rados.cc

rgw: Add RGWNameToId and RGWDefaultSystemMetaObjInfo

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix test_rgw_admin_log.cc

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cmake: add rgw_sync.cc

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: async ops can generate new stacks for parallel execution

Can now have an op generating another op that will be called at
a different stack. The current op will halt execution until new
op finishes. We can call this multiple times so that multiple ops
will be executed concurrently. This can be useful in the case we
want to generate a single op that will read from multiple objects,
after it had read some meta information. So the root op will read
the meta information and will generate multiple request that will
execute concurrently (albeit on a single thread).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>