rgw: don't require system pools to start with a period

Make sure cache operations are being done on system objects.
Separate calls where needed to make sure that separate pool/bucket
handling is done correctly. When separating calls, don't use objv_tracker
for data objects, don't use data constructs for system objects.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: rgw_bi_log_entry::decode_json(), decode ver

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate olh_epoch for obj sync correctly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: reset dirent flag appropriately

Need to mark dirent as versioned if it has instance.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch_remote_obj() uses correct version_id

This fixes one aspect of the versioned objects sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle object removal in incremental data sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: tie incremental data sync

and other fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync locking fixes

and other fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: minor cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't create connection object for zone without endpoints

warn about it though

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: async data changes notifications

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: work towards datalog sync notification

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: eliminate window for data sync marker

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: only account actual io for coroutine io window

stacks that are put in interval_wait shouldn't be accounted.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: incremental data sync adjustments

update marker, close a race

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: incremental data sync

still needs some more work, but basic structure is there.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix data sync init

Need to wait for shards' status.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add rwlock to RGWObjectCtx

It can be used from multiple threads now.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reduce the use of obj_ctx

Should really get rid of all the obj_ctx use, unless it's really needed. It's
not thread safe.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: revive io window

and several other fixes. Don't send unbounded number of IO requests,
send back http related errors (that didn't have a specific http status).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync, spawn sync on all shards

full sync should work now

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move top level data sync function into coroutine

So that it can be called from a coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync work

Instead of just keeping map of bucket instances, keep map of all the shards.
Also prepare the CRs that will call into the bucket sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: multiple fixes related to coroutines draining

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Revert "rgw: simplify unblocked by stack state change"

This reverts commit 8509ef5034818e163bebc8fd2de039a96b4a8f1a.

rgw: clean up child draining a bit

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: simplify unblocked by stack state change

If blocked by another stack that is finished, just schedule, not waiting
on IO.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: finish execution when done with bucket sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add RGWCoroutine::wait_for_child()

Will yield until one of the spawned stacks finished execution.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: no need to yield while collecting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix pending operations accounting

create RGWCoroutine::num_pending() to get the number of
pending spawned coroutines.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: collect spawned requests before completing coroutine

Need to finish all processing before we return

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: url_encode http params in requests that we send

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: skeleton for bucket index incremental sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add rgw_bi_log_entry::decode_json

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: init shard_id param for bucket listing

shard_id can now be specified in the bucket instance http param

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix shard_id use

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move bucket shard full sync to its own coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch sync state when done with full bucket sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix iteration over bucket entries in full sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: silence some verbose logs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: bucket sync commands don't require bucket id

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep bucket sync markers in object attrs

This allows to set the status separately for each of the
different stages (full, incremental), while allowing also
to set the attributes atomically.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: can read attrs when reading system obj

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: window of full sync operations per bucket

also store marker periodically, reuse marker logic from md sync.
Marker still needs some work.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api adjustment following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix identification of canceled operation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync fetches remote objects

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync, send marker when listing bucket shard

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: extend list bucket versions api

System user can now provide 'objs-container=true' in conjuction
with 'format=json' so that response is json and entries are
encoded correctly as array.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix signing for generated requests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: list bucket instance shard entries for data sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read bucket sync status

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: per-bucket instance shard state

add 'radosgw-admin bucket sync init' command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial work on data sync run

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: implement data sync init

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: data sync status command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial data sync work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: skeleton for data sync commands

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: switch mdlog sync commands to metadata sync

radosgw-admin mdlog sync init -> metadata sync init
etc.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

build: add cls_timeindex_client to DENCODER_DEPS

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: propagate rgwx-bucket-instance for copying objects

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix time parsing for delete precondition

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add no-precondition-error http param for object delete

In case we don't want to get an error response when a precondition
failed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add mtime check to object removal if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: add an objclass method to check obj mtime

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add DELETE_IF_UNMODIFIED_SINCE header to obj delete

Will delete object only if not modified after the specified time. Still
need to close a race where object is modified after the check.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more api adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adjust calls

as part of  a rebase work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: identify racing writes when using copy-if-newer

When copying an object from a different zone, and copy-if-newer is
specified, if the final meta write is canceled check whether the
destinatioin that was created is actually newer than our mtime,
otherwise retry.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: translate http error 304 to appropriate status

So that we can return the appropriate response on a conditioinal multi
zone copy operation.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix if-modified-since boundary

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add rgwx-copy-if-newer http header

Similar to IF_MOD_SINCE, but does not take a time argument, and should
protected against races (not there yet).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: make rgw_user::empty() const

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: access zone.name, zonegroup_name through accessor

fix as part of a rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: dump internal object tag when system user lists bucket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: system user can list bucket shard

By setting the HTTP-Rgwx-Shard-Id header to the shard id.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: get_bucket_stats() returns a plain marker if shard specified

E.g, will return 00001234 vs. 2#00001234 if shard 2 specified.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bilog info can get shard id

We would like to be able to get bilog info by shard if needed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix uninitialized variable

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move more code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trivial fix

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use internal wait instead of special coroutine

Instead of using a special coroutine that spawns a separate thread
and waits on a condition, use the internal coroutine stack wait.
This is useful since it since we limit the number of IO threads
we spawn.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutine stack wait util

Instead of using a separate coroutine for waiting on
a condition.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move more code around

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move code around

move code from rgw_sync.cc to rgw_cr_rados.{h,cc}

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rearrange full metadata sync order

Users need to exist before their buckets, better have bucket instances
before bucket entry points

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trivial fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix shutdown

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http client req handling fixes

The error buffer for the async requests was on the stack, and not
on the request structure. Some other use after free and locking fixes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: wake up appropriate shard sync handler

When receiving notification about modified shard, wake up the appropriate handler

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a few sync thread init/finalization fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: run and stop sync thread

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: simple thread base class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: decode mdlog notification shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: send mdlog changes notifications to zones

Will probably need to rework this, as it is now zones may interfere
with each other which is not optimal.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add simple POST requests interfaces

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: add json encoder / decoder for std::set

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: periodical thread that tracks mdlog changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle mdlog post request

with 'notify' subresource. Will be used later for handling notifications
about modified mdlog shards.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>