cls_statelog: add client api functions

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: fixes and changes to api

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: introducing new objclass to handle state tracking

Somewhat similar to the log objclass, but uses different data for
indexing. Also keeps a dual index. In general an entry has 3
identifiers:
 - object: the object id on which the operation is made
 - client_id: client's unique identifier
 - op_id: operation's unique identifier

An entry is indexed by both client_id+op_id, and by
object+op_id, make it possible to list operations
either by client_id, or by object id.
We also keep state per each entry and the new check_state request
can be used as a guard.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: object mtime the same for both object and bucket index

Make sure object mtime that's passed to bucket index is the
same as the one set on the object.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix inter-region copy (ofs wasn't correct)

Fixing an issue with data read. Since inter-region copy uses
embedded metadata within the data input, we need to fix the ofs
accordingly.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: intra-region copy, preserve mtime

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: system user get obj returns JSON encoded metadata

instead of binary encoded metadata

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: get / set user & bucket meta mtime

also update infrastructure to be able to set mtime on objects

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: preserve bucket creation time across different zones / regions

Keep bucket creation time in RGWBucketInfo, and make use
of it where necessary.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: show mtime for metadata entries

currently only shows for buckets

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fixes for intra-zone object copy

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: set bucket marker and bucket obj version from master

Make sure that bucket that's created gets the same marker as
the one that the master region generated. Also, for some reason
we weren't passing in the metadata obj version for the bucket.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: multiple changes for intra-zone object copy

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: when setting bucket metadata key, don't override placement

Bucket placement also contains data and index pools locations in
the source zone. However, we might be applying it on a different
zone, so we need to be careful not to use the source location here.
We generate a new placement if a new bucket is detected, otherwise
we keep the old one.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: forward amz headers of copy request to remote rgw

So that if request has any special conditions, it'll be processed
on the remote gateway

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: cleanup, move init_meta_info() into req_info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: remove unused variable

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket marker contains zone name

(issue 5347)
We need this so that raw object names on different zones within
the same region don't collide.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: pass original object attrs through extra request data

introduce a new mechanism that sends extra data with object
info, and use it to encode source object metadata.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: a few fixes to copy object across region

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: move most of object put processors implementation

move code from rgw_op.cc to rgw_rados.cc, so that we could
use it in the lower layer.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: separate req_state from object put processors

At least, for most of them. We'll move this code to rgw_rados.cc,
and we don't want it to have req_state.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: organize get_obj handling for copy a bit different

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: generate read request if source rgw is remote

in a copy operation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: propagate mtime from remote rgw on copy

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: propagate error from remote gateway when copying object

Also make sure that we don't continue iterating locally through
the object.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: pass grant headers to target when writeing object

When writing object to remote gateway.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: head acls target can be quoted

when passing x-amz-grant-* headers, the target (id, email,
group) may be quoted.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: send meta headers with remote PUT request

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: don't busy wait for outgoing rest requests

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: don't send redirect if copy object has a local source

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: link bucket to owner on metadata update

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix logic related to operations on different region

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: minor logging change

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix json decoding for swift keys

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: stream obj into http request

still need to figure out curl handle polling, handle client
errors correctly.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: format put request auth

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket metadata, ignore ENOENT on put

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: skeleton for obj copy across regions

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata lock/unlock implemented with test cases

metadata lock/unlock implemented with test cases
split Post operations to Lock and Unlock operations, and
checking for WRITE permission only for lock and unlock

Signed-off-by: Babu Shanmugam <anbu@enovance.com>
Conflicts:
src/rgw/rgw_rest_log.cc
src/rgw/rgw_rest_log.h

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

RESTful APIs for data changes log implemented with test cases

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: some user metadata fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw_admin: modify mdlog functionality

mdlog list, can handle cases where there is and there's no shard id
specified. mdlog trim requires shard if.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: set truncated as false if ENOENT when listing mdlog entries

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge branch 'wip-rgw-geo-enovance' into wip-rgw-geo-2

Conflicts:
src/Makefile.am
src/rgw/rgw_admin.cc

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: Do not assum rest connection to be established

Added a check to verify that rest connection is correctly initialized,
preventing a segmentation fault.

Signed-off-by: Christophe Courtaut <christophe.courtaut@gmail.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

flushed the formatter after closing the primary section in MDLOG, BILOG and GETSHARDINFO :: list()

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: verify bucket location only if exists

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: forward delete_bucket to master region

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Fixed default region creation issue

If the default region did not exist, it is created,
but the previously created RGWDefaultRegionInfo object is not updated.
This fix updates this objects after the default region creation.
It was also preventing vstart.sh script to start with -r option.

Signed-off-by: Christophe Courtaut <christophe.courtaut@gmail.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: user master region's version of bucket

When creating a bucket, retrieve the bucket object
version from the master region and use it.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: mdlog, bilog RESTful api cleanup

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

With mdlog lock and unlock functionality, listing and trimming base on shard_id

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: propagate region on bucket creation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: determine owner after effetctive user

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: forward request params through upstream REST

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: enable data sending via http client

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: multiple fixes and cleanups

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

utime: add asctime()

dump time in a useful format

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: rest_client, forward requests

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: some code cleanup

move s3 signing code to a different common file.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: separate certain request info into different structure

This is needed for later request forwarding.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: clean up s3 signing code

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add system users

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: pre-generate bucket object version before creating it

Use this version when creating the bucket instead of letting the
objclass auto-generate it, so that we could return it to the slave
region when creating its bucket on the master.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: forward bucket creation to master region

plus many fixes.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: infrastructure to send requests to different region

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Added send_response implementation for MDLOG and BILOG list APIs so that periodic flushing can happend in execute()

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: add access key to zone info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

MDLOG and BILOG REST Apis implemented along with some bug fixes

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: store region in bucket info

only handle requests that come to buckets stored in correct
region.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: parse location constraint on bucket creation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: a function to read all request input

Factor out this useful function. Also make sure that
we never read more than a specified (large enough) max.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: update json encode/decode for new bucket info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add region to bucket info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Modified PutMetadata::get_data() to handle chunked transfers

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

ceph_json: don't try to parse NULL buffer

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Removed the check for parameter validation in op_get()

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

Fixed certain bugs on rest admin APIs

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: modify metadata RESTful implementation

REST handler should derive from RGWHandler_Auth_S3,
other changes.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

With admin metadata rest API implementation and unit test cases for it

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

ceph_json: fix bool decoding

"false" means false.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: json_encode json a bit differently

Encode map as a list, it's a more friendly representation.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: tie bucket/user removal to mdlog differently

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: slightly simplify metadata abstraction

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: read bucket metadata before writing it

In order to keep track of version.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: cache obj version

Also keep bucket objv_tracker on the request state.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: op->PutACLs uses the correct set_attr for buckets

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: rados->set_attr() just calls rados->set_attrs()

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata handler for bucket set_attr operations

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: don't handle ECANCELLED anymore

Simplify, remove obsolete logic.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket metadata operations go through metadata handler

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

armor: don't break lines by default

Added a new function that breaks the lines, but by default
don't do it.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: implement metadata hander for buckets data

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

ceph_json: dump timestamp in utc

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: datalog trim

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket index log trim

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

osd: don't assert if get_omap_iterator() returns NULL

Fixes: #4949
This can happen if the object does not exist and it's
a write operation. Just return -ENOENT.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata log trim

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: resend data log entry if took too long

If took too long, we want to resend. Also, fix issue with
renew.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>