rgw: stream obj into http request

still need to figure out curl handle polling, handle client
errors correctly.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: format put request auth

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket metadata, ignore ENOENT on put

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: skeleton for obj copy across regions

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata lock/unlock implemented with test cases

metadata lock/unlock implemented with test cases
split Post operations to Lock and Unlock operations, and
checking for WRITE permission only for lock and unlock

Signed-off-by: Babu Shanmugam <anbu@enovance.com>
Conflicts:
src/rgw/rgw_rest_log.cc
src/rgw/rgw_rest_log.h

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

RESTful APIs for data changes log implemented with test cases

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: some user metadata fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw_admin: modify mdlog functionality

mdlog list, can handle cases where there is and there's no shard id
specified. mdlog trim requires shard if.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: set truncated as false if ENOENT when listing mdlog entries

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge branch 'wip-rgw-geo-enovance' into wip-rgw-geo-2

Conflicts:
src/Makefile.am
src/rgw/rgw_admin.cc

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: Do not assum rest connection to be established

Added a check to verify that rest connection is correctly initialized,
preventing a segmentation fault.

Signed-off-by: Christophe Courtaut <christophe.courtaut@gmail.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

flushed the formatter after closing the primary section in MDLOG, BILOG and GETSHARDINFO :: list()

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: verify bucket location only if exists

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: forward delete_bucket to master region

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Fixed default region creation issue

If the default region did not exist, it is created,
but the previously created RGWDefaultRegionInfo object is not updated.
This fix updates this objects after the default region creation.
It was also preventing vstart.sh script to start with -r option.

Signed-off-by: Christophe Courtaut <christophe.courtaut@gmail.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: user master region's version of bucket

When creating a bucket, retrieve the bucket object
version from the master region and use it.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: mdlog, bilog RESTful api cleanup

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

With mdlog lock and unlock functionality, listing and trimming base on shard_id

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: propagate region on bucket creation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: determine owner after effetctive user

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: forward request params through upstream REST

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: enable data sending via http client

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: multiple fixes and cleanups

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

utime: add asctime()

dump time in a useful format

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: rest_client, forward requests

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: some code cleanup

move s3 signing code to a different common file.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: separate certain request info into different structure

This is needed for later request forwarding.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: clean up s3 signing code

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add system users

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: pre-generate bucket object version before creating it

Use this version when creating the bucket instead of letting the
objclass auto-generate it, so that we could return it to the slave
region when creating its bucket on the master.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: forward bucket creation to master region

plus many fixes.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: infrastructure to send requests to different region

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Added send_response implementation for MDLOG and BILOG list APIs so that periodic flushing can happend in execute()

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: add access key to zone info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

MDLOG and BILOG REST Apis implemented along with some bug fixes

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: store region in bucket info

only handle requests that come to buckets stored in correct
region.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: parse location constraint on bucket creation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: a function to read all request input

Factor out this useful function. Also make sure that
we never read more than a specified (large enough) max.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: update json encode/decode for new bucket info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add region to bucket info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Modified PutMetadata::get_data() to handle chunked transfers

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

ceph_json: don't try to parse NULL buffer

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Removed the check for parameter validation in op_get()

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

Fixed certain bugs on rest admin APIs

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: modify metadata RESTful implementation

REST handler should derive from RGWHandler_Auth_S3,
other changes.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

With admin metadata rest API implementation and unit test cases for it

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

ceph_json: fix bool decoding

"false" means false.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: json_encode json a bit differently

Encode map as a list, it's a more friendly representation.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: tie bucket/user removal to mdlog differently

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: slightly simplify metadata abstraction

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: read bucket metadata before writing it

In order to keep track of version.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: cache obj version

Also keep bucket objv_tracker on the request state.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: op->PutACLs uses the correct set_attr for buckets

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: rados->set_attr() just calls rados->set_attrs()

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata handler for bucket set_attr operations

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: don't handle ECANCELLED anymore

Simplify, remove obsolete logic.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket metadata operations go through metadata handler

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

armor: don't break lines by default

Added a new function that breaks the lines, but by default
don't do it.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: implement metadata hander for buckets data

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

ceph_json: dump timestamp in utc

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: datalog trim

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket index log trim

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

osd: don't assert if get_omap_iterator() returns NULL

Fixes: #4949
This can happen if the object does not exist and it's
a write operation. Just return -ENOENT.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata log trim

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: resend data log entry if took too long

If took too long, we want to resend. Also, fix issue with
renew.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: more data changes log implementation

Remove a bunch of hard coded stuff. Make renew thread updates
expiration. Only renew if there was more than one request through the
current window.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

RefCounteCond: keep return val, wait() returns it

It is necessary in some cases to notify waiters of the
actual return value for the action they were waiting on.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: changed data log renew thread

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: limit num of buckets in data changes log

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: use shared_ptr instead of RefCountedObject

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

lru_map: infrastructure for a bounded map

Useful for cache-like maps, where we want to control
the max number of entries in the map.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: data changes log, don't always send new requests

We may piggy back on older entries that hasn't expired yet.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: data changes log, naive implementation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: decouple bucket data pool from bucket index pool

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix get_obj() with zero sized obj

Now that even zero sized objs have manifest a
test had to be modified.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: don't set shadow obj attr in object metadata

This is incorrect, was only right for pre-manifest objects.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: object stat also decodes policy

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: update cli test for radosgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Makefile.am: add missing header file

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: add object stat command

for retrieving object metadata.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: bucket list also specifies object namespace

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: copy_obj uses req_id as tag

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: radosgw-admin bucket list --bucket lists bucket objects

Also lists tag for each object.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: call rgw_store_user_info() with objv_tracker

another rebase casualty

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: fix some more merge issues

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: fix user_id initialization

broken due to rebase

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: share object tag and index tag

object tag is now being written to the index, so that both
object and index hold the same tag. This is needed so that we
could know whether object and index refer to the same instance.

Also cleanup old legacy code that did atomic ops through cloning
objects.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket index log fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin: bilog list gets marker and max-entries params

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin, cls_rgw: list bucket index log

a new radosgw-admin command to list bucket index log.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_rgw: bucket index versioning

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix broken radosgw-admin user * commands

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_rgw, rgw: bucket index logs modifications

Add a log to the bucket index.

This commit also ties the "epoch" version that is kept
per index entry to the relevant ceph pool id. This fixes
the dependency on the specific pool where the object was
created, and makes it possible to copy rgw data pools.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata rm

Still needs to fix the way we remove user entries; need to take
different path, similar to put_entry())

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: cls_log_trim boundary change

end_time is not inclusive

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: show metadata log through radosgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: use new cls_log listing interface

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: adjust listing api

Listing api now also gets end time and a marker.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: log user operations

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: track object versions between reads and writes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>