flushed the formatter after closing the primary section in MDLOG, BILOG and GETSHARDINFO :: list()

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: mdlog, bilog RESTful api cleanup

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

With mdlog lock and unlock functionality, listing and trimming base on shard_id

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

Added send_response implementation for MDLOG and BILOG list APIs so that periodic flushing can happend in execute()

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

MDLOG and BILOG REST Apis implemented along with some bug fixes

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: parse location constraint on bucket creation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: a function to read all request input

Factor out this useful function. Also make sure that
we never read more than a specified (large enough) max.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: update json encode/decode for new bucket info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add region to bucket info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Modified PutMetadata::get_data() to handle chunked transfers

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

ceph_json: don't try to parse NULL buffer

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Removed the check for parameter validation in op_get()

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

Fixed certain bugs on rest admin APIs

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: modify metadata RESTful implementation

REST handler should derive from RGWHandler_Auth_S3,
other changes.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

With admin metadata rest API implementation and unit test cases for it

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

ceph_json: fix bool decoding

"false" means false.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: json_encode json a bit differently

Encode map as a list, it's a more friendly representation.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: tie bucket/user removal to mdlog differently

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: slightly simplify metadata abstraction

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: read bucket metadata before writing it

In order to keep track of version.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: cache obj version

Also keep bucket objv_tracker on the request state.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: op->PutACLs uses the correct set_attr for buckets

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: rados->set_attr() just calls rados->set_attrs()

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata handler for bucket set_attr operations

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: don't handle ECANCELLED anymore

Simplify, remove obsolete logic.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket metadata operations go through metadata handler

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

armor: don't break lines by default

Added a new function that breaks the lines, but by default
don't do it.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: implement metadata hander for buckets data

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

ceph_json: dump timestamp in utc

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: datalog trim

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket index log trim

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

osd: don't assert if get_omap_iterator() returns NULL

Fixes: #4949
This can happen if the object does not exist and it's
a write operation. Just return -ENOENT.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata log trim

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: resend data log entry if took too long

If took too long, we want to resend. Also, fix issue with
renew.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: more data changes log implementation

Remove a bunch of hard coded stuff. Make renew thread updates
expiration. Only renew if there was more than one request through the
current window.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

RefCounteCond: keep return val, wait() returns it

It is necessary in some cases to notify waiters of the
actual return value for the action they were waiting on.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: changed data log renew thread

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: limit num of buckets in data changes log

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: use shared_ptr instead of RefCountedObject

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

lru_map: infrastructure for a bounded map

Useful for cache-like maps, where we want to control
the max number of entries in the map.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: data changes log, don't always send new requests

We may piggy back on older entries that hasn't expired yet.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: data changes log, naive implementation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: decouple bucket data pool from bucket index pool

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix get_obj() with zero sized obj

Now that even zero sized objs have manifest a
test had to be modified.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: don't set shadow obj attr in object metadata

This is incorrect, was only right for pre-manifest objects.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: object stat also decodes policy

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: update cli test for radosgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Makefile.am: add missing header file

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: add object stat command

for retrieving object metadata.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: bucket list also specifies object namespace

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: copy_obj uses req_id as tag

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: radosgw-admin bucket list --bucket lists bucket objects

Also lists tag for each object.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: call rgw_store_user_info() with objv_tracker

another rebase casualty

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: fix some more merge issues

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: fix user_id initialization

broken due to rebase

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: share object tag and index tag

object tag is now being written to the index, so that both
object and index hold the same tag. This is needed so that we
could know whether object and index refer to the same instance.

Also cleanup old legacy code that did atomic ops through cloning
objects.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket index log fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin: bilog list gets marker and max-entries params

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin, cls_rgw: list bucket index log

a new radosgw-admin command to list bucket index log.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_rgw: bucket index versioning

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix broken radosgw-admin user * commands

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_rgw, rgw: bucket index logs modifications

Add a log to the bucket index.

This commit also ties the "epoch" version that is kept
per index entry to the relevant ceph pool id. This fixes
the dependency on the specific pool where the object was
created, and makes it possible to copy rgw data pools.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata rm

Still needs to fix the way we remove user entries; need to take
different path, similar to put_entry())

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: cls_log_trim boundary change

end_time is not inclusive

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: show metadata log through radosgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: use new cls_log listing interface

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: adjust listing api

Listing api now also gets end time and a marker.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: log user operations

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: track object versions between reads and writes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: initialize meta_mgr earlier

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test_cls_log: remove warning

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: extend unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: more fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: fixes, other adjustments

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

objclass: provide new api for unique subop versioning

We need to be able to generate a unique identifier for
each subop. This can be useful e.g., if we want to keep multiple
omap data entries indexed by the same key. The unique id
is being generated by the current object id and the current
osd subop count. For write operations it's unique.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

osd: keep track of current osd subop num

This can be used later to generate unique subop versions
in compound write operations.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: trim works in chunks, returns -ENODATA when done

Also created a higher level interface that iterates until
done.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_log: a class to handle info sorted by timestamp

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata list user, only show uids

don't show unrelated object names

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: user metadata updates also key version

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add top level metadata handler

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: put metadata, other cleanups

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata manager, api to list keys

Also, implement key listing for user metadata.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata set/get infrastructure

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: rgw_get_system_obj() can return obj_version

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_version: add cls_version_read(ObjectReadOpeation&)

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

librados: add two more ObjectOperation::exec()

 - one that also gets out bufferlist and ret value pointer
 - one that gets a callback context

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: start tying metadata objs to version objclass

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_version: unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_version: various fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_version: create a new objclass

New objclass to track and modify objects versions.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix json decoding of rgw_bucket

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: regionmap update

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: get/set region map

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: zone list, setup changes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

common/ceph_parser.cc: cleanup

remove extra logging to stdout

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: set region info, default region

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: can list regions, show default region info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: some region/zone related cleanups/fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>