rgw: keep objv_tracker on bucket info

This ties it to the specific object instance info.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: refactor policy reading

reuse bucket info, no need to reread it again.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix policy read

policy reading used to have exact same logic for both buckets
and objects, however, we now need to read bucket attributes
through get_bucket_info().

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: init member variable

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: encode bucket info only after setting a flag

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: initial work to separate bucket name and instance

Bucket info now resides in an instance specific object. The
old bucket info object now contains a pointer to the new
instance.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: initialize system flag in RGWUserInfo

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: various object put processor fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix broken obj put

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: handle bucket creation with specified placement pool

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: user configuration for bucket placement

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: data structures for new data/index placement rules

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: make replica log object name configurable

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

RGWReplicaBucketLogger: store bucket replica logs in the .logs pool

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge branch 'wip-rgw-geo-2' into wip-rgw-geo

Conflicts:
src/test/cli/radosgw-admin/help.t

rgw: buffer atomic put handler

Since we tied the atomic put handler to libcurl output
data, which uses much smaller chunks, we need to buffer
data, otherwise we'd end up with a huge amount of small
writes.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: tie opstate into intra-region copy operations

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge branch 'wip-rgw-geo-enovance' into wip-rgw-geo-2

fixup "radosgw-admin: new commands to look at and clean up replica logs"

radosgw-admin: new commands to look at and clean up replica logs

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: integrate with RGWRados

We introduce an implementation class RGWReplicaLogger, and two user
classes RGWReplicaObjectLogger (for the data/metadata logs) and
RGWReplicaBucketLogger (for the bucket logs).

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add tests.

We do several good updates, a bad update, a good delete,
a bad delete, several good gets, and a bad get.

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add client with user documentation

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add the actual class

This class can be applied to any object, and will keep track of
the bounds, matching times, and matching sets of in-progress items.
We're about to add a cls_replica_log_client that will provide a friendly
user interface and provide more documentation.

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add ops for new class

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add types for new class

Signed-off-by: Greg Farnum <greg@inktank.com>

rgw: lock related modifications

1. zone-id inclusion in lock/unlock for mdlog and datalog
2. renewal of lock if the locker request lock again
3. modified lock_id param to locker-id

Signed-off-by: Babu Shanmugam <anbu@enovance.com>
Conflicts:
src/rgw/rgw_rest_log.cc

rgw: add max-entries, marker for log operations

Signed-off-by: Babu Shanmugam <anbu@enovance.com>
Conflicts:
src/rgw/rgw_admin.cc
src/rgw/rgw_rest_log.cc

radosgw_admin: fixup cli test

Signed-off-by: Greg Farnum <greg@inktank.com>

test: update cli test

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_rgw: skip namespaced entries when listing bucket

We added a namespace to the entries, and sadly there can be
valid entries beyond it. Previously we assumed that once we
encounter the namespace tag we're done. That was wrong. We
need to skip the namespace and continue the listing from
there.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin: interface to control ops state

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: OpState internal api

Add new higher level functions to set, renew, list, and remove
logged operation state.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix check_state call

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: specialized obj zone copy state for statelog

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: internal api for statelog objclass

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: pass in timestamp

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: improve unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: fix a few issues

Issues that came up when developed the unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: handle racing default region creation

If region is not specified and there's no default region set
we try to create a default region. However, we may race with
another process trying to do the same, so we should handle
that gracefully.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test: test for cls_statelog

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: add client api functions

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: fixes and changes to api

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: introducing new objclass to handle state tracking

Somewhat similar to the log objclass, but uses different data for
indexing. Also keeps a dual index. In general an entry has 3
identifiers:
 - object: the object id on which the operation is made
 - client_id: client's unique identifier
 - op_id: operation's unique identifier

An entry is indexed by both client_id+op_id, and by
object+op_id, make it possible to list operations
either by client_id, or by object id.
We also keep state per each entry and the new check_state request
can be used as a guard.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: move a couple of unitests to a different Makefile rule

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: object mtime the same for both object and bucket index

Make sure object mtime that's passed to bucket index is the
same as the one set on the object.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix inter-region copy (ofs wasn't correct)

Fixing an issue with data read. Since inter-region copy uses
embedded metadata within the data input, we need to fix the ofs
accordingly.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: intra-region copy, preserve mtime

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: system user get obj returns JSON encoded metadata

instead of binary encoded metadata

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add 'delete' to signed subresources list

This fixes a regression that removed 'delete' from the signed
subresources list, thus made it impossible to authenticate
certain multipart object operations.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test: update cli test

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: set default region as master

A default region that is automaticall created needs to be
set as the master.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: get / set user & bucket meta mtime

also update infrastructure to be able to set mtime on objects

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: preserve bucket creation time across different zones / regions

Keep bucket creation time in RGWBucketInfo, and make use
of it where necessary.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: show mtime for metadata entries

currently only shows for buckets

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fixes for intra-zone object copy

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: set bucket marker and bucket obj version from master

Make sure that bucket that's created gets the same marker as
the one that the master region generated. Also, for some reason
we weren't passing in the metadata obj version for the bucket.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: multiple changes for intra-zone object copy

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: when setting bucket metadata key, don't override placement

Bucket placement also contains data and index pools locations in
the source zone. However, we might be applying it on a different
zone, so we need to be careful not to use the source location here.
We generate a new placement if a new bucket is detected, otherwise
we keep the old one.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: forward amz headers of copy request to remote rgw

So that if request has any special conditions, it'll be processed
on the remote gateway

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: cleanup, move init_meta_info() into req_info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: remove unused variable

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket marker contains zone name

(issue 5347)
We need this so that raw object names on different zones within
the same region don't collide.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: pass original object attrs through extra request data

introduce a new mechanism that sends extra data with object
info, and use it to encode source object metadata.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: a few fixes to copy object across region

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: move most of object put processors implementation

move code from rgw_op.cc to rgw_rados.cc, so that we could
use it in the lower layer.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: separate req_state from object put processors

At least, for most of them. We'll move this code to rgw_rados.cc,
and we don't want it to have req_state.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: organize get_obj handling for copy a bit different

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: generate read request if source rgw is remote

in a copy operation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: propagate mtime from remote rgw on copy

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: propagate error from remote gateway when copying object

Also make sure that we don't continue iterating locally through
the object.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: pass grant headers to target when writeing object

When writing object to remote gateway.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: head acls target can be quoted

when passing x-amz-grant-* headers, the target (id, email,
group) may be quoted.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: send meta headers with remote PUT request

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: don't busy wait for outgoing rest requests

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: don't send redirect if copy object has a local source

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: link bucket to owner on metadata update

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix logic related to operations on different region

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: minor logging change

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix json decoding for swift keys

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: stream obj into http request

still need to figure out curl handle polling, handle client
errors correctly.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: format put request auth

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket metadata, ignore ENOENT on put

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: skeleton for obj copy across regions

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: metadata lock/unlock implemented with test cases

metadata lock/unlock implemented with test cases
split Post operations to Lock and Unlock operations, and
checking for WRITE permission only for lock and unlock

Signed-off-by: Babu Shanmugam <anbu@enovance.com>
Conflicts:
src/rgw/rgw_rest_log.cc
src/rgw/rgw_rest_log.h

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

RESTful APIs for data changes log implemented with test cases

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: some user metadata fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw_admin: modify mdlog functionality

mdlog list, can handle cases where there is and there's no shard id
specified. mdlog trim requires shard if.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: set truncated as false if ENOENT when listing mdlog entries

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge branch 'wip-rgw-geo-enovance' into wip-rgw-geo-2

Conflicts:
src/Makefile.am
src/rgw/rgw_admin.cc

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: Do not assum rest connection to be established

Added a check to verify that rest connection is correctly initialized,
preventing a segmentation fault.

Signed-off-by: Christophe Courtaut <christophe.courtaut@gmail.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

flushed the formatter after closing the primary section in MDLOG, BILOG and GETSHARDINFO :: list()

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: verify bucket location only if exists

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: forward delete_bucket to master region

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Fixed default region creation issue

If the default region did not exist, it is created,
but the previously created RGWDefaultRegionInfo object is not updated.
This fix updates this objects after the default region creation.
It was also preventing vstart.sh script to start with -r option.

Signed-off-by: Christophe Courtaut <christophe.courtaut@gmail.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: user master region's version of bucket

When creating a bucket, retrieve the bucket object
version from the master region and use it.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: mdlog, bilog RESTful api cleanup

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

With mdlog lock and unlock functionality, listing and trimming base on shard_id

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: propagate region on bucket creation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>