rgw: initialize user system flag

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: log in the same shard for bucket entry point and instance

We'd like to have bucket entry point and instance info at the same
log shard, so that we can process them in order.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: unlink/link don't always update entry point

Some operations already update the entry point, so no
need to do it again.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: tie metadata put to bucket link/unlink

and lot's of constifying

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_rgw: cleanup

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: some more internal api cleanups

Use of rgw_bucket when referring to the bucket instance,
use bucket name when referring to the bucket entry point.
Also, remove bucket input param where not needed (internally
was using the bucket structure from the bucket info).

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: unlink bucket from user on metadata rm bucket:< bucket>

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fixes to object versioning tracking

There are a few different cases for setting the object version.
Either we need to create a new version, or we need to set the
version provided (one metadata put). We also need to make sure
that we log the correct previous version of the object. This
commit fixes a few cases.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: filter read xattrs

We're only interested in object xattrs that have specific rgw.user
prefix.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add str_startswith()

useful util

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_ver: rename version xattr, add some more logging

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: multiple fixes related to metadata, bucket creation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket entry point contains owner, other fix

We now keep the owner at the bucket entry point. Also, reading
bucket info at RGWCreateBucket::execute(), because previous changes
has made it so that we didn't have the required bucket info
initialized there.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add a system request param to select op bucket instance

Added rgwx-bucket-instance request param that selects the actual bucket
instance to use for this request.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: data log contains bucket instance info in key

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: format bucket.instance meta entries nicely

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: use new instance handler for bucket operations

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: create meta handler for bucket instance

Create utility functions for reading writing bucket entry
point and bucket instance. Add a separate meta handler for
bucket instance.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: put_bucket_info() uses objv_tracker in bucket info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: keep objv_tracker on bucket info

This ties it to the specific object instance info.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: refactor policy reading

reuse bucket info, no need to reread it again.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix policy read

policy reading used to have exact same logic for both buckets
and objects, however, we now need to read bucket attributes
through get_bucket_info().

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: init member variable

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: encode bucket info only after setting a flag

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: initial work to separate bucket name and instance

Bucket info now resides in an instance specific object. The
old bucket info object now contains a pointer to the new
instance.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: initialize system flag in RGWUserInfo

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: various object put processor fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix broken obj put

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: handle bucket creation with specified placement pool

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: user configuration for bucket placement

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: data structures for new data/index placement rules

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: make replica log object name configurable

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

RGWReplicaBucketLogger: store bucket replica logs in the .logs pool

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge branch 'wip-rgw-geo-2' into wip-rgw-geo

Conflicts:
src/test/cli/radosgw-admin/help.t

rgw: buffer atomic put handler

Since we tied the atomic put handler to libcurl output
data, which uses much smaller chunks, we need to buffer
data, otherwise we'd end up with a huge amount of small
writes.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: tie opstate into intra-region copy operations

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge branch 'wip-rgw-geo-enovance' into wip-rgw-geo-2

fixup "radosgw-admin: new commands to look at and clean up replica logs"

radosgw-admin: new commands to look at and clean up replica logs

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: integrate with RGWRados

We introduce an implementation class RGWReplicaLogger, and two user
classes RGWReplicaObjectLogger (for the data/metadata logs) and
RGWReplicaBucketLogger (for the bucket logs).

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add tests.

We do several good updates, a bad update, a good delete,
a bad delete, several good gets, and a bad get.

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add client with user documentation

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add the actual class

This class can be applied to any object, and will keep track of
the bounds, matching times, and matching sets of in-progress items.
We're about to add a cls_replica_log_client that will provide a friendly
user interface and provide more documentation.

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add ops for new class

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add types for new class

Signed-off-by: Greg Farnum <greg@inktank.com>

rgw: lock related modifications

1. zone-id inclusion in lock/unlock for mdlog and datalog
2. renewal of lock if the locker request lock again
3. modified lock_id param to locker-id

Signed-off-by: Babu Shanmugam <anbu@enovance.com>
Conflicts:
src/rgw/rgw_rest_log.cc

rgw: add max-entries, marker for log operations

Signed-off-by: Babu Shanmugam <anbu@enovance.com>
Conflicts:
src/rgw/rgw_admin.cc
src/rgw/rgw_rest_log.cc

radosgw_admin: fixup cli test

Signed-off-by: Greg Farnum <greg@inktank.com>

test: update cli test

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_rgw: skip namespaced entries when listing bucket

We added a namespace to the entries, and sadly there can be
valid entries beyond it. Previously we assumed that once we
encounter the namespace tag we're done. That was wrong. We
need to skip the namespace and continue the listing from
there.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin: interface to control ops state

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: OpState internal api

Add new higher level functions to set, renew, list, and remove
logged operation state.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix check_state call

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: specialized obj zone copy state for statelog

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: internal api for statelog objclass

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: pass in timestamp

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: improve unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: fix a few issues

Issues that came up when developed the unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: handle racing default region creation

If region is not specified and there's no default region set
we try to create a default region. However, we may race with
another process trying to do the same, so we should handle
that gracefully.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test: test for cls_statelog

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: add client api functions

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: fixes and changes to api

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: introducing new objclass to handle state tracking

Somewhat similar to the log objclass, but uses different data for
indexing. Also keeps a dual index. In general an entry has 3
identifiers:
 - object: the object id on which the operation is made
 - client_id: client's unique identifier
 - op_id: operation's unique identifier

An entry is indexed by both client_id+op_id, and by
object+op_id, make it possible to list operations
either by client_id, or by object id.
We also keep state per each entry and the new check_state request
can be used as a guard.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: move a couple of unitests to a different Makefile rule

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: object mtime the same for both object and bucket index

Make sure object mtime that's passed to bucket index is the
same as the one set on the object.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix inter-region copy (ofs wasn't correct)

Fixing an issue with data read. Since inter-region copy uses
embedded metadata within the data input, we need to fix the ofs
accordingly.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: intra-region copy, preserve mtime

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: system user get obj returns JSON encoded metadata

instead of binary encoded metadata

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add 'delete' to signed subresources list

This fixes a regression that removed 'delete' from the signed
subresources list, thus made it impossible to authenticate
certain multipart object operations.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test: update cli test

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: set default region as master

A default region that is automaticall created needs to be
set as the master.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: get / set user & bucket meta mtime

also update infrastructure to be able to set mtime on objects

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: preserve bucket creation time across different zones / regions

Keep bucket creation time in RGWBucketInfo, and make use
of it where necessary.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: show mtime for metadata entries

currently only shows for buckets

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fixes for intra-zone object copy

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: set bucket marker and bucket obj version from master

Make sure that bucket that's created gets the same marker as
the one that the master region generated. Also, for some reason
we weren't passing in the metadata obj version for the bucket.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: multiple changes for intra-zone object copy

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: when setting bucket metadata key, don't override placement

Bucket placement also contains data and index pools locations in
the source zone. However, we might be applying it on a different
zone, so we need to be careful not to use the source location here.
We generate a new placement if a new bucket is detected, otherwise
we keep the old one.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: forward amz headers of copy request to remote rgw

So that if request has any special conditions, it'll be processed
on the remote gateway

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: cleanup, move init_meta_info() into req_info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: remove unused variable

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket marker contains zone name

(issue 5347)
We need this so that raw object names on different zones within
the same region don't collide.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: pass original object attrs through extra request data

introduce a new mechanism that sends extra data with object
info, and use it to encode source object metadata.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: a few fixes to copy object across region

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: move most of object put processors implementation

move code from rgw_op.cc to rgw_rados.cc, so that we could
use it in the lower layer.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: separate req_state from object put processors

At least, for most of them. We'll move this code to rgw_rados.cc,
and we don't want it to have req_state.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: organize get_obj handling for copy a bit different

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: generate read request if source rgw is remote

in a copy operation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: propagate mtime from remote rgw on copy

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: propagate error from remote gateway when copying object

Also make sure that we don't continue iterating locally through
the object.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: pass grant headers to target when writeing object

When writing object to remote gateway.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: head acls target can be quoted

when passing x-amz-grant-* headers, the target (id, email,
group) may be quoted.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: send meta headers with remote PUT request

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: don't busy wait for outgoing rest requests

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: don't send redirect if copy object has a local source

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: link bucket to owner on metadata update

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix logic related to operations on different region

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: minor logging change

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix json decoding for swift keys

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>