rgw-admin: Adds option --shard-id to help.t

Signed-off-by: Christophe Courtaut <christophe.courtaut@gmail.com>

rgw: admin: Add --shard-id option to help

http://tracker.ceph.com/issues/5324 fixes #5324
Explains operations dependency,
which is more important than the semantic.

Signed-off-by: Christophe Courtaut <christophe.courtaut@gmail.com>

rgw: metadata, data logs can be turned on / off

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: modify RGWBucketInfo::creation_time encoding / decoding

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: end-marker serves as last value, not as upperbound

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: relax marker requirements for log apis

Don't require start-time, start-marker.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: update log APIs to use markers

a few APIs were missing markers, fixed that.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: cls_log_entry has id field

The id field can be used as the marker for log list / trim
operations.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Makefile.am: some more makefile rules fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: date string parsing also parses seconds fraction

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix cli test

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Makefile.am: add missing rule

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: initialize sent_header in RGWOp_Opstate_List

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw-admin: restructure replicalog commands

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge remote-tracking branch 'enovance/wip-rgw-geo-2-opstate' into wip-rgw-geo-2

Conflicts:
src/Makefile.am
src/rgw/rgw_admin.cc
src/rgw/rgw_common.h

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: don't remove bucket index when removing a bucket

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bilog list by bucket instance (RESTful api)

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

ceph_json: reset values

Call the default constructor if entry not found. Also,
reset containers before inserting new data.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: user param in access key is not manadatory

for json decoding

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: advance ptr when sending data

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

changes to accomodate review comments on previous version

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: fix regionmap json decoding

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: automatic pool creation for placement pools

With the new pools configuration, now we auto create the
pools when needed (through bucket creation). Also, make
sure only to configure default placement in zone structure,
if old config hasn't been done yet.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: remove test placement info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw (test): remove some warnings

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: initialize user system flag

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: log in the same shard for bucket entry point and instance

We'd like to have bucket entry point and instance info at the same
log shard, so that we can process them in order.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: unlink/link don't always update entry point

Some operations already update the entry point, so no
need to do it again.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: tie metadata put to bucket link/unlink

and lot's of constifying

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_rgw: cleanup

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: some more internal api cleanups

Use of rgw_bucket when referring to the bucket instance,
use bucket name when referring to the bucket entry point.
Also, remove bucket input param where not needed (internally
was using the bucket structure from the bucket info).

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: unlink bucket from user on metadata rm bucket:< bucket>

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fixes to object versioning tracking

There are a few different cases for setting the object version.
Either we need to create a new version, or we need to set the
version provided (one metadata put). We also need to make sure
that we log the correct previous version of the object. This
commit fixes a few cases.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: filter read xattrs

We're only interested in object xattrs that have specific rgw.user
prefix.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add str_startswith()

useful util

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_ver: rename version xattr, add some more logging

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: multiple fixes related to metadata, bucket creation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: bucket entry point contains owner, other fix

We now keep the owner at the bucket entry point. Also, reading
bucket info at RGWCreateBucket::execute(), because previous changes
has made it so that we didn't have the required bucket info
initialized there.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

opstate RESTful API implementation with unit test application

Signed-off-by: Babu Shanmugam <anbu@enovance.com>

rgw: add a system request param to select op bucket instance

Added rgwx-bucket-instance request param that selects the actual bucket
instance to use for this request.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: data log contains bucket instance info in key

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: format bucket.instance meta entries nicely

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: use new instance handler for bucket operations

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: create meta handler for bucket instance

Create utility functions for reading writing bucket entry
point and bucket instance. Add a separate meta handler for
bucket instance.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: put_bucket_info() uses objv_tracker in bucket info

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: keep objv_tracker on bucket info

This ties it to the specific object instance info.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: refactor policy reading

reuse bucket info, no need to reread it again.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix policy read

policy reading used to have exact same logic for both buckets
and objects, however, we now need to read bucket attributes
through get_bucket_info().

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: init member variable

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: encode bucket info only after setting a flag

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: initial work to separate bucket name and instance

Bucket info now resides in an instance specific object. The
old bucket info object now contains a pointer to the new
instance.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: initialize system flag in RGWUserInfo

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: various object put processor fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix broken obj put

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: handle bucket creation with specified placement pool

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: user configuration for bucket placement

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: data structures for new data/index placement rules

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: make replica log object name configurable

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

RGWReplicaBucketLogger: store bucket replica logs in the .logs pool

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge branch 'wip-rgw-geo-2' into wip-rgw-geo

Conflicts:
src/test/cli/radosgw-admin/help.t

rgw: buffer atomic put handler

Since we tied the atomic put handler to libcurl output
data, which uses much smaller chunks, we need to buffer
data, otherwise we'd end up with a huge amount of small
writes.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: tie opstate into intra-region copy operations

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge branch 'wip-rgw-geo-enovance' into wip-rgw-geo-2

fixup "radosgw-admin: new commands to look at and clean up replica logs"

radosgw-admin: new commands to look at and clean up replica logs

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: integrate with RGWRados

We introduce an implementation class RGWReplicaLogger, and two user
classes RGWReplicaObjectLogger (for the data/metadata logs) and
RGWReplicaBucketLogger (for the bucket logs).

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add tests.

We do several good updates, a bad update, a good delete,
a bad delete, several good gets, and a bad get.

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add client with user documentation

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add the actual class

This class can be applied to any object, and will keep track of
the bounds, matching times, and matching sets of in-progress items.
We're about to add a cls_replica_log_client that will provide a friendly
user interface and provide more documentation.

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add ops for new class

Signed-off-by: Greg Farnum <greg@inktank.com>

cls_replica_log: add types for new class

Signed-off-by: Greg Farnum <greg@inktank.com>

rgw: lock related modifications

1. zone-id inclusion in lock/unlock for mdlog and datalog
2. renewal of lock if the locker request lock again
3. modified lock_id param to locker-id

Signed-off-by: Babu Shanmugam <anbu@enovance.com>
Conflicts:
src/rgw/rgw_rest_log.cc

rgw: add max-entries, marker for log operations

Signed-off-by: Babu Shanmugam <anbu@enovance.com>
Conflicts:
src/rgw/rgw_admin.cc
src/rgw/rgw_rest_log.cc

radosgw_admin: fixup cli test

Signed-off-by: Greg Farnum <greg@inktank.com>

test: update cli test

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_rgw: skip namespaced entries when listing bucket

We added a namespace to the entries, and sadly there can be
valid entries beyond it. Previously we assumed that once we
encounter the namespace tag we're done. That was wrong. We
need to skip the namespace and continue the listing from
there.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin: interface to control ops state

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: OpState internal api

Add new higher level functions to set, renew, list, and remove
logged operation state.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix check_state call

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: specialized obj zone copy state for statelog

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: internal api for statelog objclass

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: pass in timestamp

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: improve unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: fix a few issues

Issues that came up when developed the unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: handle racing default region creation

If region is not specified and there's no default region set
we try to create a default region. However, we may race with
another process trying to do the same, so we should handle
that gracefully.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test: test for cls_statelog

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: fixes

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: add client api functions

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: fixes and changes to api

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls_statelog: introducing new objclass to handle state tracking

Somewhat similar to the log objclass, but uses different data for
indexing. Also keeps a dual index. In general an entry has 3
identifiers:
 - object: the object id on which the operation is made
 - client_id: client's unique identifier
 - op_id: operation's unique identifier

An entry is indexed by both client_id+op_id, and by
object+op_id, make it possible to list operations
either by client_id, or by object id.
We also keep state per each entry and the new check_state request
can be used as a guard.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: move a couple of unitests to a different Makefile rule

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: object mtime the same for both object and bucket index

Make sure object mtime that's passed to bucket index is the
same as the one set on the object.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix inter-region copy (ofs wasn't correct)

Fixing an issue with data read. Since inter-region copy uses
embedded metadata within the data input, we need to fix the ofs
accordingly.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: intra-region copy, preserve mtime

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: system user get obj returns JSON encoded metadata

instead of binary encoded metadata

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add 'delete' to signed subresources list

This fixes a regression that removed 'delete' from the signed
subresources list, thus made it impossible to authenticate
certain multipart object operations.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test: update cli test

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: set default region as master

A default region that is automaticall created needs to be
set as the master.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: get / set user & bucket meta mtime

also update infrastructure to be able to set mtime on objects

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: preserve bucket creation time across different zones / regions

Keep bucket creation time in RGWBucketInfo, and make use
of it where necessary.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>