rgw: allow setting of explicit object version on non-versioned bucket

If system user has set explicit olh version and/or object instance id,
treat it as the bucket was versioned anyway. This can happen with a
racing bucket metadata vs. data sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: don't increase epoch when converting plain entry

Plain entries that are getting converted gets epoch 1, as these are non
versioned ones, and every entry following these will be versioned. Don't
increase the epoch for the operation.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix a few regressions

 - need to increase olh epoch if converting plain entry, otherwise
   ordering is skewed
 - if entry is olh (e.g., colocated data + olh), need to make sure we
   write olh idtag xattr
 - completely revalidate obj state after writing it, otherwise attrs
   won't have idtag, and later check to verify if it's olh will fail

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move versioning handling to Object::Write::write_meta()

Instead of having to call set_olh() after calling write_meta(), do it
inside that method. This way we (mostly) don't need to care about
versioning outside of it. This required moving bucket info around
instead of just passing the bucket owner.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: copy obj does versioning too

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate version id when putting obj

A system user can set the rgwx-version-id http param, and it's now sent
to the objclass.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cleaup RGWRados::copy_obj()

separate into multiple methods.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls_rgw: provide optional version id, versioned epoch to olh ops

Let system users specify optional params: rgwx-version-id,
rgwx-versioned-epoch. These will apply to object modification operations
(put object, copy object, delete object), and provide the mechanism that
the sync agent will use to apply remote object changes.
Copy operation hasn't been done yet.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: link, unlink olh ops can get epoch

If epoch is specified, we need to handle updates differently, only
update the olh itself if passed epoch is newer than the olh current
epoch. Otherwise this is a replay of an older operation that doesn't
affect the olh (or the "current" property).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw, rgw: pending_log can hold multiple entries per epoch

This is needed, so that we can later replay operations, using the epoch
for idempotency.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: parse copy location version id

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reduce use of Object::get_obj()

We should avoid using this one, as it's not going to point at the object
instance in case it's an olh. Only kept the use in
Object::Delete, as we make explicit checks before using it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: Object::Read operations should use state->obj

and not the provided obj, as it might have been an olh, therefore we
have prepare() that sets the state.obj as the effective object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: Object::Read::read() returns total bytes read

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: time out pending olh entries

remove old pending entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: don't write list entry when converting when deleting

when converting the plain entry into a versioned entry, don't write the
list entry if this is a delete op with a null instance.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: revise null object instance handling, versioned epoch

Separate the versioned epoch from the object entry version, since in the
case of null instane entries the object version actually contains
meaningful data that can be overwritten.
Keep separate instance entries for null versioned object and delete
marker, since the two can coexist for a period of time. With suspended
versioning bucket, when deleting a null versioned object we create an
delete marker, and set the olh log to remove the object instance, which
will make the gateway send a removal request. Had we kept the same
entry it'll remove the wrong entry.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read bucket owner when following olh if pending entries

We need to read the bucket owner, so that we'd be able to apply any
changes to the object. Bucket owner is needed for quota related
accounting.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove warning

This cleanup is not essential at this point

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unused code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: clean up compilation warnings

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: implement dump() and generate test instances

Add missing methods for some cls_rgw op types.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: guard certain operations using olh tag

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls_rgw: improve olh atomicity

Add olh_tag attr to olh object, and use it instead of obj_tag as object
data may be colocated with the olh. Use the olh_tag to guard various
operations.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: encode timestamp in pending olh info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: remove instance entry when removing delete marker

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: send x-amz-version-id and x-amz-delete_marker header fields

send these fields on obj delete as defined by the api. This required
adding an index read so that we can identify whether an instance is
actually a delete marker.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add dump_string_header()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: add missing cls_cxx_create()

This was found during code read, not necessarily a real issue, however,
we counted on the object being created implicitly by setting xattrs on
it. Instead make sure the object is created.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls_rgw: fix update of olh to reflect non existing object

need to update olh appropriately when removing the last reference of an
object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: inc olh epoch when updating log

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix access to object through the null instance

Beforehand we were following olh if the explicit 'null' instance was
specified.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adjust versioning enable/suspend api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate object owner and mtime for deletion marker

deletion marker needs to have object owner and mtime properties.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls_rgw: multiple changes related to obj removal

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: send "null" version id if needed

on bucket list versions

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

formatter: no need for dynamic allocation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

osd: fix filter_prefix scoping in omap_get_vals

filter_prefix should only affect the case where it's greater than
start_after, otherwise we'll end up with a key equals to start_after.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls_rgw: add bi put

An admin operation to set bucket index entries.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw, rgw-admin: create bi list operation

Can now list bucket index entries that correspond to a single key. This
is useful when debugging versioned objects issues.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw, rgw-admin: move bi_get() entry encoding to cls

Create a new structure to hold entry info, so that we can later creata a
bi_list() operation.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: add missing flags encoding to rgw_bucket_dir_entry::dump()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

common, rgw: json escaping gets input buf size

We now deal with strings that embed a null character.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin, cls_rgw: add bi_get objclass operation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: keep null-versioned object as versioned object

when olh is created. Mark the object as such, so that we'll know to
handle it correctly. This is needed so that we could handle the 'null'
version as required.
Still doesn't work correctly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: follow olh where needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: unlink obj instance

use new objclass method

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: unlink object instance

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: update olh log when unlinking entry

Need to update log so that later client will be able to appropriately
remove the object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: more cleanup

move versioned data and olh handling into class methods.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: more cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: reorganize rgw_bucket_link_olh()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: convert RGWRados::get_attr() to new interface

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch get_obj_iterate() to new interface

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unused code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: purge intent log

We don't use the intent log anymore, just remove it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adjust to new interfaces

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: s/RGWRados::ObjectCtx/RGWObjectCtx

Can't forward declare nested class.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clean up system obj interfaces

Create a new structure to use, RGWRados::SystemObject, similar to the
previous changes made for the data objects.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: change RGWRados::get_attr()

This is an interim fix, will need to make it work similarly to the other
calls.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework prepare_get_obj(), get_obj()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove more unused code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix missing state initalization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch RGWRados::delete_obj() to new interface

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unused code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove old index update calls

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: get rid of put_obj_meta(), replace with put_system_obj()

Last use of put_obj_meta() was for system object, simplify it for that
specific use case and rename.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't use put_system_obj() for data objects

Use the new interface instead.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pass around object context refrences, remove unused code

We want to make it clear that the object context is required for
non-system objects, pass it by reference (rather than by pointer). Make
the put processors get it as a constructor param and keep it on the
object.
Still haven't updated everything, as some stuff is going to change
anyway.
Remove the plain put processor, as it is not used anymore.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove plain object processor

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start reorganizing RGWRados

Need to reduce the various states of the different operations. The aim
is to separate the system objects operations from the data objects
operations, and compartmentalize the various systems.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: s/RGWRadosCtx/ObjectCtx

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move RGWRadosCtx into RGWRados

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: set olh if object has been versioned

Bucket could have versioning suspended, but object has been versioned,
so once object has olh pointer in it, we need to continue maintainig it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: allow olh linking to null instance objects

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls_rgw: don't remove olh objects

A single head object can contain both olh data, and object data. Don't
remove object if it has olh information on it, just reset the other
info.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial versioned object removal implementation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket versioning status is tri-state

disabled, enabled, suspended.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add support for version-id-marker

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls_rgw: cls_bucket_list returns raw key in map

This is needed so that we dump the objects in the required order (from
newest to oldest versions for each object). While we're at it, revert to
the original encoding of rgw_bucket_dir, which will be better for
backward compatibility.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: update the appropriate prev key entry

We need to update the previous key entry that it's no longer the latest,
need to use the list entry, not the data entry (which only serves as a
pointer to the list entry).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix rgw_obj initialization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: request state and various op functionality use rgw_obj_key

instead of plain string. This is needed so that we use the object
instance in the processing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cleanup, get rid of req_state::object

We don't want this extra memory allocation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: restful api now dumps versions

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate dirent flags to rgw (from cls), other fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: deletion marker needs to keep instance entry

Otherwise we wouldn't be able to find it when we remove it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls_rgw: list object versions is optional

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket listing gets extra param for versioning

Can specify whether we want to see versions or not.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: revise the data model

We don't need to keep a listing entry for the olh.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add get_type() to rgw ops

This is needed so that we can identify ops correctly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add 'versioning', and 'versions' to handled subresources

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: object manifest should reflect instance

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update json encoding for rgw_obj

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: follow olh if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: olh trim op is read/write

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: tie set_olh() to object completion

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more atomicity fixes, set_olh()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: guard against racing writes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: olh atomicity groundwork

still more to do

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>