test/rgw/test_multi: fix a few tests to only iterate over rw zones

Some of the tests require at least two read-write (regular rgw) zones

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: zone_cloud: deal with key representation and other fixes

Needed to present a key to the tests that reflected its original name
and version_id (and etag), so that the callers don't need to be modified.
However, this can only be achieved if we get the key, which doesn't work
if the caller was just listing the bucket objects. Created a new CloudKey
class to deal with the different issues there.
Also, other test related fixes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: streaming put also stores content_type and other fields

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix parse_tier_config_param function

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: don't call http data callbacks under lock

There is no need to hold req_data->lock when calling into client
callbacks. This removes an unneeded lock dependency (that is a
problem when cancelliing coroutines stack).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutines: cancel stacks on teardown

If we don't cancel stacks, ops might not be destructed, so ops callbacks
could still be active.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't store etag with extra null character at the end

head objects etag attr doesn't need to store an extra null char.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cloud sync: store source object info in destination object

store extra meta params on target object (original name, version_id, etag,
etc.)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cloud sync: store versioned epoch in target object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: initial work on cloud sync test

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api adjustment following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework cloud sync configuration

Change the configuration structure to separate between
connections and acl mappings, and other changes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fixes following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use different io id when waking up interval wait

Can't use a zero channel id as a wildcard.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix compilation warning

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

vstart: fixes for mstart

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: apply acl mappings

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: sync acls

still not transforming acls according to configuration

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: acl_mappings config handling

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: remove bucket suffix configurable

No need to have a special suffix, target path can be used instead.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: use configurable target path

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: dump config

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

formatter: add two utility classes

For automatic section scoping

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: configure different targets

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: new config structure, support multiple connections

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

formattable: more fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync module tier config fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

formattable: fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cloud sync, sync object meta

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

formattable: adjust array access interface

appending new entry by providing empty index, setting existing entries
by providing negative index.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: formattable, set, erase, unitest

Extend the formattable api to provide a mechanism to set and erase
entities by a string key that references them. E.g., "foo.bar[123].xyz"

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle the situation that get a 409 response from S3 correctly
pass the http body and use it when creating a bucket

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: avoid use Chunked transfer encoding in RGW's request

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: add a bucket-suffix tier-config

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: virtual hosted-style support

add host-style field in tier-config to specify the related
zone's hosted-style used in request from RGW, if this config
is not specified, path hosted-style will be used as default.

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: RGWRESTStreamRWRequest::send_prepare(), url encode dest

encode the resource in send_prepare(), as a result, all the callers
to do_send_prepare() have do url_encode before calling it.

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: add encode_slash param in url_encode

when sent request to S3, we should not encode the forward
slash character('/') in the object key name, so we need add
a encode_slash param in url_encode to decide whether to encode
the slash or not.

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: use GMT time format defined by RFC1123 in http request header

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: sync modules: adapt to formattable based config

still need to fix rgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/ceph_json_formattable: add unitest

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: formattable fix, api changes

add implicit casts

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: refine Formattable api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync modules, use JSONFormattable

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rgw_str_to_bool() should also accept "true"

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

common/formattable: initial work

initial work on a dict-like api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw/cleanup: fix compile warnning

1. overloaded virtual function
2. virtual functions but no virtual destructors

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: modify testcase in src/test/rgw

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: add populate_zonegroup() and populate_uid() in RGWRESTConn

populate_params() is too generic, add populate_zonegroup() and
populate_uid() in RGWRESTConn, then have populate_params() call them.

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: avoid returning CURL_READFUNC_PAUSE forever

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: implement S3RESTConn to send request to S3
add a sub-class of RGWRESTConn called S3RESTConn that do not
populate zonegroup-related params, it shall only used for send request
to S3-like interface.

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: decrease the log level in RGWCoroutinesStack::unwind

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: RGWRESTStreamS3PutObj::send_init(), url encode dest

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cr rest splice, reorder checks

First finish output init, then check if there's no data. Could be an empty
object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove warning

No need to clean io_ids when stack finishes, as list of io_ids is kept inside
the stack object, and will be cleaned up anyway. We also have a list of finished
ios in the completion manager (complete_reqs), but this is a fifo, and will be
cleaned up once pulled out (and we identify that io references an obsolete
stack).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't allow for io_id to complete more than once

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cr rest splice, both reads and writes are throttled

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http_client, add the ability to pause/unpause incoming data

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rest_client: work towards throttling of http read requests

Adjust the interfaces to provide the ability for the read callback
to pause the reads. While doing that, define a new class interface
for this instead of RGWGetDataCB. This had a butterfly effect that
required modifications to the obj read filters, but the end result
is a bit cleaner.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cr rest splice, work towards write throttling

Need to throttle writes, so that we don't just accumulate all data read
from source endpoint in memory, in the case where the write endpoint is
too slow.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cr: introduce io channels

ios can have multiple channels, so that we can differentiate between
waiting on read/write/control events. We can then block on a specific,
any, or multiple events.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: configurable multipart threshold, part size

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync conf, consolidate configurables

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http req, guard out_headers

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: check that source object doesn't change

Make suret that while syncing the object it doesn't change, which can
be a problem when uploading the object piecemeal.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync, store temp per-object sync info

When doing a multipart object sync, need to store the object's info
so that we can either continue if upload is interrupted, and abort
older upload_id in case object changed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: multipart upload complete

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't schedule a cr if already scheduled

This can be triggered when multiple IOs return for the same cr. We
don't want to allow it to be scheduled multiple times, as this
might wake up blocking IOs that shouldn't be woken up. If we try to
block on an IO that we already complete, we won't block anyway.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync, initial work on multipart data parts creation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix io blocked accounting

Now that we can have multiple IOs per coroutine, need to only decrease
num_blocked one time when switching from io blocked to unblocked. Also,
is_blocked() should not return true if is_done.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync, more work on large object sync via multipart upload

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync module, fix target object naming

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync module, fix target object naming

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix rest read crf handling of extra data

also fix splice coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: recover from empty meta.history object

Fixes: http://tracker.ceph.com/issues/21802
Empty meta.history object is problematic, as when reading it when doing
meta log trimming, we'd just initialize it to an empty structure, and
end up asserting because cursor is empty. Cannot just return -ENOENT
here, because it wouldn't solve the issue.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync, add hooks for decoding/encoding rest obj

Object sync is now functional. Create abstraction that will later help
with different cloud providers.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync, in_crf init abstraction

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: generate target bucket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a bit cleaner interfaces for sending put crf

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: first go at sync to aws splice

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWRESTConn() don't override provided key

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more streaming crf abstraction

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: separte stream crfs for read and write

also add a base class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: splice cr drains writes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rest cr drains writes, returns errors

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: groundwork for supporting concurrent IOs in single cr

Add io_id field to io operation, allow blocking on a specific IO.
Locking changes in RGWCoroutinesManager::run().

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial work for integrating streaming read/write with cr

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: amend http client manager interface

s/set_threaded/start
s/is_threaded/is_started

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http client, simplify interfaces

work towards removal of duplicate synchronous api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create global http manager

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: restructure http/rest client class hierarchy

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: tweak http client interface

RGWHTTPClient now holds the url and method.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add api to finish streaming writes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http client, streaming writes api

Add a new class that enables streaming writes, that is --
doesn't require having all the data before the start of the
send.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fixes following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: typo fix during rebase (squash)

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_rest_client: clarify comment on set content length

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

aws: remove comment

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

aws: use slashes in obj name

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_sync_module_aws: make conn a unique ptr

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_sync_module_aws: use the readrawrestresource cr

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>