Merge pull request #28 from ceph/wip-10852

better logging for users

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

use dev_log for locking and unlocking of shards

Signed-off-by: Alfredo Deza <adeza@redhat.com>

only report bucket to retry if there are some

Signed-off-by: Alfredo Deza <adeza@redhat.com>

worker bound operations should also go to dev_log

Signed-off-by: Alfredo Deza <adeza@redhat.com>

swap log statements to go to dev_log

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove double 'syncing object' lines

Signed-off-by: Alfredo Deza <adeza@redhat.com>

use the new dev_logger for some shard sync items

use obj.name for logging objects that are not found in master

remove len() for generator object

Signed-off-by: Alfredo Deza <adeza@redhat.com>

full sync of a bucket with better logging

Signed-off-by: Alfredo Deza <adeza@redhat.com>

inform the user about data sync operations

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #27 from ceph/wip-11246

correctly import InvalidProtocol from its right location

Merge pull request #26 from ceph/wip-11245

configure logging before parsing arguments

correctly import InvalidProtocol from its right location

Signed-off-by: Alfredo Deza <adeza@redhat.com>

configure logging before parsing arguments

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22 from ceph/wip-10453

Wip 10453

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

when getting the worker bounds map the retries to objects

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rename the helper to to_obj

Signed-off-by: Alfredo Deza <adeza@redhat.com>

tests for the new keys to attrs helper

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create a helper to get keys to attributes in an object

Signed-off-by: Alfredo Deza <adeza@redhat.com>

make it explicit that obj refers to the utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>

use the new utility to slap extra metadata to retries

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add tests for the new utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create an obj util that can convert objects to_dict

Signed-off-by: Alfredo Deza <adeza@redhat.com>

make the list_object_in_bucket use the config for versioned support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

inform the user what request is hapenning

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25 from ceph/wip-log-location

fallback to cwd if log location is unavailable

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

report back to the user when falling back with the log location

Signed-off-by: Alfredo Deza <adeza@redhat.com>

fallback to cwd if log location is unavailable

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24 from ceph/wip-stream-logger

set the stream logging before anything else

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

set the stream logging before anything else

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23 from ceph/wip-10852

initial work for better logging structure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

use the actual line instead of string replacement

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove extra setLevel call

Signed-off-by: Alfredo Deza <adeza@redhat.com>

just re-raise AgentErrors when getting the region map

Signed-off-by: Alfredo Deza <adeza@redhat.com>

use the new logging handlers in cli.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create a RegionMapError exception

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add a decorator utility module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

make the client use the new NetworkError exception

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create a new 'AgentError' base exception

Signed-off-by: Alfredo Deza <adeza@redhat.com>

move the log module into util

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create a utility module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

1.2.1

update the changelog

Signed-off-by: Alfredo Deza <adeza@redhat.com>

bumping to 1.2.1

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20 from ceph/wip-9686-sync-marker

new sync marker and `get_worker_bound` API

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

worker will now use the new get_worker_bound values

Signed-off-by: Alfredo Deza <adeza@redhat.com>

better comment on default values for worker bound

Signed-off-by: Alfredo Deza <adeza@redhat.com>

incrementalSync will use the new worker_bound behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>

use the right key for timestamp: oldest_time

Signed-off-by: Alfredo Deza <adeza@redhat.com>

tests for the new get_worker_bounds behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>

move the constants from worker.py to constants.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

move constant values to a new module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

make get_worker_bound return a dict to support new keys automatically

Signed-off-by: Alfredo Deza <adeza@redhat.com>

fix attribute errors for exceptions in worker.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #21 from ktdreyer/rpm-dist

packaging: use dist tag in RPM Release field

packaging: use dist tag in RPM Release field

This change adds ".el6", ".el7", etc to the end of the RPM filename.

More information about the dist tag can be found at
http://fedoraproject.org/wiki/Packaging:DistTag

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #19 from ceph/wip-10584

fix exception attribute errors in sync.py

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

fix exception attribute errors in sync.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18 from ceph/wip-10569

exceptions refactoring

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

fix the worker exception tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add the worker exceptions into their own module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

get worker exceptions into the exceptions module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

update tests for the new module changes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create an exceptions module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove exceptions and errors from client onto exceptions.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add tests for listing objects in buckets with real http body from responses

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Merge pull request #17 from ktdreyer/spec-cleanup

packaging: RPM .spec cleanup

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

packaging: use full PyPI URL for RPM Source tarball

Alfredo has published the radosgw-agent package to PyPI today. Add the
full URL.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: add missing build deps to RPM

In order to build the radosgw-agent package in mock, the python-devel
and python-setuptools packages must be present in the buildroot.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: remove RPM defattr setting

The %defattr setting in the %files list is only necessary for very old
versions of RPM.  For RHEL 6 and later, it's fine to leave it out.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: remove RPM %clean section

The %clean section is only necessary for very old versions of RPM. For
RHEL 6 and later, it's fine to leave it out.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: simplify RPM %setup section

Remove duplicated -n settings to %setup. There were two that both did
the same thing, and neither one is needed, since %setup unpacks
%{name}-%{version} by default.

Make %setup run quietly (with -q). This fixes an rpmlint warning.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: add RPM %description

Add a simple description to the RPM packaging. This phrase was
more-or-less taken from the docs in ceph.git.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: use upper-case URL tag in RPM

Either "Url:" or "URL:" works, but the Fedora project has standardized
on "URL:"

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: rm unnecessary RPM tags

The BuildRoot tag is only needed on RHEL 5. RHEL 6 and newer are fine
without this setting.

The %{_prefix} macro is already /usr by default.

The Vendor tag is something that should be left undefined in the
packaging .spec itself. Ordinarily this is added during the build by the
build system.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: rm unnecessary RPM macro definitions

Remove the extraneous macro definitions, %{name}, %{version},
%{unmangled_version}, %{release}. RPM automatically sets these when we
use the Name, Version, and Release tags.

Discussed %{unmangled_version} in #ceph-devel today; not sure where that
one came from.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

create a changelog

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

update author email

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Merge pull request #16 from ktdreyer/fixes-from-v1.2

#10518: add fixes from v1.2

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Update Debian Changelog.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
(cherry picked from commit cc845481a60d6d0da92879e5ef411d8d89c572f3)

Branch for 1.2 release.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
(cherry picked from commit 17fa0f7dc85f658547dd79a5416f55fa709a4956)

Merge pull request #15 from ceph/wip-socket-leak

client: workaround socket leak

client: workaround socket leak

Recreate the boto connection object after every 512 requests in an
attempt to workaround a socket leak when the remote end closes the
connection. Boto does not seem to be causing httplib to close() the
sockets appropriately in some cases. I'm not sure exactly where
the leak is occurring, but forcing boto to reinitialize its
connection pool like this avoids it.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14 from ceph/wip-full-sync

Remove dependency on requests and other full sync fixes

worker: check op state for progress on any HTTP error

500 can be caused by an fcgi timeout when the operation still ends up
succeeding. It doesn't hurt to check for op state being in progress in
general in case other error codes happen but don't indicate actual
failure of the copy, since wait_for_object fails immediately if the
op state is not in progress.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

sync: use ' ' as the default shard so retries can be stored

The replica log api doesn't allow empty markers, but we may need to
store retries in the replica log even if there are no log entries. Use
' ' as a marker instead, since it is before all possible markers.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

full data sync: treat 404 from bucket list as success

Since results are paged, the exception may not occur immediately, so
add a new type to distinguish it from other client exceptions, and
catch it at any point during full sync of a bucket.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

worker: ignore 404 when removing op_state

It's fine if the gateway deleted it first.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

re-purpose urlencode to craft the query string for us without quoting again

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

create a test to check/prevent double quoting of urls

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

add get_bucket_list test

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

tests for get API calls

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

pep8/style changes to client.py

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

http response does not have .raw anymore

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Merge pull request #13 from ceph/wip-full-sync

Bug fixes for full sync and a few other data sync cases

client: return object names instead of boto resultsets

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Add tests for boto_call decorator

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

More tests for wait_for_object()

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Immediately fail when op state is not found

There's a very narrow window in which our connection sending the copy
request could fail before radosgw records the op state, and we query
the op state in that time. Retrying later at that point is fine, and
avoids a long timeout if there is a bug causing us to reach this point
without the op state ever being recorded. This race hasn't been
observed in practice.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>