tests for the new network utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add a network utility to check for ipv6

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add more tests for client that involve ipv6

Signed-off-by: Alfredo Deza <adeza@redhat.com>

properly detect ipv6 in client.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

use netloc instead of hostname

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #29 from Ved-vampir/master

radosgw-agent: supress override of config settings by argparse defaults

Reviewed-by: Alfredo Deza <adeza@redhat.com>

1.2.2

Merge pull request #30 from ceph/release-1.2.2

Release 1.2.2

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

update the changelog

Signed-off-by: Alfredo Deza <adeza@redhat.com>

bump the version of the agent to 1.2.2

Signed-off-by: Alfredo Deza <adeza@redhat.com>

radosgw-agent: supress override of config settings by argparse defaults
Fixes #11365

Sign-off-by: Alyona Kiseleva <akiselyova@mirantis.com>

manually bisecting check for old way of bucket_instance from agent

worker: use current bucket instance, if plain bucket name passed
from: 52c409a782d46ab2f21135de05e2af00656dcf2c
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #28 from ceph/wip-10852

better logging for users

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

use dev_log for locking and unlocking of shards

Signed-off-by: Alfredo Deza <adeza@redhat.com>

only report bucket to retry if there are some

Signed-off-by: Alfredo Deza <adeza@redhat.com>

worker bound operations should also go to dev_log

Signed-off-by: Alfredo Deza <adeza@redhat.com>

swap log statements to go to dev_log

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove double 'syncing object' lines

Signed-off-by: Alfredo Deza <adeza@redhat.com>

use the new dev_logger for some shard sync items

use obj.name for logging objects that are not found in master

remove len() for generator object

Signed-off-by: Alfredo Deza <adeza@redhat.com>

full sync of a bucket with better logging

Signed-off-by: Alfredo Deza <adeza@redhat.com>

inform the user about data sync operations

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #27 from ceph/wip-11246

correctly import InvalidProtocol from its right location

Merge pull request #26 from ceph/wip-11245

configure logging before parsing arguments

correctly import InvalidProtocol from its right location

Signed-off-by: Alfredo Deza <adeza@redhat.com>

configure logging before parsing arguments

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22 from ceph/wip-10453

Wip 10453

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

when getting the worker bounds map the retries to objects

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rename the helper to to_obj

Signed-off-by: Alfredo Deza <adeza@redhat.com>

tests for the new keys to attrs helper

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create a helper to get keys to attributes in an object

Signed-off-by: Alfredo Deza <adeza@redhat.com>

make it explicit that obj refers to the utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>

use the new utility to slap extra metadata to retries

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add tests for the new utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create an obj util that can convert objects to_dict

Signed-off-by: Alfredo Deza <adeza@redhat.com>

make the list_object_in_bucket use the config for versioned support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

inform the user what request is hapenning

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25 from ceph/wip-log-location

fallback to cwd if log location is unavailable

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

report back to the user when falling back with the log location

Signed-off-by: Alfredo Deza <adeza@redhat.com>

fallback to cwd if log location is unavailable

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24 from ceph/wip-stream-logger

set the stream logging before anything else

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

set the stream logging before anything else

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23 from ceph/wip-10852

initial work for better logging structure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

use the actual line instead of string replacement

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove extra setLevel call

Signed-off-by: Alfredo Deza <adeza@redhat.com>

just re-raise AgentErrors when getting the region map

Signed-off-by: Alfredo Deza <adeza@redhat.com>

use the new logging handlers in cli.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create a RegionMapError exception

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add a decorator utility module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

make the client use the new NetworkError exception

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create a new 'AgentError' base exception

Signed-off-by: Alfredo Deza <adeza@redhat.com>

move the log module into util

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create a utility module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

1.2.1

update the changelog

Signed-off-by: Alfredo Deza <adeza@redhat.com>

bumping to 1.2.1

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20 from ceph/wip-9686-sync-marker

new sync marker and `get_worker_bound` API

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

worker will now use the new get_worker_bound values

Signed-off-by: Alfredo Deza <adeza@redhat.com>

better comment on default values for worker bound

Signed-off-by: Alfredo Deza <adeza@redhat.com>

incrementalSync will use the new worker_bound behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>

use the right key for timestamp: oldest_time

Signed-off-by: Alfredo Deza <adeza@redhat.com>

tests for the new get_worker_bounds behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>

move the constants from worker.py to constants.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

move constant values to a new module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

make get_worker_bound return a dict to support new keys automatically

Signed-off-by: Alfredo Deza <adeza@redhat.com>

fix attribute errors for exceptions in worker.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #21 from ktdreyer/rpm-dist

packaging: use dist tag in RPM Release field

packaging: use dist tag in RPM Release field

This change adds ".el6", ".el7", etc to the end of the RPM filename.

More information about the dist tag can be found at
http://fedoraproject.org/wiki/Packaging:DistTag

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #19 from ceph/wip-10584

fix exception attribute errors in sync.py

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

fix exception attribute errors in sync.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18 from ceph/wip-10569

exceptions refactoring

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

fix the worker exception tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add the worker exceptions into their own module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

get worker exceptions into the exceptions module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

update tests for the new module changes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create an exceptions module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove exceptions and errors from client onto exceptions.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add tests for listing objects in buckets with real http body from responses

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Merge pull request #17 from ktdreyer/spec-cleanup

packaging: RPM .spec cleanup

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

packaging: use full PyPI URL for RPM Source tarball

Alfredo has published the radosgw-agent package to PyPI today. Add the
full URL.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: add missing build deps to RPM

In order to build the radosgw-agent package in mock, the python-devel
and python-setuptools packages must be present in the buildroot.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: remove RPM defattr setting

The %defattr setting in the %files list is only necessary for very old
versions of RPM.  For RHEL 6 and later, it's fine to leave it out.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: remove RPM %clean section

The %clean section is only necessary for very old versions of RPM. For
RHEL 6 and later, it's fine to leave it out.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: simplify RPM %setup section

Remove duplicated -n settings to %setup. There were two that both did
the same thing, and neither one is needed, since %setup unpacks
%{name}-%{version} by default.

Make %setup run quietly (with -q). This fixes an rpmlint warning.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: add RPM %description

Add a simple description to the RPM packaging. This phrase was
more-or-less taken from the docs in ceph.git.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: use upper-case URL tag in RPM

Either "Url:" or "URL:" works, but the Fedora project has standardized
on "URL:"

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: rm unnecessary RPM tags

The BuildRoot tag is only needed on RHEL 5. RHEL 6 and newer are fine
without this setting.

The %{_prefix} macro is already /usr by default.

The Vendor tag is something that should be left undefined in the
packaging .spec itself. Ordinarily this is added during the build by the
build system.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: rm unnecessary RPM macro definitions

Remove the extraneous macro definitions, %{name}, %{version},
%{unmangled_version}, %{release}. RPM automatically sets these when we
use the Name, Version, and Release tags.

Discussed %{unmangled_version} in #ceph-devel today; not sure where that
one came from.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

create a changelog

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

update author email

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Merge pull request #16 from ktdreyer/fixes-from-v1.2

#10518: add fixes from v1.2

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Update Debian Changelog.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
(cherry picked from commit cc845481a60d6d0da92879e5ef411d8d89c572f3)

Branch for 1.2 release.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
(cherry picked from commit 17fa0f7dc85f658547dd79a5416f55fa709a4956)

Merge pull request #15 from ceph/wip-socket-leak

client: workaround socket leak

client: workaround socket leak

Recreate the boto connection object after every 512 requests in an
attempt to workaround a socket leak when the remote end closes the
connection. Boto does not seem to be causing httplib to close() the
sockets appropriately in some cases. I'm not sure exactly where
the leak is occurring, but forcing boto to reinitialize its
connection pool like this avoids it.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14 from ceph/wip-full-sync

Remove dependency on requests and other full sync fixes

worker: check op state for progress on any HTTP error

500 can be caused by an fcgi timeout when the operation still ends up
succeeding. It doesn't hurt to check for op state being in progress in
general in case other error codes happen but don't indicate actual
failure of the copy, since wait_for_object fails immediately if the
op state is not in progress.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

sync: use ' ' as the default shard so retries can be stored

The replica log api doesn't allow empty markers, but we may need to
store retries in the replica log even if there are no log entries. Use
' ' as a marker instead, since it is before all possible markers.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

full data sync: treat 404 from bucket list as success

Since results are paged, the exception may not occur immediately, so
add a new type to distinguish it from other client exceptions, and
catch it at any point during full sync of a bucket.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>