Merge pull request #24 from ceph/wip-stream-logger

set the stream logging before anything else

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

set the stream logging before anything else

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23 from ceph/wip-10852

initial work for better logging structure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

use the actual line instead of string replacement

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove extra setLevel call

Signed-off-by: Alfredo Deza <adeza@redhat.com>

just re-raise AgentErrors when getting the region map

Signed-off-by: Alfredo Deza <adeza@redhat.com>

use the new logging handlers in cli.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create a RegionMapError exception

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add a decorator utility module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

make the client use the new NetworkError exception

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create a new 'AgentError' base exception

Signed-off-by: Alfredo Deza <adeza@redhat.com>

move the log module into util

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create a utility module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

1.2.1

update the changelog

Signed-off-by: Alfredo Deza <adeza@redhat.com>

bumping to 1.2.1

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20 from ceph/wip-9686-sync-marker

new sync marker and `get_worker_bound` API

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

worker will now use the new get_worker_bound values

Signed-off-by: Alfredo Deza <adeza@redhat.com>

better comment on default values for worker bound

Signed-off-by: Alfredo Deza <adeza@redhat.com>

incrementalSync will use the new worker_bound behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>

use the right key for timestamp: oldest_time

Signed-off-by: Alfredo Deza <adeza@redhat.com>

tests for the new get_worker_bounds behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>

move the constants from worker.py to constants.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

move constant values to a new module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

make get_worker_bound return a dict to support new keys automatically

Signed-off-by: Alfredo Deza <adeza@redhat.com>

fix attribute errors for exceptions in worker.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #21 from ktdreyer/rpm-dist

packaging: use dist tag in RPM Release field

packaging: use dist tag in RPM Release field

This change adds ".el6", ".el7", etc to the end of the RPM filename.

More information about the dist tag can be found at
http://fedoraproject.org/wiki/Packaging:DistTag

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #19 from ceph/wip-10584

fix exception attribute errors in sync.py

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

fix exception attribute errors in sync.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18 from ceph/wip-10569

exceptions refactoring

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

fix the worker exception tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add the worker exceptions into their own module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

get worker exceptions into the exceptions module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

update tests for the new module changes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

create an exceptions module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove exceptions and errors from client onto exceptions.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add tests for listing objects in buckets with real http body from responses

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Merge pull request #17 from ktdreyer/spec-cleanup

packaging: RPM .spec cleanup

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

packaging: use full PyPI URL for RPM Source tarball

Alfredo has published the radosgw-agent package to PyPI today. Add the
full URL.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: add missing build deps to RPM

In order to build the radosgw-agent package in mock, the python-devel
and python-setuptools packages must be present in the buildroot.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: remove RPM defattr setting

The %defattr setting in the %files list is only necessary for very old
versions of RPM.  For RHEL 6 and later, it's fine to leave it out.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: remove RPM %clean section

The %clean section is only necessary for very old versions of RPM. For
RHEL 6 and later, it's fine to leave it out.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: simplify RPM %setup section

Remove duplicated -n settings to %setup. There were two that both did
the same thing, and neither one is needed, since %setup unpacks
%{name}-%{version} by default.

Make %setup run quietly (with -q). This fixes an rpmlint warning.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: add RPM %description

Add a simple description to the RPM packaging. This phrase was
more-or-less taken from the docs in ceph.git.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: use upper-case URL tag in RPM

Either "Url:" or "URL:" works, but the Fedora project has standardized
on "URL:"

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: rm unnecessary RPM tags

The BuildRoot tag is only needed on RHEL 5. RHEL 6 and newer are fine
without this setting.

The %{_prefix} macro is already /usr by default.

The Vendor tag is something that should be left undefined in the
packaging .spec itself. Ordinarily this is added during the build by the
build system.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

packaging: rm unnecessary RPM macro definitions

Remove the extraneous macro definitions, %{name}, %{version},
%{unmangled_version}, %{release}. RPM automatically sets these when we
use the Name, Version, and Release tags.

Discussed %{unmangled_version} in #ceph-devel today; not sure where that
one came from.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

create a changelog

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

update author email

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Merge pull request #16 from ktdreyer/fixes-from-v1.2

#10518: add fixes from v1.2

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Update Debian Changelog.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
(cherry picked from commit cc845481a60d6d0da92879e5ef411d8d89c572f3)

Branch for 1.2 release.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
(cherry picked from commit 17fa0f7dc85f658547dd79a5416f55fa709a4956)

Merge pull request #15 from ceph/wip-socket-leak

client: workaround socket leak

client: workaround socket leak

Recreate the boto connection object after every 512 requests in an
attempt to workaround a socket leak when the remote end closes the
connection. Boto does not seem to be causing httplib to close() the
sockets appropriately in some cases. I'm not sure exactly where
the leak is occurring, but forcing boto to reinitialize its
connection pool like this avoids it.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14 from ceph/wip-full-sync

Remove dependency on requests and other full sync fixes

worker: check op state for progress on any HTTP error

500 can be caused by an fcgi timeout when the operation still ends up
succeeding. It doesn't hurt to check for op state being in progress in
general in case other error codes happen but don't indicate actual
failure of the copy, since wait_for_object fails immediately if the
op state is not in progress.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

sync: use ' ' as the default shard so retries can be stored

The replica log api doesn't allow empty markers, but we may need to
store retries in the replica log even if there are no log entries. Use
' ' as a marker instead, since it is before all possible markers.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

full data sync: treat 404 from bucket list as success

Since results are paged, the exception may not occur immediately, so
add a new type to distinguish it from other client exceptions, and
catch it at any point during full sync of a bucket.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

worker: ignore 404 when removing op_state

It's fine if the gateway deleted it first.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

re-purpose urlencode to craft the query string for us without quoting again

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

create a test to check/prevent double quoting of urls

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

add get_bucket_list test

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

tests for get API calls

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

pep8/style changes to client.py

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

http response does not have .raw anymore

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Merge pull request #13 from ceph/wip-full-sync

Bug fixes for full sync and a few other data sync cases

client: return object names instead of boto resultsets

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Add tests for boto_call decorator

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

More tests for wait_for_object()

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Immediately fail when op state is not found

There's a very narrow window in which our connection sending the copy
request could fail before radosgw records the op state, and we query
the op state in that time. Retrying later at that point is fine, and
avoids a long timeout if there is a bug causing us to reach this point
without the op state ever being recorded. This race hasn't been
observed in practice.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Treat HTTP 500 as any other HTTP error for a copy request

We still want to retry later, since 500 could be the gateway timing
out, or it being upgraded to fix a bug, etc.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

client: escape copy source header

Need to esacpe the x-amz-copy-source header.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

worker: process full sync for buckets with no log

This way we catch objects that were uploaded before zones or data logs
were set up. If there are actually no objects in the bucket, the
bucket listing will tell us that.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

client: don't hide boto return values

The decorator for translating boto exceptions needs to return
explicitly, or it will lose the return value of the original function,
and cause full sync to fail for objects that aren't in the bucket
index log.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

add a test that ensures that wait_for_object is not called if object is not found

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

by default object is not found, changing only when sync_object_intra_region() works

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

raise SyncFailed on transient errors

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

allow a stringified version of the http status code

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

tests for sync_object in worker.py

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

configure logging for testing

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

log the exception as error

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Merge pull request #10 from ceph/wip-coverage

add parametrization for endpoint tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

add parametrization for endpoint tests

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

client: fix unicode handling

Use unicode format strings, and don't url-encode the object copy path
before passing it to requests, which will url-encode it for us.

Partially-fixes: #8202
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

logrotate: don't rotate empty logs

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

debian: fix init script installation

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

packaging: fix up spec file for non-python files

Correct the installation of the binary, logrotate conf, and init script.
Stop using INSTALLED_FILES. Instead explicitly list files and glob the
python package, as recommended by Fedora's python packaging guide.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

cli: add a default log file

Now that the packages include /var/log/ceph/radosgw-agent, default to
storing logs there, named after the configuration file or
radosgw-agent.log if no config file was used.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

debian: depend on python 2.6 or higher

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

packaging: add logrotate config

There's no need to signal running agents since they use
a WatchedFileHandler that detects rotation itself.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

packaging: add init scripts

create log, run, and etc dirs as well

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

packaging: add dependency on python-requests

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>