test/radosgw-admin: update the expected usage outputs

to match with the latest radosgw-admin changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7717 from cbodley/wip-cmake-rbd-image-watcher

cmake: add missing librbd image_watcher sources

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

cmake: add missing librbd image_watcher sources

Fixes: #14823
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7709 from ceph/wip-rgw-new-multisite-merge

rgw multisite v2 (Yehuda Sadeh, Orit Wasserman, Casey Bodley)
 - rename region to zonegroup
 - new zone and zonegroup configuration
 - new realm and period structures
 - rgw handles synchronization (no need for sync agent)
 - active-active data

Signed-off-by: Yehuda Sadeh

Merge pull request #7371 from theanalyst/mon/quota-msg-cleanup

mon: cleanup set-quota error msg

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7708 from jdurgin/wip-cython-cleanup

pybind: replace __del__ with __dealloc__ for rbd

Reviewed-by: Hector Martin <marcan@marcan.st>
Reviewed-by: Mehdi Abaakouk <sileht@sileht.net>

mon: cleanup set-quota error msg

When we send in an unrecognized field for set-quota, specify that the
options should be `max_bytes` or `max_objects`. This doesn't affect the
ceph-cli as the choices are caught at an earlier stage, however the
error message is seen while reaching the cluster via rados api for eg.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #7668 from dillaman/wip-librbd-replay-locks

librbd: use async librados notifications

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: avoid empty object names

Don't try to sync an object with empty name (shouldn't happen, but
bugs may happen). Also, don't try to write_meta() of an empty object
name, in case we get there through a different path somehow.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

.gitignore: include recently added rgw programs

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

rgw: fix s3 list bucket (affects format=json)

another merge casualty. The section wasn't opened properly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

pybind: replace __del__ with __dealloc__ for rbd

Cython extension types like these don't call __del__, but use
__dealloc__ instead:

http://docs.cython.org/src/userguide/special_methods.html#finalization-method-dealloc

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7706 from sileht/sileht/rados-cython

pybind: remove next() on iterators

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: only validate bucket name if not empty

This got broken due to code movement

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: resurrect lost code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

pybind: remove next() on iterator

Cython recommends to implement only __next__() methods.

http://docs.cython.org/src/userguide/special_methods.html#the-next-method

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Merge pull request #7641 from batrick/vstart-usage

Clarify usage on starting single osd/mds/mon.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test/Makefile-client.am: adjustment following merge

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7692 from sileht/sileht/rados-cython

pybind/rados: use __dealloc__ since __del__ is ignored by cython

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'origin/master' into wip-rgw-new-multisite

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Conflicts:
src/CMakeLists.txt
src/rgw/Makefile.am
src/rgw/rgw_admin.cc
src/rgw/rgw_common.h
src/rgw/rgw_main.cc
src/rgw/rgw_op.cc
src/rgw/rgw_rados.h
src/rgw/rgw_rest_s3.cc
src/test/Makefile-client.am

rgw: add drain_all() before exiting a cr

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: silence compilation #warnings

these are now FIXMEs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more error reporting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync error handling (for sync init)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a bit better hashing

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: don't hold owner lock while refreshing image synchronously

The write lock will be taken when the new state is applied.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: need owner write lock when disabling exclusive lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: correct memory leaks discovered via valgrind

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: notifications should be flushed between exclusive lock states

Avoid leaving in-flight notification messages when transitioning lock
states.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: RGWMetaSyncCR loops through period history

RGWMetaSyncCR uses a period history Cursor to track its position. it
uses this to get the max sync markers for each shard from the following
period, so that RGWMetaSyncShardCR knows when to stop syncing and return
control to RGWMetaSyncCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: get period history cursor when starting sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove RGWMetadataManager::store_md_log_entries

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rados coroutines take bucket by const ref

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add first log period to mdlog info response

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: factor get_log_shard_id() out of RGWMetadataLog

by using RGWRados::key_to_shard_id(), we can avoid the dependency on
RGWMetadataLog's prefix (and period)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add period id to rgw_rest_log operations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw-admin: update callers of get_log

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add map of period_id -> RGWMetadataLog

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add period id to metadata log oids

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRemoteMetaLog passes period id with requests to master

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move sync_status into RGWRemoteMetaLog

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: include period id in rgw_meta_sync_info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use std::move for strings in RGWFetchAllMetaCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: remove duplicate rgw source files

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove RemoteMetaLog::fetch() and admin command

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused RGWRemoteMetaLog::list_shards

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused class RGWReadMDLogShardInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused clone_shards() in metadata sync

clone_shards() was an exact duplicate for fetch()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unneeded RGWRados arg for RGWMetadataLog

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: we need to clear the id in create_default in case of a race

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw-admin: command to list sync error logs

$ radosgw-admin sync error list

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: set quota when converting regionmap

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: update image name upon rename operations

Previously the image could not have been renamed twice without
re-opening the image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: missing owner lock on snap rollback cache invalidation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: journal replay should execute ops in clean context

lockdep will complain about loop cycles that won't cause an
issue in reality as replay and record are two different
journal states.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use AIO notifications to prevent blocking ops

If two or more images share the same CephContext, notifications
from one image can block the work queue which will potentially
block acknowledging the notification until after it times out.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: cleanup header update notifications

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Revert "librbd: lock notifications should be executed outside librados thread"

This reverts commit d898995b0e3ea301b1325f68a0532d57afa3c816.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: support for AIO notifications

The header update and lock notifications might be invoked
from the librados AIO thread.  Update the close state
machine to flush any potential AIO notifications.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rstats: enable by default on client; move test workunit to its own dir

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7323 from ukernel/wip-client-dirsize

client: add option to control how directory size is calculated

This lets you disable rstats if your workload is unhappy about directories
changing size (eg, tar of recently-moved/created/untarred files).

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7457 from weiqiaomiao/w1

Rgw: check the return value when call fe->run()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7666 from rzarzynski/wip-rgw-sysreq-over-swift

rgw: add support for system requests over Swift API

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7678 from linuxbox2/rgw-sigusr1

rgw: move signal.h dependency from rgw_front.h

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge branch 'greg-pr-7034'

Merge pull request #7679 from linuxbox2/rgw-ssconst

sstring.hh: return type from str_len(...) need not be const

Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind: rados cleanup

This change uses:
* a pystring to store ioctx state like other state attributes and like
  previous binding
* use __dealloc__ instead of __del__ to ensure Object are freed correctly

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Merge pull request #7684 from ceph/wip-pybind-install

pybind: use correct subdir for rados install-exec rule

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge branch 'wip-sharded-scan' of git://github.com/jcsp/ceph into greg-pr-7034

Conflicts:
src/osdc/Objecter.cc

pybind: use correct subdir for rados install-exec rule

This fixes package builds and 'make install'

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7621 from sileht/sileht/rados-cython

python binding of librados with cython

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: remove unnecessary overloads in RGWHTTPArgs.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

sstring.hh: return type from str_len(...) need not be const

Removes a compile warning.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: move signal.h dependency from rgw_front.h

Missed in refactoring of rgw_main.cc.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Update the documentation

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

librbdpy: Use new rados lib

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

ceph.in: Use new python rados module

This implements run_in_thread inside the ceph command itself.

And fixes the ceph command bootstrap when it run inside the
source tree to correctly load the new rados python module.

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Fix rpm/deb packaging

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Remove old rados pybinding

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Cython Rados module

Notable changes:

* run_in_thread have disapeared
* timeout argument of some methods are ignored
* rados_create_write_op/rados_create_read_op returns WriteOp/ReadOp
  instead of the pointer address
* rados_monitor_log callback 'arg' arguments was broken in previous python
  binding (callback was called with the pointer address instead pointed object)
* object attributes that was pointer addresses are now private and not accessible in python

Some tests have been added to cover all methods

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Merge pull request #7603 from roidayan/xio_fixes

Xio fixes

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: support system requests over Swift API.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: enable access to system arguments of RGWHTTPArgs.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Move pybind rbd module into it own directory

To allow to create a autonomous rados module with cython.
We move the current librbdpy to the rbd sub directory.

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Merge pull request #7520 from renhwsky/renhw-wip-mon-monitor

mon/monitor: some clean up

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7629 from ceph/wip-14750-op-timeouts

Objecter: detect laggy ops with objecter_timeout, not osd_timeout

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #7655 from gregsfortytwo/wip-metarequest-lock

client: removed unused Mutex from MetaRequest

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #7616 from gregsfortytwo/wip-timer-14697

mds: don't double-shutdown the timer when suiciding

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7490 from x11562/mds_mdstype_ref

mds: function parameter 'df' should be passed by reference

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7610 from dillaman/wip-bit-vector

common/bit_vector: use hard-coded value for block size

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #7653 from dillaman/wip-librbd-race-conditions

librbd: several race conditions discovered under single CPU environment

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't try to parse some forwarded requests

Not every forwarded requests are expected to get a json response,
don't pass the parser when not needed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adjust data sync async notification locking

This is a workaround to avoid issues related to the way lockdep works

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7408 from ceph/wip-14549

rgw: don't use s->bucket for metadata api path entry

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7633 from zaitcev/cleanup-various-1

rgw: cleanups to comments and messages

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

vstart.sh: clarify usage on single osd/mds/mon.

vstart.sh expects the literal string "osd"/"mds"/"mon". To me, it was confusing
as I interpreted the usage hint as an optional count for each daemon.

Signed-off-by: Patrick Donnelly <batrick@batbytes.com>

librados_test_stub: watch_flush should block until notifies complete

It shouldn't block until all the acks are received as that is not
how librados handles the method.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: lock notifications should be executed outside librados thread

Otherwise it's possible that the notification will be blocked if
the librados AIO thread isn't available to invoke the notification
handler.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: potential race on image close

If multiple states are pending, it's possible for an image to
be closed while the state lock is unlocked during the callback.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>