rgw: change naming of utility functions in rgw_keystone.cc.

Slightly improve code formatting and const-correctness as well.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: add support for caching of Keystone admin token.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: ONLY move PKI-related things from rgw_swift to rgw_keystone.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #7723 from tchaikov/wip-rgw-test

test/radosgw-admin: update the expected usage outputs

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7647 from yuyuyu101/wip-spdk

NVMEDevice: refactor probe/attach codes and support zero command

Reviewed-by: Kefu Chai <kchai@redhat.com>

NVMEDevice: accept spdk api name changes

Signed-off-by: Haomai Wang <haomai@xsky.com>

spdk: update spdk submodule to accept new interface changes

Signed-off-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: add default constructor for Task and clean up

Signed-off-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: only activate zero command when backend supported

Signed-off-by: Haomai Wang <haomai@xsky.com>

BlockDevice: detect symbol file basename

Signed-off-by: Haomai Wang <haomai@xsky.com>

BlueStore: fix spdk link file create failed

Signed-off-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: fix atomic and lock changes

Signed-off-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: use nvme_probe to simply init

Signed-off-by: Haomai Wang <haomai@xsky.com>

spdk: update spdk submodule

Signed-off-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: use nvme zero command instead of writing zero

Signed-off-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: cleanup task iterator process

Signed-off-by: Haomai Wang <haomai@xsky.com>

test/radosgw-admin: update the expected usage outputs

to match with the latest radosgw-admin changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7627 from dillaman/wip-librbd-read-replay

librbd: block read requests until journal replayed

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7718 from dillaman/wip-librbd-mirror-peer

librbd: update of mirror pool mode and mirror peer handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7698 from dillaman/wip-librbd-replay-flush

librbd: differentiate journal replay flush vs shut down

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7720 from ceph/wip-rgw-aws4

rgw support for aws authentication v4 (Javier M. Mellid)

Merge remote-tracking branch 'jmunhoz/wip-aws4' into wip-aws4

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Conflicts:
src/rgw/rgw_auth_s3.h
src/rgw/rgw_client_io.cc
src/rgw/rgw_client_io.h
src/rgw/rgw_common.h
src/rgw/rgw_main.cc
src/rgw/rgw_op.cc
src/rgw/rgw_op.h
src/rgw/rgw_rest.cc
src/rgw/rgw_rest_metadata.cc
src/rgw/rgw_rest_s3.cc

Merge pull request #7719 from ceph/wip-rgw-keystone-v3

rgw keystone v3 (Mark Barnes, Radoslaw Zarzynski)

support for keystone v3

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: update mirror peer data structure

Replace cluster uuid with mirror peer uuid and support an
optional pool id within a peer (when using internal APIs
for testing).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: support image and pool-level mirroring modes

If pool-level mirroring is enabled, all journaled images within
the pool are eligible for mirroring.  If image-level mirroring
is enabled, mirroring needs to be explicitly enabled on each
image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: Keystone token parsing doesn't need to know API version.

This patch targets an issue with S3Extension in Keystone:
requested tokens always conform to Keystone API v2 - regardless
of the version used to make the request.

Previous implementation of KeystoneToken::parse() as well
as KeystoneToken::decode_json() had to know the API version
explicitly. Thus, they might be affected in the future by
changes in S3-compatibility middleware of Keystone.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: S3 always must use token format of Keystone v2.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: Keystone token parsing should fail on misformed JSONs.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: improve debugs in RGWPostHTTPData.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: ONLY formatting improvements for Keystone-related code.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: handle Keystone API version with dedicated enum.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: fix wrong format of admin token request in Keystone v3.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: rework serialization in token retrieval for Keystone's admin.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: fix Keystone v2 regression in role's JSON parsing

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: accommodate Keystone v3 in S3-related auth mechanisms.

This patch is based on changes proposed by Mark Barnes
<mark.barnes@ocado.com> (lperiquito) in commit
af71f6a518529ea6cccb25bd46da2b6d1458c1da.

Signed-off-by: Mark Barnes <mark.barnes@ocado.com>
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: accommodate Keystone v3 in RGWSwift class.

This patch is based on changes proposed by Mark Barnes
<mark.barnes@ocado.com> (lperiquito) in commit
af71f6a518529ea6cccb25bd46da2b6d1458c1da.

Signed-off-by: Mark Barnes <mark.barnes@ocado.com>
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: basic data structures for Keystone v3.

The change has been extracted from changeset proposed
in commit af71f6a518529ea6cccb25bd46da2b6d1458c1da by
Mark Barnes <mark.barnes@ocado.com> (lperiquito).

Signed-off-by: Mark Barnes <mark.barnes@ocado.com>
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: unify RGWValidateKeystoneToken with RGWPostHTTPData.

The change has been extracted from changeset proposed
in commit af71f6a518529ea6cccb25bd46da2b6d1458c1da by
Mark Barnes <mark.barnes@ocado.com> (lperiquito).

Signed-off-by: Mark Barnes <mark.barnes@ocado.com>
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: make RGWPostHTTPData able to extract X-Subject-Token.

The change has been extracted from changeset proposed
in commit af71f6a518529ea6cccb25bd46da2b6d1458c1da by
Mark Barnes <mark.barnes@ocado.com> (lperiquito).

Signed-off-by: Mark Barnes <mark.barnes@ocado.com>
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

librbd: read ops require journal replay

When journaling is enabled, a read op should acquire the exclusive
lock and replay the journal in case uncommitted writes are stored
in the journal.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7717 from cbodley/wip-cmake-rbd-image-watcher

cmake: add missing librbd image_watcher sources

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

cmake: add missing librbd image_watcher sources

Fixes: #14823
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7709 from ceph/wip-rgw-new-multisite-merge

rgw multisite v2 (Yehuda Sadeh, Orit Wasserman, Casey Bodley)
 - rename region to zonegroup
 - new zone and zonegroup configuration
 - new realm and period structures
 - rgw handles synchronization (no need for sync agent)
 - active-active data

Signed-off-by: Yehuda Sadeh

Merge pull request #7371 from theanalyst/mon/quota-msg-cleanup

mon: cleanup set-quota error msg

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7708 from jdurgin/wip-cython-cleanup

pybind: replace __del__ with __dealloc__ for rbd

Reviewed-by: Hector Martin <marcan@marcan.st>
Reviewed-by: Mehdi Abaakouk <sileht@sileht.net>

mon: cleanup set-quota error msg

When we send in an unrecognized field for set-quota, specify that the
options should be `max_bytes` or `max_objects`. This doesn't affect the
ceph-cli as the choices are caught at an earlier stage, however the
error message is seen while reaching the cluster via rados api for eg.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #7668 from dillaman/wip-librbd-replay-locks

librbd: use async librados notifications

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: avoid empty object names

Don't try to sync an object with empty name (shouldn't happen, but
bugs may happen). Also, don't try to write_meta() of an empty object
name, in case we get there through a different path somehow.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

.gitignore: include recently added rgw programs

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

rgw: fix s3 list bucket (affects format=json)

another merge casualty. The section wasn't opened properly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

pybind: replace __del__ with __dealloc__ for rbd

Cython extension types like these don't call __del__, but use
__dealloc__ instead:

http://docs.cython.org/src/userguide/special_methods.html#finalization-method-dealloc

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7706 from sileht/sileht/rados-cython

pybind: remove next() on iterators

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: only validate bucket name if not empty

This got broken due to code movement

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: resurrect lost code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

pybind: remove next() on iterator

Cython recommends to implement only __next__() methods.

http://docs.cython.org/src/userguide/special_methods.html#the-next-method

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Merge pull request #7641 from batrick/vstart-usage

Clarify usage on starting single osd/mds/mon.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test/Makefile-client.am: adjustment following merge

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: helper method for flushing journal commit positions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: added interface to flush pending commit position updates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7692 from sileht/sileht/rados-cython

pybind/rados: use __dealloc__ since __del__ is ignored by cython

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'origin/master' into wip-rgw-new-multisite

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Conflicts:
src/CMakeLists.txt
src/rgw/Makefile.am
src/rgw/rgw_admin.cc
src/rgw/rgw_common.h
src/rgw/rgw_main.cc
src/rgw/rgw_op.cc
src/rgw/rgw_rados.h
src/rgw/rgw_rest_s3.cc
src/test/Makefile-client.am

librbd: differentiate journal replay flush vs shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: add drain_all() before exiting a cr

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: silence compilation #warnings

these are now FIXMEs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more error reporting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync error handling (for sync init)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a bit better hashing

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: don't hold owner lock while refreshing image synchronously

The write lock will be taken when the new state is applied.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: need owner write lock when disabling exclusive lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: correct memory leaks discovered via valgrind

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: notifications should be flushed between exclusive lock states

Avoid leaving in-flight notification messages when transitioning lock
states.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: RGWMetaSyncCR loops through period history

RGWMetaSyncCR uses a period history Cursor to track its position. it
uses this to get the max sync markers for each shard from the following
period, so that RGWMetaSyncShardCR knows when to stop syncing and return
control to RGWMetaSyncCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: get period history cursor when starting sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove RGWMetadataManager::store_md_log_entries

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rados coroutines take bucket by const ref

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add first log period to mdlog info response

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: factor get_log_shard_id() out of RGWMetadataLog

by using RGWRados::key_to_shard_id(), we can avoid the dependency on
RGWMetadataLog's prefix (and period)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add period id to rgw_rest_log operations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw-admin: update callers of get_log

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add map of period_id -> RGWMetadataLog

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add period id to metadata log oids

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWRemoteMetaLog passes period id with requests to master

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move sync_status into RGWRemoteMetaLog

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: include period id in rgw_meta_sync_info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use std::move for strings in RGWFetchAllMetaCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: remove duplicate rgw source files

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove RemoteMetaLog::fetch() and admin command

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused RGWRemoteMetaLog::list_shards

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused class RGWReadMDLogShardInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused clone_shards() in metadata sync

clone_shards() was an exact duplicate for fetch()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unneeded RGWRados arg for RGWMetadataLog

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: we need to clear the id in create_default in case of a race

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw-admin: command to list sync error logs

$ radosgw-admin sync error list

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: set quota when converting regionmap

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: update image name upon rename operations

Previously the image could not have been renamed twice without
re-opening the image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: missing owner lock on snap rollback cache invalidation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: journal replay should execute ops in clean context

lockdep will complain about loop cycles that won't cause an
issue in reality as replay and record are two different
journal states.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use AIO notifications to prevent blocking ops

If two or more images share the same CephContext, notifications
from one image can block the work queue which will potentially
block acknowledging the notification until after it times out.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: cleanup header update notifications

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Revert "librbd: lock notifications should be executed outside librados thread"

This reverts commit d898995b0e3ea301b1325f68a0532d57afa3c816.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>