tests/ceph-disk: Using dummy device mappers

When doing a make test, if your local setup was running a "dm-0" or "dm-1",
the make check was failing.

The detection of the local "dm-x" were leading to a wrong comparaison with
the expected simulated values.

The fix is using a dummy name for that "dm" to prevent any collision with the local setup.

Prior that patch, a typical output of that error looked like :

>               assert expect == main.list_devices()
E               assert [{'partitions...: '/dev/Xda'}] == [{'partitions'...: '/dev/Xda'}]
E                 At index 0 diff: {'partitions': [{'dmcrypt': {'holders': ['dm-0'], 'type': 'plain'}, 'ptype': '4fbd7e29-9d25-41b8-afd0-5ec00ceff05d', 'path': '/dev/Xda1', 'is_partition': True, 'mount': None, 'uuid': '56244cf5-83ef-4984-888a-2d8b8e0e04b2', 'type': 'data', 'state': 'unprepared', 'fs_type': None}], 'path': '/dev/Xda'} != {'partitions': [{'ptype': '4fbd7e29-9d25-41b8-afd0-5ec00ceff05d', 'whoami': None, 'path': '/dev/Xda1', 'is_partition': True, 'mount': '/var/cache/ccache', 'uuid': '56244cf5-83ef-4984-888a-2d8b8e0e04b2', 'ceph_fsid': None, 'fs_type': 'btrfs', 'dmcrypt': {'holders': ['dm-0'], 'type': 'plain'}, 'type': 'data', 'state': 'active'}], 'path': '/dev/Xda'}
E                 Full diff:
E                 - [{'partitions': [{'dmcrypt': {'holders': ['dm-0'], 'type': 'plain'},
E                 + [{'partitions': [{'ceph_fsid': None,
E                 +                   'dmcrypt': {'holders': ['dm-0'], 'type': 'plain'},
E                 -                   'fs_type': None,
E                 ?                              ^^^^
E                 +                   'fs_type': 'btrfs',
E                 ?                              ^^^^^^^
E                 'is_partition': True,
E                 -                   'mount': None,
E                 +                   'mount': '/var/cache/ccache',
E                 'path': '/dev/Xda1',
E                 'ptype': '4fbd7e29-9d25-41b8-afd0-5ec00ceff05d',
E                 -                   'state': 'unprepared',
E                 ?                             ^^^^ -----
E                 +                   'state': 'active',
E                 ?                             ^^^^^
E                 'type': 'data',
E                 -                   'uuid': '56244cf5-83ef-4984-888a-2d8b8e0e04b2'}],
E                 ?                                                                 --
E                 +                   'uuid': '56244cf5-83ef-4984-888a-2d8b8e0e04b2',
E                 +                   'whoami': None}],
E                 'path': '/dev/Xda'}]

tests/test_main.py:342: AssertionError

Signed-off-by: Erwan Velu <erwan@redhat.com>

tests/ceph-disk: Creating missing working dir

When running run-tox.sh in a very simple env,
the test will fail if '/var/lib/ceph/tmp' doesn't exist.

This patch adds a check to create this directory if required as mkdtemp doesn't do it for you.

Prior this patch, the following behavior was seen :

tests/test_main.py:342:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ceph_disk/main.py:3753: in list_devices
    fstype=fs_type, options='')
ceph_disk/main.py:1217: in mount
    dir=STATEDIR + '/tmp',
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

suffix = '', prefix = 'mnt.', dir = '/var/lib/ceph/tmp'

    def mkdtemp(suffix="", prefix=template, dir=None):
        """User-callable function to create and return a unique temporary
        directory.  The return value is the pathname of the directory.

        Arguments are as for mkstemp, except that the 'text' argument is
        not accepted.

        The directory is readable, writable, and searchable only by the
        creating user.

        Caller is responsible for deleting the directory when done with it.
        """

        if dir is None:
            dir = gettempdir()

        names = _get_candidate_names()

        for seq in xrange(TMP_MAX):
            name = names.next()
            file = _os.path.join(dir, prefix + name + suffix)
            try:
>               _os.mkdir(file, 0700)
E               OSError: [Errno 2] No such file or directory: '/var/lib/ceph/tmp/mnt.KoAV85'

/usr/lib64/python2.7/tempfile.py:333: OSError

Signed-off-by: Erwan Velu <erwan@redhat.com>

Merge pull request #7649 from yuyuyu101/wip-async-watch

librados: implement async watch/unwatch

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7761 from dillaman/wip-14847

librbd: fix state machine race conditions during shut down

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

RadosClient: call watch_flush before finisher stop

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7632 from zaitcev/cleanup-dead-code-3

rgw: Drop unused usage_exit from rgw_admin.cc

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7285 from magicrobotmonkey/patch-1

Increase verbosity level on RGWObjManifest line

Reviewed-by: Yehuda Saeh <yehuda@redhat.com>

Merge pull request #7618 from zaitcev/wip-5073-subuser-bis

rgw: Fix subuser harder with tenants

Reviewed-y: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7763 from dachary/wip-releases

doc: fix 0.94.4 and 0.94.5 ordering

Merge pull request #7364 from javenwu/master

common: default cluster name to config file prefix

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7548 from yuyuyu101/librbd-avoid-memcpy

librbd: reduce mem copies to user-buffer during read

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7667 from jdurgin/wip-14780

Revert "librbd: use task finisher per CephContext"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7625 from branch-predictor/bp-async-more

msg/async: smarter MSG_MORE

Reviewed-by: Haomai Wang <haomai@xsky.com>

doc: fix 0.94.4 and 0.94.5 ordering

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7760 from theanalyst/fix/rgw-fcgi-build

rgw: fcgi should include acconfig

librbd: deterministically unblock writes on snap set

Previously this was handled in the state machine's destructor,
but that can lead to nondeterministic behavior due to interactions
between threads.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: avoid close race-condition within ImageState

The lock was previously unlocked/locked to avoid lock cycles. This
creates an issue when closing the image because the another thread
might have already deleted the ImageState before the lock could be
re-locked.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: fcgi should include acconfig

As it references FASTCGI_INCLUDE_DIR which is referenced from it

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc/release-notes: v0.94.6 final notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7689 from dachary/wip-release-notes

release-notes: draft v0.94.6 release notes

msg/async: cut the middle-man

Get rid of complete_bl and let messenger write directly to outcoming_bl of
a connection. Also, if message bufferlist is small enough, append its
contents to outcoming_bl directly, so we'll use less iovecs and in best
case, pack entire message (together with header and footer added in
write_message()) in single bufferptr.

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #7725 from ceph/wip-fix-async-valgrind

AsyncMessenger: fix valgrind leak

Merge pull request #7758 from dachary/wip-ceph-disk-prepare-help

ceph-disk: fix prepare --help

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: fix prepare --help

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7756 from dachary/wip-ceph-disk-args

ceph-disk: deactivate / destroy PATH arg are optional

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: deactivate / destroy PATH arg are optional

And the mandatory positional arguments need not be == 1, it is enough to
leave the nargs parameter out.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7687 from yangdongsheng/ceph-disk_nargs

ceph-disk: make some arguments as required if necessary

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7640 from batrick/vstart-btrfs-missing

vstart.sh: silence a harmless msg where btrfs is not found

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7737 from trociny/wip-librados_test_stub-debug

test: more debug logging for TestWatchNotify

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7714 from SUSE/wip-qa-typo

doc: standardize @param (not @parma, @parmam, @params)

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #7738 from trociny/test-librbd-poolname

test: create pools for rbd tests with different prefix

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7727 from jmunhoz/ceph-fix-14829

vstart.sh: avoid race condition starting rgw via vstart.sh

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7614 from trociny/wip-rbd-mirroring-image-reply

rbd-mirror: implement ImageReplayer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7744 from dachary/wip-14838-ceph-disk-dmcrypt

ceph-disk: s/dmcrpyt/dmcrypt/

Reviewed-by: Nathan Cutler <ncutler@suse.com>

ceph-disk: s/dmcrpyt/dmcrypt/

http://tracker.ceph.com/issues/14838 Fixes: #14838

Signed-off-by: Frode Sandholtbraaten <fsa@basefarm.no>
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7688 from tchaikov/wip-14788

buffer: hide iterator_impl symbols

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: create pools for rbd tests with different prefix

This makes easier to find out what test fails to cleanup.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #7723 from tchaikov/wip-rgw-test

test/radosgw-admin: update the expected usage outputs

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test_rados_watch_notify: add async_watch_flush tests

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7647 from yuyuyu101/wip-spdk

NVMEDevice: refactor probe/attach codes and support zero command

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: more debug for TestWatchNotify

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

NVMEDevice: accept spdk api name changes

Signed-off-by: Haomai Wang <haomai@xsky.com>

spdk: update spdk submodule to accept new interface changes

Signed-off-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: add default constructor for Task and clean up

Signed-off-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: only activate zero command when backend supported

Signed-off-by: Haomai Wang <haomai@xsky.com>

BlockDevice: detect symbol file basename

Signed-off-by: Haomai Wang <haomai@xsky.com>

BlueStore: fix spdk link file create failed

Signed-off-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: fix atomic and lock changes

Signed-off-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: use nvme_probe to simply init

Signed-off-by: Haomai Wang <haomai@xsky.com>

spdk: update spdk submodule

Signed-off-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: use nvme zero command instead of writing zero

Signed-off-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: cleanup task iterator process

Signed-off-by: Haomai Wang <haomai@xsky.com>

vstart.sh: avoid race condition starting rgw via vstart.sh

In slow environments running vstart.sh hits a race condition setting up
the rgw server. Creating the S3 and Swift users before booting rgw
avoids the issue.

Fixes: #14829
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

test/radosgw-admin: update the expected usage outputs

to match with the latest radosgw-admin changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: rbd-mirror: add tool to test ImageReplayer

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test: rbd-mirror: add ImageReplayer tests

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: implement ImageReplayer

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librados: add async_watch_flush api

Signed-off-by: Haomai Wang <haomai@xsky.com>

RadosClient: add async watch_flush version

Signed-off-by: Haomai Wang <haomai@xsky.com>

Objecter: make linger_callback_flush accept context instead of blocking

Signed-off-by: Haomai Wang <haomai@xsky.com>

librados: add C api impl and tracing codes

Signed-off-by: Haomai Wang <haomai@xsky.com>

librados: remove unused variables

Signed-off-by: Haomai Wang <haomai@xsky.com>

test: add aio_unwatch unittest

Signed-off-by: Haomai Wang <haomai@xsky.com>

librados: support aio_unwatch api

Signed-off-by: Haomai Wang <haomai@xsky.com>

librados: add unittest for rados aio watch

Signed-off-by: Haomai Wang <haomai@xsky.com>

librados: add async watch api

Signed-off-by: Haomai Wang <haomai@xsky.com>

journal: wrappers to get journal metadata

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: allow to use journal replay externally

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cls::journal: don't print client data

It makes `rbd journal status` output look ugly.

It is a temporary solution and is going to be improved in
the future (it could be useful to see data but decoded).

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test/bufferlist:Add test case for  bufferlist::is_provided_buffer.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Striper: Add function 'assemble_result(CephContext *cct, char *buffer, size_t len)' in StripedReadResult.

For sparse read, it can avoid copying zero around.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Striper: Add total_intended_len in StripedReadResult

Using this field record the total len which we wanted.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osdc/objectCacher: Remove the unsed field.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

bufferlist: Add new func is_provided_buffer(const char *dst)

Using this function to judge whether copy data from bufferlist to
dst. It mainly used librados C interface to avoid copy data from librados
to caller.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #7627 from dillaman/wip-librbd-read-replay

librbd: block read requests until journal replayed

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7718 from dillaman/wip-librbd-mirror-peer

librbd: update of mirror pool mode and mirror peer handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7698 from dillaman/wip-librbd-replay-flush

librbd: differentiate journal replay flush vs shut down

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7720 from ceph/wip-rgw-aws4

rgw support for aws authentication v4 (Javier M. Mellid)

Merge remote-tracking branch 'jmunhoz/wip-aws4' into wip-aws4

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Conflicts:
src/rgw/rgw_auth_s3.h
src/rgw/rgw_client_io.cc
src/rgw/rgw_client_io.h
src/rgw/rgw_common.h
src/rgw/rgw_main.cc
src/rgw/rgw_op.cc
src/rgw/rgw_op.h
src/rgw/rgw_rest.cc
src/rgw/rgw_rest_metadata.cc
src/rgw/rgw_rest_s3.cc

Merge pull request #7719 from ceph/wip-rgw-keystone-v3

rgw keystone v3 (Mark Barnes, Radoslaw Zarzynski)

support for keystone v3

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: update mirror peer data structure

Replace cluster uuid with mirror peer uuid and support an
optional pool id within a peer (when using internal APIs
for testing).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: support image and pool-level mirroring modes

If pool-level mirroring is enabled, all journaled images within
the pool are eligible for mirroring.  If image-level mirroring
is enabled, mirroring needs to be explicitly enabled on each
image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: Keystone token parsing doesn't need to know API version.

This patch targets an issue with S3Extension in Keystone:
requested tokens always conform to Keystone API v2 - regardless
of the version used to make the request.

Previous implementation of KeystoneToken::parse() as well
as KeystoneToken::decode_json() had to know the API version
explicitly. Thus, they might be affected in the future by
changes in S3-compatibility middleware of Keystone.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: S3 always must use token format of Keystone v2.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: Keystone token parsing should fail on misformed JSONs.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: improve debugs in RGWPostHTTPData.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: ONLY formatting improvements for Keystone-related code.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: handle Keystone API version with dedicated enum.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: fix wrong format of admin token request in Keystone v3.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: rework serialization in token retrieval for Keystone's admin.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: fix Keystone v2 regression in role's JSON parsing

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: accommodate Keystone v3 in S3-related auth mechanisms.

This patch is based on changes proposed by Mark Barnes
<mark.barnes@ocado.com> (lperiquito) in commit
af71f6a518529ea6cccb25bd46da2b6d1458c1da.

Signed-off-by: Mark Barnes <mark.barnes@ocado.com>
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: accommodate Keystone v3 in RGWSwift class.

This patch is based on changes proposed by Mark Barnes
<mark.barnes@ocado.com> (lperiquito) in commit
af71f6a518529ea6cccb25bd46da2b6d1458c1da.

Signed-off-by: Mark Barnes <mark.barnes@ocado.com>
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: basic data structures for Keystone v3.

The change has been extracted from changeset proposed
in commit af71f6a518529ea6cccb25bd46da2b6d1458c1da by
Mark Barnes <mark.barnes@ocado.com> (lperiquito).

Signed-off-by: Mark Barnes <mark.barnes@ocado.com>
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: unify RGWValidateKeystoneToken with RGWPostHTTPData.

The change has been extracted from changeset proposed
in commit af71f6a518529ea6cccb25bd46da2b6d1458c1da by
Mark Barnes <mark.barnes@ocado.com> (lperiquito).

Signed-off-by: Mark Barnes <mark.barnes@ocado.com>
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: make RGWPostHTTPData able to extract X-Subject-Token.

The change has been extracted from changeset proposed
in commit af71f6a518529ea6cccb25bd46da2b6d1458c1da by
Mark Barnes <mark.barnes@ocado.com> (lperiquito).

Signed-off-by: Mark Barnes <mark.barnes@ocado.com>
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

PendingReleaseNotes: document the dropped buffer symbols

the symbols of buffer::list::iterator_impl<> were wrongly exposed
in previous infernalis release, and the clients linked against
librados are very likely using them. so we need to document this
change.

Signed-off-by: Kefu Chai <kchai@redhat.com>

buffer: hide iterator_impl symbols

buffer::list::iterator_impl symbols are referenced by const_iterator
and iterator, and are exposed as weak symbols. if a source file is
compiled using the buffer.h, the produced object file will reference
these symbols as well, so we'd better hiding them and avoid using
them in the header file.
as a side-effect, buffer::list::const_iterator is also hidden, but
currently we don't have any librados client using this class, so
we can just leave it as an internal class at this moment.

Fixes: #14788
Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: read ops require journal replay

When journaling is enabled, a read op should acquire the exclusive
lock and replay the journal in case uncommitted writes are stored
in the journal.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7717 from cbodley/wip-cmake-rbd-image-watcher

cmake: add missing librbd image_watcher sources

Reviewed-by: Orit Wasserman <owasserm@redhat.com>