build/ops: rbd-replay moved from ceph-test-dbg to ceph-common-dbg

http://tracker.ceph.com/issues/13785 Fixes: #13785

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #6532 from dachary/wip-mailmap

mailmap: Ubuntu Kylin name changed to Kylin Cloud

Reviewed-by: Li Wang <li.wang@kylin-cloud.com>

Merge pull request #5848 from storage-zuiwanyuan/wip-nonblock-connect

msg/async: support of non-block connect in async messenger

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #6478 from yuyuyu101/wip-13666

msg/async: let receiver ack message ASAP

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async: support of non-block connect in async messenger

Fixes: #12802
Signed-off-by: Jianhui Yuan <zuiwanyuan@gmail.com>

Merge pull request #6534 from kylinstorage/wip-trivial-scrub-cleanup

osd: clarify the scrub result report

Reviewed-by: Kefu Chai <kchai@redhat.com>

scrub: clarify the result report

It may happen that the authoritative object
such that auth.size != be_get_ondisk_size(auth_oi.size),
in that case, clarify the error report.

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

Merge branch 'wip-py3'

pybind: a few more python 3 fixes for rbd and rados

Reviewed-by: David Coles <dcoles@gaikai.com>

pybind/rados: return pool_reverse_lookup() result as a string

This makes it symmetric with create_pool() in python 3.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

pybind/test_rbd: convert a few more str to bytes for py3

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

pybind/rbd: encode snap_rename args for py3

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

pybind/rbd: decode stat() and list_children() results for py3

For stat(), only block_name_prefix is filled in - parent and
parent_pool are always blank.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

pybind/rbd: decode parent_info() to str types for py3

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

pybind/test_rbd: fix map() usage for py3 compat

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

pybind/test_rbd: use // for division for py3

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge branch 'pybind3' of https://github.com/dcoles/ceph into wip-pybind3

pybind: Add Python 3 support for rados and rbd modules

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/pybind/rbd.py (new create args, minor fix to work with py3)

ceph: Make stdout/stderr always output Unicode (UTF-8)

If a stream is not interactive, then under Python 2, then the encoding for
stdout/stderr may be None. This means that it's not possible to print Unicode
characters since the encoding will fall back to ASCII.

This explicitly makes sys.stdout/sys.stderr always use UTF-8 encoding for
strings, regardless of the system's local or if the console is interactive or
not.
This matches the existing tests that assume that output of non-ASCII pool names
will be UTF-8 encoded.

When outputting raw binary data (such as the CRUSH-map), we must bypass the
codec and write directly to raw streams (since the new stream will only accept
ASCII byte-strings or Unicode strings).

Signed-off-by: David Coles <dcoles@gaikai.com>

pybind: Add decode_cstr helper function

This function attempts to decode a C-style string into a Python Unicode string.
It accepts an optional "size" parameter for the string length, otherwise it is
assumed that the string is NUL-terminated.

If the pointer is NULL, then this function returns None.

Signed-off-by: David Coles <dcoles@gaikai.com>

pybind: Add test for creating pool by raw UTF-8

Some clients try providing non-ASCII pool names by sending raw encoded bytes.
This check ensures that we still support this behaviour for Python 2.

In Python 3, bytestrings will fail since strings are Unicode strings and thus
clients should use Unicode escapes instead.

Signed-off-by: David Coles <dcoles@gaikai.com>

pybind: Import cstr from the rados module

Since rados is required for rbd, we can avoid duplication of code across these
two modules.

Signed-off-by: David Coles <dcoles@gaikai.com>

pybind: Don't encode str on Python 2

If you attempt to call encode on a non-ASCII string, then a UnicodeDecodeError
will be raised.

Since str on Python 2 is an 8-bit string, it's possible that it's already UTF-8
encoded. As such we should just pass it through to the C API unmodified.

On Python 3 or if the user explicitly uses unicode, then we'll encode it to
UTF-8 for them.

Signed-off-by: David Coles <dcoles@gaikai.com>

Merge branch 'wip-13504' of https://github.com/trociny/ceph

rbd: API: options on image create

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/test/librbd/test_librbd.cc (trivial, two tests added at end of file)

librbd: API: options on image create: update tests

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: API: options on image create

Pass options as key/value configuration pairs on image create instead
of expanding rbd_create/rbd_clone/rbd_copy for every possible
configuration override.

Fixes: #13504
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6557 from ceph/wip-rbd-cmake-fixes

cmake: librbd and libjournal build fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6556 from athanatos/wip-cmake-nss

cmake: add nss as a suffix for pk11pub.h

cmake: add nss as a suffix for pk11pub.h

Signed-off-by: Samuel Just <sjust@redhat.com>

librbd: provide an out-of-class definition for MAX_DESCRIPTION_OFFSET

std::min() takes its parameters by reference, so provide a definition
for OptionPrinter::MAX_DESCRIPTION_OFFSET constant.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

cmake: fix librbd and add src/journal

librbd was missing a recently added file, src/journal was missing
entirely - this brings the build back.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

mailmap: Ubuntu Kylin name changed to Kylin Cloud

Reviewed-by: Li Wang <li.wang@kylin-cloud.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: sort files

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #6517 from kylinstorage/wip-trivial-bug

osd: fix bug in last_* PG state timestamps

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6552 from trociny/wip-journal

journal: incremental improvements and fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6462 from javacruft/wip/ec-modules

Build internal plugins and classes as modules

Reviewed-by: Loic Dachary <ldachary@redhat.com>

journal: update allocated tid when skipping committed entry in player

Otherwise, if on image open, there are no any uncommitted entries in
journal, allocated tid is not updated to the latest commited and
recording always starts from tid=0.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6539 from yehudasa/wip-cmake-fix

cmake: fix files list

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6316 from dillaman/wip-13494

rbd: dynamically generated bash completion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'origin/jewel'

cmake: fix files list

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls::journal: fixup: constify dump functions

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

journal: call metadata shutdown on journal remove

This fixes error observed when disabling journal:

  JournalMetadata: journal watch error: (107) Transport endpoint is not connected

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

journal: don't use object_number when comparing positions

Only tid does matter.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

journal: make commit and flush params configurable

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

journal: allow alternate pool for journal objects

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

journal: output operators for journal types

They are going to be used for diagnostic.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6471 from mslovy/wip-check-reply-guard-again

osd: avoid double-check for replaying and can_checkpoint() in FileStore::_check_replay_guard

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6426 from xiexingguo/xxg-wip-13642

librados: fix rare race where pool op callback may hang forever

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6407 from renhwztetecs/renhw-wip-13627

mon: initialize recorded election epoch properly even when standalone

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6385 from Vicente-Cheng/mkfs-does-not-decode-an-existing-superblock

osd: fix reactivate (check OSDSuperblock in mkfs() when we already have the superblock)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6373 from liewegas/wip-promoteon2ndread-ec

librados: fix PromoteOn2ndRead test for EC

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6520 from trociny/fix-rbd-apply_changes

rbd: make config changes actually apply

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6347 from xiexingguo/xxg-wip-13566

osd: fixes for several cases where op result code was not checked or set

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5168 from majianpeng/dynamic-control-optracker

common: allow enable/disable of optracker at runtime

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6089 from SUSE/wip-init-ceph-getopt

init-ceph: use getopt to make option processing more flexible

Reviewed-by: Kefu Chai <kchai@redhat.com>

src/init-ceph.in: remove unused variables

Signed-off-by: Nathan Cutler <ncutler@suse.com>

src/init-ceph.in: improve usage message

Signed-off-by: Nathan Cutler <ncutler@suse.com>

src/init-ceph.in: process command-line options using getopt

http://tracker.ceph.com/issues/3015 Fixes: #3015

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rbd: RBD::clone2: fix tracepoint

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6516 from trociny/wip-13719

librbd: perf section name: use hyphen to separate components

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: make config changes actually apply

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

tests: rbd/admin_socket: use xmlstarlet when parsing perf dump

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: perf section name: use hyphen to separate components

"/" and "@" characters make invalid xml format output.

Fixes: #13719
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

doc: Adding --cluster option to rbd man page.

Fixes #13457
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6519 from kylinstorage/wip-revise-organization

revise organization

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6525 from dillaman/wip-13636-jewel

WorkQueue: new PointerWQ base class for ContextWQ

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge branch 'wip-11286' of https://github.com/dillaman/ceph

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5987 from dillaman/wip-rbd-refactor

rbd: refactor cli command handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: dynamically generated bash completion

The rbd CLI bash completion is now dynamically generated by extracting
the available commands and command options from the CLI application.

Fixes: #13494
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: hidden 'bash-completion' command dumps all available commands

It also extracts all available options for a given command for use
which can be used by an improved bash completion script.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: updated rbd CLI --image-feature optional

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph_test_rados_api_tier: fix PromoteOn2ndRead for EC case

The ec and non-ec cases are copy&pasted code.  Yuck.  This duplicates the
non-ec fix from 347ac0f8.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: corrected handling of '--image-feature' optional

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6488 from xiexingguo/xxg-wip-rados-cleanup

librados: do cleanup

Reviewed-by: Sebastien Ponce <sebastien.ponce@cern.ch>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6523 from dillaman/wip-13740

librbd: perf counters might not be initialized on error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

WorkQueue: new PointerWQ base class for ContextWQ

The existing work queues do not properly function if added to a running
thread pool.  librbd uses a singleton thread pool which requires
dynamically adding/removing work queues as images are opened and closed.

Fixes: #13636
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: perf counters might not be initialized on error

Fixes: #13740
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

PendingReleaseNotes: document updated rbd CLI options

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6522 from branch-predictor/bp-temporary-mon-fix

mon: revert MonitorDBStore's WholeStoreIteratorImpl::get

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mon: revert MonitorDBStore's WholeStoreIteratorImpl::get

Revert MonitorDBStore's WholeStoreIteratorImpl::get method to state
before commit 66b7b920cf5a0a9c71212573ef47fb2c7ea9b5ff until better,
long-term solution for #13742 will be provided.

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

revise organization

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

osd: fix trivial bug

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>
Reported-by: Jie Wang <jie.wang@kylin-cloud.com>

Merge pull request #6394 from dillaman/wip-13560

krbd: remove deprecated --quiet param from udevadm

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6508 from dillaman/wip-13726

librbd: deadlock while attempting to flush AIO requests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6447 from dillaman/wip-13674

librbd: resize should only update image size within header

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

internal: remove unused local variables

Remove unused variables bl and bl2 in snap_list and ictx_refresh.
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

librados: cast oid to object explicitly before call ioctx methods
Cast oid to object explicitly before call ioctx methods.
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #6513 from ceph/revert-6419-wip-openssl

Revert 0374bb4a2f5054d606e4aba2d97b5e6765e781b0

IoCtxImpl: remove unused variable sName

remove unused local variable sName in rollback.
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Revert 0374bb4a2f5054d606e4aba2d97b5e6765e781b0

Merge pull request #6419 from ceph/wip-openssl

rgw: link against system openssl (instead of dlopen at runtime)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6453 from trociny/wip-2468

librbd: flush and invalidate cache via admin socket

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6511 from liewegas/wip-fix-kv

kv: fix bug in kv key optimization

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge branch 'wip-12955'

rgw: fix modification to index attrs when setting acls

Fixes: #12955
 - add ACL change after initial population
 - populate bufferlist with object attributes

Signed-off-by: Brian Felton <bjfelton@gmail.com>

kv: fix string ctor usage

When constructing a string from a char* there is only a single
length argument, no offset.  The 3 argument variant we were using
was converting to a std::string first (an prematurely terminating
on \0).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6047 from Sandy4999/wip-13140

rgw: fix swift API returning incorrect account metadata

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6504 from dillaman/wip-13727

tests: fix typo in TestClsRbd.snapshots test case

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: fixed deadlock while attempting to flush AIO requests

In-flight AIO requests might force a flush if a snapshot was created
out-of-band.  The flush completion was previously invoked asynchronously,
potentially via the same thread worker handling the AIO request. This
resulted in the flush operation deadlocking since it can't complete.

Fixes: #13726
Backport: infernalis, hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: new test case to catch deadlock on RBD image refresh

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: librbd: admin socket commands to flush and invalidate cache

Signed-off-by: Mykola Golub <mgolub@mirantis.com>