ceph-disk: remove the redundant try except and minor nits

  - remove the redundant try except

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: add wait_for_osd_down() in ceph-disk-test.py of qa

  - add wait_for_osd_down() to avoid the side effect of deactivate

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: modify the ceph-disk qa test cases

  - minor correct for the latest ceph-disk qa test cases

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: improve the unit test to coverage all deactivate/destroy function.

  - rework the unit test to coverage all deactivate/destroy function
  - make test item simple and easier to read

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: improve the device query stage on deactivate/destroy feature.

  - Make code path much easier to get device info. (get little bit overhead)
  - Let some error rasie the correct execption
  - for dmcrypt device, we unmap on the deactivate stage. (consist with activate)

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: add deactivate/reactivate/destroy test cases.

  - using the deactivate/destroy feature to destroy osd
  - test reactivate option when the osd goes deactive
  - add check_osd_status to check osd status when osd goes up

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: add some unittest functions to coverage destroy/deactivate feature.

  - Add new unittest functions to coverage the part of dmcrypt/mpath handling.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: modify the destroy/deactivate behavior to handle dmcrypt/mpath feature.

  - modify the behavior to handle dmcrypt/mpath osd
  - add some functions to get the information of dmcrypt osd
  - fixed the logging format Use `%(filename)s` instead of `%(filename)`

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: Make unit test coverage all ceph-disk destroy/deactivate feature

  - Add some unit test to coverage all destroy/deactivate feature.
  - Do some minor modifications on the ceph-diskw

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: modified the destroy_osd test function.

  - use the new implementation (ceph-disk deactivate/destroy) instead of step by step remove

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: Implement unittest for ceph-disk deactivate/destroy feature

  - The unit tests cover all the implemention about deactivate/destroy.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add --reactivate option, modify parameter about deactivate and destroy

  - add `--reactivate` option (activate do no-op without reactivate with deactive flag)
  - for consistency, make both deactivate and destroy take the device/partition name
  - add `--deactivate-by-id` to deactivate and destroy for ease of use

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add `--mark-out` option on deactivate feature.

  - Using `--mark-out` option to mark osd out when
    deactivate this osd instead of always mark osd out.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: use `ceph osd dump` to check osd status

  - Before deactivated osd, we need to know the status
    including IN/OUT, UP/DOWN on this osd.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add destroy feature

  Implement destroy option on ceph-disk.

  - remove OSD from CRUSH map
  - remove OSD cephx key
  - deallocate OSD ID
  - destroy data (with --zap option)

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add deactivate feature

  Implement deactivate option on ceph-disk.

  - stop ceph-osd service if needed (If osd still in osd map, make it out first)
  - remove 'ready', 'active', and INIT-specific files
  - remove gpt partition type and change partition name (prevent triggered by udev)
  - create deactive flag
  - umount device and remove mount point

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

Merge pull request #6607 from dillaman/wip-13784

rbd: support negative boolean command-line optionals

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6606 from dillaman/wip-13806

rbd: add missing command aliases to refactored CLI

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: support negative boolean command-line optionals

Fixes: #13784
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: new rbd CLI command aliases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add missing command aliases to refactored CLI

Fixes: #13806
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6590 from ceph/wip-rbd-user-option

rbd: accept --user, refuse -i command-line optionals

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6593 from trociny/fixup-rbd-refactor

rbd: stripe unit/count set incorrectly from config

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6594 from chenji-kael/patch-2

Update .mailmap

Update .mailmap

rbd: stripe unit/count set incorrectly from config

(after rbd-refactor fixup)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6549 from kylinstorage/global_flags

fix: use right init_flags to finish CephContext

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6575 from ceph/update-infernalis-releasenotes

doc: update infernalis release notes

Merge pull request #6484 from XinzeChi/wip-journal-optimization

os: write file journal optimization

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #6371 from liewegas/wip-osd-less-crush

osd: avoid calculating crush mapping for most ops

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6416 from rohanmars/wip-solaris-port

librados: Solaris port

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6439 from XinzeChi/wip-merge-tnx

osd: merge local_t and op_t txn to single one

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6547 from xiaoxichen/fix1

osd: check do_shutdown before do_restart

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6566 from kylinstorage/wip-fix-wrong-use-parenthesis

osd: fix wrong use of right parenthesis in localized read logic

Reviewed-by: Sage Weil <sage@redhat.com>

rbd: accept --user, refuse -i

--user is the equivalent of --id (mentioned in docs and ceph -h) and
used by at least a couple of qa scripts and teuthology.  Add it to the
list of valid rbd options - it's handled inside global_init() before
Shell:execute() is called.

OTOH allowing -i is bogus - global_init() wouldn't process it for
clients (CEPH_ENTITY_TYPE_CLIENT), which rbd tool is one of.  Drop it.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #6584 from dachary/wip-test

tests: ignore test-suite.log

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6583 from ceph/wip-krbd-map-args

rbd: unbreak rbd map + cephx_sign_messages option

Reviewed-by: Sage Weil <sage@redhat.com>

tests: ignore test-suite.log

Now that the tests at the top level directory are run via TESTS and not
check-local:, the test-suite.log file is created and must bit
gitignored.

Signed-off-by: Loic Dachary <loic@dachary.org>

rbd: recognize cephx_sign_messages option

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

rbd: unbreak rbd map CLI

Previously, if snapname wasn't specified we would pass NULL to
krbd_map(), which was a cue for it to use "-" as a snapshot name.  With
the new rbd CLI, "" is passed in; same goes for map options.

Change krbd_map() accordingly and update its other user.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

tests: fix test case using new api

Signed-off-by: Xinze Chi <xinze@xsky.com>

Merge pull request #6574 from yuyuyu101/fix-broken-kinects

kv/KineticStore: fix broken split_key

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6571 from dachary/wip-test-run-cli

tests: restore run-cli-tests

Merge pull request #6578 from dachary/wip-13785-debian-rbd-replay

build/ops: rbd-replay moved from ceph-test-dbg to ceph-common-dbg

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4737 from kylinstorage/wip-temp-based-object-eviction

osd: improve temperature calculation for cache tier agent

Reviewed-by: Sage Weil

Merge pull request #6422 from xiexingguo/xxg-wip13639

librados: fix potential null pointer access when do pool_snap_list

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6486 from XinzeChi/wip-multiple-finisher

osd: FileStore: support multiple ondisk finish and apply finishers

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6518 from kylinstorage/wip-trivial-optimization

osd: optimize scrub subset_last_update calculation

Reviewed-by: Sage Weil <sage@redhat.com>

build/ops: rbd-replay moved from ceph-test-dbg to ceph-common-dbg

http://tracker.ceph.com/issues/13785 Fixes: #13785

Signed-off-by: Loic Dachary <loic@dachary.org>

tests: avoid bashism

The shell used by the cli tests is not always bash. Not using the
here-word is also more readable in this specific case.

Signed-off-by: Loic Dachary <loic@dachary.org>

rbd: hardcode application name into help

Avoid dynamically detecting the application name and instead hardcode
the rbd CLI name into the help output.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

kv/KineticStore: Fix broken split_key

Introduced by PR #6312

Signed-off-by: Haomai Wang <haomai@xsky.com>

Update infernalis release notes

Update infernalis release notes

Signed-off-by: Vasu Kulkarni vasu@redhat.com

tests: restore run-cli-tests

e4ca468 moved src/test/run-cli-tests from check-local to check_SCRIPTS
but did not add it to the TESTS variable.

Signed-off-by: Loic Dachary <loic@dachary.org>

SIGPIPE suppression for platforms without SO_NOSIGPIPE or MSG_NOSIGNAL

Signed-off-by: Rohan Mars <code@rohanmars.com>

radosgw-admin: fix cli tests

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: fix cli tests

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: fix cli tests

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: fix cli test help

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6532 from dachary/wip-mailmap

mailmap: Ubuntu Kylin name changed to Kylin Cloud

Reviewed-by: Li Wang <li.wang@kylin-cloud.com>

Merge pull request #5848 from storage-zuiwanyuan/wip-nonblock-connect

msg/async: support of non-block connect in async messenger

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #6478 from yuyuyu101/wip-13666

msg/async: let receiver ack message ASAP

Reviewed-by: Sage Weil <sage@redhat.com>

osd: fix wrong use of right parenthesis

Signed-off-by: Jie Wang <jie.wang@kylin-cloud.com>
Reviewed-by: Li Wang <li.wang@kylin-cloud.com>

msg/async: support of non-block connect in async messenger

Fixes: #12802
Signed-off-by: Jianhui Yuan <zuiwanyuan@gmail.com>

Merge pull request #6534 from kylinstorage/wip-trivial-scrub-cleanup

osd: clarify the scrub result report

Reviewed-by: Kefu Chai <kchai@redhat.com>

scrub: clarify the result report

It may happen that the authoritative object
such that auth.size != be_get_ondisk_size(auth_oi.size),
in that case, clarify the error report.

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

Merge branch 'wip-py3'

pybind: a few more python 3 fixes for rbd and rados

Reviewed-by: David Coles <dcoles@gaikai.com>

pybind/rados: return pool_reverse_lookup() result as a string

This makes it symmetric with create_pool() in python 3.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

pybind/test_rbd: convert a few more str to bytes for py3

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

pybind/rbd: encode snap_rename args for py3

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

pybind/rbd: decode stat() and list_children() results for py3

For stat(), only block_name_prefix is filled in - parent and
parent_pool are always blank.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

pybind/rbd: decode parent_info() to str types for py3

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

pybind/test_rbd: fix map() usage for py3 compat

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

pybind/test_rbd: use // for division for py3

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge branch 'pybind3' of https://github.com/dcoles/ceph into wip-pybind3

pybind: Add Python 3 support for rados and rbd modules

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/pybind/rbd.py (new create args, minor fix to work with py3)

solaris big endian fixes

Signed-off-by: Rohan Mars <code@rohanmars.com>

error code translation logic

Signed-off-by: Rohan Mars <code@rohanmars.com>

Solaris compilation/build changes

Signed-off-by: Rohan Mars <code@rohanmars.com>

updated common library linked objects

cross platform pthreads

Signed-off-by: Rohan Mars <code@rohanmars.com>

fix sem_open() call to be compliant

Signed-off-by: Rohan Mars <code@rohanmars.com>

import missing free function definition

Signed-off-by: Rohan Mars <code@rohanmars.com>

stdin/stdout/stderr clash with standard definition, renamed to getter/setter

Signed-off-by: Rohan Mars <code@rohanmars.com>

ceph: Make stdout/stderr always output Unicode (UTF-8)

If a stream is not interactive, then under Python 2, then the encoding for
stdout/stderr may be None. This means that it's not possible to print Unicode
characters since the encoding will fall back to ASCII.

This explicitly makes sys.stdout/sys.stderr always use UTF-8 encoding for
strings, regardless of the system's local or if the console is interactive or
not.
This matches the existing tests that assume that output of non-ASCII pool names
will be UTF-8 encoded.

When outputting raw binary data (such as the CRUSH-map), we must bypass the
codec and write directly to raw streams (since the new stream will only accept
ASCII byte-strings or Unicode strings).

Signed-off-by: David Coles <dcoles@gaikai.com>

pybind: Add decode_cstr helper function

This function attempts to decode a C-style string into a Python Unicode string.
It accepts an optional "size" parameter for the string length, otherwise it is
assumed that the string is NUL-terminated.

If the pointer is NULL, then this function returns None.

Signed-off-by: David Coles <dcoles@gaikai.com>

pybind: Add test for creating pool by raw UTF-8

Some clients try providing non-ASCII pool names by sending raw encoded bytes.
This check ensures that we still support this behaviour for Python 2.

In Python 3, bytestrings will fail since strings are Unicode strings and thus
clients should use Unicode escapes instead.

Signed-off-by: David Coles <dcoles@gaikai.com>

pybind: Import cstr from the rados module

Since rados is required for rbd, we can avoid duplication of code across these
two modules.

Signed-off-by: David Coles <dcoles@gaikai.com>

pybind: Don't encode str on Python 2

If you attempt to call encode on a non-ASCII string, then a UnicodeDecodeError
will be raised.

Since str on Python 2 is an 8-bit string, it's possible that it's already UTF-8
encoded. As such we should just pass it through to the C API unmodified.

On Python 3 or if the user explicitly uses unicode, then we'll encode it to
UTF-8 for them.

Signed-off-by: David Coles <dcoles@gaikai.com>

Merge branch 'wip-13504' of https://github.com/trociny/ceph

rbd: API: options on image create

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/test/librbd/test_librbd.cc (trivial, two tests added at end of file)

os: write file journal optimezation

Currently, there is single write thread for file journal, so it would be bottleneck.
It is important to keep logic of the journal write thread simple. According to the
implementation of transaction encoding, it is almost impossible that the write
bufferlist would be align. So write journal would call rebuild_aligned almost every time.
Because of the memory fragmentation, the bufferlist crc and rebuild would be bottleneck.

My implementation would move the complex logic out of journal write thread.

Signed-off-by: Xinze Chi <xinze@xsky.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

librbd: API: options on image create: update tests

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: API: options on image create

Pass options as key/value configuration pairs on image create instead
of expanding rbd_create/rbd_clone/rbd_copy for every possible
configuration override.

Fixes: #13504
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6557 from ceph/wip-rbd-cmake-fixes

cmake: librbd and libjournal build fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6556 from athanatos/wip-cmake-nss

cmake: add nss as a suffix for pk11pub.h

cmake: add nss as a suffix for pk11pub.h

Signed-off-by: Samuel Just <sjust@redhat.com>

librbd: provide an out-of-class definition for MAX_DESCRIPTION_OFFSET

std::min() takes its parameters by reference, so provide a definition
for OptionPrinter::MAX_DESCRIPTION_OFFSET constant.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

cmake: fix librbd and add src/journal

librbd was missing a recently added file, src/journal was missing
entirely - this brings the build back.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

mailmap: Ubuntu Kylin name changed to Kylin Cloud

Reviewed-by: Li Wang <li.wang@kylin-cloud.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: sort files

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #6517 from kylinstorage/wip-trivial-bug

osd: fix bug in last_* PG state timestamps

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>