librbd: commit journal op events immediately

Op events are split into a start and finish record, so they can
be immediately marked as committed within the journal.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: improve debug output for object map state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: owner_lock should be held while opening parent image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ImageWatcher shouldn't block the notification thread

Now that all RPC operations are asynchronous, blocking the notification
thread will also result in librados async callbacks becoming blocked
(since they use the same thread).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: add missing test cases for rbd exclusive-lock RPC

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: possible recursive reader lock during resize

With multiple resizes concurrently queued, it's possible for
the owner lock to be locked multiple times.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: possible unit test race condition

It's possible during unit tests for the requests to complete
prior to marking the Context as complete.  In this case, the
Context will fire while locks are being held.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: updated librbd::Journal API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: journal op requests

Any op request that has the potential to impact the layout or data
within an image needs to be journaled for RBD mirroring purposes.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new journal types for maintenance commands

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: added snap protect/unprotect and rename to client update test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: new watch/notify rename image tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: added async image rename op

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: added ObjectWriteOperation::tmap_update

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: update cls_rbd test for async dir_rename_image method

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: dir_rename_image is now async

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: new tests for watch/notify snap protect/unprotect ops

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: snap protect/unprotect now connected to watch/notify

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: async ops should return status via the Context

Avoid handling a return value which could indicate an error and
instead return initialization errors via the Context callback.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: new ImageWatcher tests for snap protect/unprotect and rename

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new watch/notify types for snap protect/unprotect and rename

When journaling is enabled, these actions will need to be executed via the
lock owner to ensure that they are journaled for RBD mirroring.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: rename librbd::WatchNotify namespace to librbd::watch_notify

The new namespace follows the expected naming conventions.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not send duplicate lock event

No need to alert clients that the lock has been released since it
is already handled by ImageWatcher::release_lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: drain op work queue after shutting down journal

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: migrate object map snapshot ops to async state machines

Since all snapshot handling was migrated to async state machines, the
object map snapshot handling also needs to be converted.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: add async version of object_map_load

Most librbd object map interactions are moving to async state machines.
The sync version must remain until the librbd image refresh operation
is converted to a state machine due to the sheer volume of locks held
blocking the AIO thread.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: consolidate object map invalidation to new state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move object map async ops to standalone classes

In preparation for creating new async snapshot-related object map
operations, move the existing object map async ops to there own
namespace.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: initial conversion of snapshot ops to async versions

Object map updates are not currently integrated until those methods
are converted to async state machines.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: update cls_rbd tests for async snapshot methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: convert snapshot methods to async versions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new intermediate request class for handling journal ops

All operations that should be recorded to the journal will inherit
from the new librbd::operation::Request class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move all image operation state machines to new namespace

Prior to creating multiple new state machines for image operations,
move all existing op requests to a new namespace and folder.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: remove AsyncRequest::safely_cancel

Long-running operations controlled via the AsyncObjectThrottle will
automatically cancel with an -ERESTART return code when requested.
Other states are transitional and have a high-potential for race
conditions if aborted between states.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify state machine lock assumptions

Use asynchronous callbacks to ensure that no locks are held when
re-entering a state machine.  This helps to avoid odd edge conditions
where locks may or may not be held depending on whether a AIO
operation was required.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: async version of AioImageRequestWQ::block_writes

Snapshot create will uses the async version to block write ops
prior to creating the new snapshot context.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: removed Async* prefix from request state machines

In preparation for adding more snapshot handling request state machines,
simplify the naming of the existing classes by removing the Async prefix.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6541 from dillaman/wip-11287

librbd: integrate journaling support for IO operations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: copy operation needs to use AIO work queue for writes

This ensures that the exclusive lock is properly acquired prior to
writing to the new image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify IO flush handling

Add a new convenience method to ImageCtx for handling flush
requests and cleanup flush handling with dealing with the cache.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: possible deadlock attempting to drain parent image WQs

Ensure all AIO to the parent image is properly flushed and assert
that all work queues are empty before closing the parent image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

WorkQueue: PointerWQ drain no longer waits for other queues

If another (independent) queue was processing, drain could
block waiting.  Instead, allow drain to exit quickly if
no items are being processed and the queue is empty for
the current WQ.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5867 from Vicente-Cheng/ceph-disk-destroy-implementation

ceph-disk: Add destroy and deactivate option

ceph-disk now understands deactivate to shutdown an OSD and umount the partition. It will remain in a "down" state until it is activated again with --reactivate. If a deactivated OSD is destroyed with ceph-disk destroy, it will be permanently removed from the cluster and from the crushmap.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: remove the redundant try except and minor nits

  - remove the redundant try except

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: add wait_for_osd_down() in ceph-disk-test.py of qa

  - add wait_for_osd_down() to avoid the side effect of deactivate

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: modify the ceph-disk qa test cases

  - minor correct for the latest ceph-disk qa test cases

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: improve the unit test to coverage all deactivate/destroy function.

  - rework the unit test to coverage all deactivate/destroy function
  - make test item simple and easier to read

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: improve the device query stage on deactivate/destroy feature.

  - Make code path much easier to get device info. (get little bit overhead)
  - Let some error rasie the correct execption
  - for dmcrypt device, we unmap on the deactivate stage. (consist with activate)

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: add deactivate/reactivate/destroy test cases.

  - using the deactivate/destroy feature to destroy osd
  - test reactivate option when the osd goes deactive
  - add check_osd_status to check osd status when osd goes up

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: add some unittest functions to coverage destroy/deactivate feature.

  - Add new unittest functions to coverage the part of dmcrypt/mpath handling.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: modify the destroy/deactivate behavior to handle dmcrypt/mpath feature.

  - modify the behavior to handle dmcrypt/mpath osd
  - add some functions to get the information of dmcrypt osd
  - fixed the logging format Use `%(filename)s` instead of `%(filename)`

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: Make unit test coverage all ceph-disk destroy/deactivate feature

  - Add some unit test to coverage all destroy/deactivate feature.
  - Do some minor modifications on the ceph-diskw

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

tests: ceph-disk: modified the destroy_osd test function.

  - use the new implementation (ceph-disk deactivate/destroy) instead of step by step remove

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: Implement unittest for ceph-disk deactivate/destroy feature

  - The unit tests cover all the implemention about deactivate/destroy.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add --reactivate option, modify parameter about deactivate and destroy

  - add `--reactivate` option (activate do no-op without reactivate with deactive flag)
  - for consistency, make both deactivate and destroy take the device/partition name
  - add `--deactivate-by-id` to deactivate and destroy for ease of use

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add `--mark-out` option on deactivate feature.

  - Using `--mark-out` option to mark osd out when
    deactivate this osd instead of always mark osd out.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: use `ceph osd dump` to check osd status

  - Before deactivated osd, we need to know the status
    including IN/OUT, UP/DOWN on this osd.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add destroy feature

  Implement destroy option on ceph-disk.

  - remove OSD from CRUSH map
  - remove OSD cephx key
  - deallocate OSD ID
  - destroy data (with --zap option)

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

ceph-disk: add deactivate feature

  Implement deactivate option on ceph-disk.

  - stop ceph-osd service if needed (If osd still in osd map, make it out first)
  - remove 'ready', 'active', and INIT-specific files
  - remove gpt partition type and change partition name (prevent triggered by udev)
  - create deactive flag
  - umount device and remove mount point

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

Merge pull request #6607 from dillaman/wip-13784

rbd: support negative boolean command-line optionals

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6606 from dillaman/wip-13806

rbd: add missing command aliases to refactored CLI

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: support negative boolean command-line optionals

Fixes: #13784
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: new rbd CLI command aliases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add missing command aliases to refactored CLI

Fixes: #13806
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6590 from ceph/wip-rbd-user-option

rbd: accept --user, refuse -i command-line optionals

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6593 from trociny/fixup-rbd-refactor

rbd: stripe unit/count set incorrectly from config

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6594 from chenji-kael/patch-2

Update .mailmap

Update .mailmap

rbd: stripe unit/count set incorrectly from config

(after rbd-refactor fixup)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6549 from kylinstorage/global_flags

fix: use right init_flags to finish CephContext

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6575 from ceph/update-infernalis-releasenotes

doc: update infernalis release notes

Merge pull request #6484 from XinzeChi/wip-journal-optimization

os: write file journal optimization

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #6371 from liewegas/wip-osd-less-crush

osd: avoid calculating crush mapping for most ops

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6416 from rohanmars/wip-solaris-port

librados: Solaris port

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6439 from XinzeChi/wip-merge-tnx

osd: merge local_t and op_t txn to single one

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6547 from xiaoxichen/fix1

osd: check do_shutdown before do_restart

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6566 from kylinstorage/wip-fix-wrong-use-parenthesis

osd: fix wrong use of right parenthesis in localized read logic

Reviewed-by: Sage Weil <sage@redhat.com>

rbd: accept --user, refuse -i

--user is the equivalent of --id (mentioned in docs and ceph -h) and
used by at least a couple of qa scripts and teuthology.  Add it to the
list of valid rbd options - it's handled inside global_init() before
Shell:execute() is called.

OTOH allowing -i is bogus - global_init() wouldn't process it for
clients (CEPH_ENTITY_TYPE_CLIENT), which rbd tool is one of.  Drop it.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #6584 from dachary/wip-test

tests: ignore test-suite.log

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6583 from ceph/wip-krbd-map-args

rbd: unbreak rbd map + cephx_sign_messages option

Reviewed-by: Sage Weil <sage@redhat.com>

tests: ignore test-suite.log

Now that the tests at the top level directory are run via TESTS and not
check-local:, the test-suite.log file is created and must bit
gitignored.

Signed-off-by: Loic Dachary <loic@dachary.org>

rbd: recognize cephx_sign_messages option

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

rbd: unbreak rbd map CLI

Previously, if snapname wasn't specified we would pass NULL to
krbd_map(), which was a cue for it to use "-" as a snapshot name.  With
the new rbd CLI, "" is passed in; same goes for map options.

Change krbd_map() accordingly and update its other user.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

tests: fix test case using new api

Signed-off-by: Xinze Chi <xinze@xsky.com>

Merge pull request #6574 from yuyuyu101/fix-broken-kinects

kv/KineticStore: fix broken split_key

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6571 from dachary/wip-test-run-cli

tests: restore run-cli-tests

Merge pull request #6578 from dachary/wip-13785-debian-rbd-replay

build/ops: rbd-replay moved from ceph-test-dbg to ceph-common-dbg

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4737 from kylinstorage/wip-temp-based-object-eviction

osd: improve temperature calculation for cache tier agent

Reviewed-by: Sage Weil

Merge pull request #6422 from xiexingguo/xxg-wip13639

librados: fix potential null pointer access when do pool_snap_list

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6486 from XinzeChi/wip-multiple-finisher

osd: FileStore: support multiple ondisk finish and apply finishers

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6518 from kylinstorage/wip-trivial-optimization

osd: optimize scrub subset_last_update calculation

Reviewed-by: Sage Weil <sage@redhat.com>

build/ops: rbd-replay moved from ceph-test-dbg to ceph-common-dbg

http://tracker.ceph.com/issues/13785 Fixes: #13785

Signed-off-by: Loic Dachary <loic@dachary.org>

tests: avoid bashism

The shell used by the cli tests is not always bash. Not using the
here-word is also more readable in this specific case.

Signed-off-by: Loic Dachary <loic@dachary.org>

rbd: hardcode application name into help

Avoid dynamically detecting the application name and instead hardcode
the rbd CLI name into the help output.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

kv/KineticStore: Fix broken split_key

Introduced by PR #6312

Signed-off-by: Haomai Wang <haomai@xsky.com>

Update infernalis release notes

Update infernalis release notes

Signed-off-by: Vasu Kulkarni vasu@redhat.com

tests: restore run-cli-tests

e4ca468 moved src/test/run-cli-tests from check-local to check_SCRIPTS
but did not add it to the TESTS variable.

Signed-off-by: Loic Dachary <loic@dachary.org>

SIGPIPE suppression for platforms without SO_NOSIGPIPE or MSG_NOSIGNAL

Signed-off-by: Rohan Mars <code@rohanmars.com>

radosgw-admin: fix cli tests

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: fix cli tests

Signed-off-by: Sage Weil <sage@redhat.com>