librbd: copy operation needs to use AIO work queue for writes

This ensures that the exclusive lock is properly acquired prior to
writing to the new image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify IO flush handling

Add a new convenience method to ImageCtx for handling flush
requests and cleanup flush handling with dealing with the cache.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: possible deadlock attempting to drain parent image WQs

Ensure all AIO to the parent image is properly flushed and assert
that all work queues are empty before closing the parent image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

WorkQueue: PointerWQ drain no longer waits for other queues

If another (independent) queue was processing, drain could
block waiting.  Instead, allow drain to exit quickly if
no items are being processed and the queue is empty for
the current WQ.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: avoid holding lock while marking ops are complete

There is a possibility of a race if the JournalTrimmer destructor is
waiting for an async op to complete but the op is flagged as complete
while the mutex is being held.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cmake: updates for refactored librbd IO path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: flush journal entries prior to releasing lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: only erase IO events after they are marked safe

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: librbd updates for C++11

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: verify librbd journal entries are written to disk

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: treat flush op as a write op

This will ensure that the journal is properly opened to handle
appending events.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: verify proper handling of librbd replay of IO journal events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: updated librbd ImageWatcher listener signature

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: addressed possible race conditions / deadlocks from unit testing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add replay support for IO events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix integration with journal library

Valgrind-related updates to the journal library required tweaks to
current implementation of librbd journaling.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: inform the journal when pending IO safely commits

When caching is disabled, the AioCompletion notifies the journal that
the update is safe.  When caching is enabled, writeback can result
in partial write extents being overwritten (and no longer associated
to the original journal event).  In this case, the writeback handler
is responsible for informing the journal when writes are safe.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: integrate cache with journal

Cache writeback should be delayed until after journal event has been
commmitted to disk.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc: track journal commit tid within ObjectCacher for writes

Writebacks from the journal will provide the associated journal commit
tid so that writebacks can be delayed until after the journal entry is
safe on disk.  This allows asynchronously submitting an event to the
journal and submitting the write operation to the ObjectCacher.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add support for removing the journal from disk

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: add journal support to RBD test cases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librdb: initial interface with journal library

Rough draft of journal library integration within librbd. Non-cached
IO paths are now recorded to the journal.  Incoming IO ops are blocked
if the exclusive lock isn't held or if the journal hasn't been replayed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: prevent concurrent image refreshes

With multiple concurrent AIO requests, it's possible for multiple
image refreshes to be scheduled.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: added journal::EventEntry::get_event_type() helper

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: removed ImageWatcher header namespace indentation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: re-use common logic between aio_write/discard

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc: improved discard_set signature const correctness

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: encoding: add new librbd journal event type

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify watch/notify type encoding

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: initial journal entry structures

New journal entries to cover AIO write/discard/flush operations.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: update librbd tests to support new AIO / lock handling

ImageWatcher is no longer responsible for queueing write ops while
waiting for the exclusive lock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ImageWatcher no longer maintains pending AIO op list

Previously the ImageWatcher stored delayed ops that were waiting
on the image exclusive lock.  This management has been moved to the
AioImageRequestWQ to ensure requests are processed in-order.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: updates for AIO locking changes

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: consistent owner_lock handling for AIO paths

Required moving non-AIO read/write/discard methods to
AioImageRequestWQ to avoid deadlock on lock request.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify AioImageRequestWQ function signatures

The ImageCtx is known so there is no need to pass it with each
function call.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: cleanup ictx_check's handling of owner_lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: aio_write/aio_discard have been refactored

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refactor AIO request handling

Move aio_read/write/discard/flush to new AioImageRequest classes
in support of a unified aio queue / journaling.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: AIO submissions processed by new AioImageRequestWQ

New work queue has the ability to suspend write operations, which
should occur when exclusive locking is enabled and the client doesn't
own the lock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: rename AioRequest classes to AioObjectRequest

This better reflects the fact that these represent requests against
an object extent and helps differentiate it from the future
AioImageRequest classes.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: rename C_AioWrite to C_AioRequest

This context is used for more than just write requests. It is always
tied to an AioCompletion, so moved request reference counting
management within class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: removed unused method declarations and definitions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add support for dynamically enabling/disabling journaling

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/rbd.py: add new journaling feature code

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add new RBD_FEATURE_JOURNALING feature code

When enabled, all mutable operations against an RBD image
will be recorded to a journal prior to changing the underlying
RBD image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge branch 'wip-13504' of https://github.com/trociny/ceph

rbd: API: options on image create

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/test/librbd/test_librbd.cc (trivial, two tests added at end of file)

librbd: API: options on image create: update tests

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: API: options on image create

Pass options as key/value configuration pairs on image create instead
of expanding rbd_create/rbd_clone/rbd_copy for every possible
configuration override.

Fixes: #13504
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6557 from ceph/wip-rbd-cmake-fixes

cmake: librbd and libjournal build fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6556 from athanatos/wip-cmake-nss

cmake: add nss as a suffix for pk11pub.h

cmake: add nss as a suffix for pk11pub.h

Signed-off-by: Samuel Just <sjust@redhat.com>

librbd: provide an out-of-class definition for MAX_DESCRIPTION_OFFSET

std::min() takes its parameters by reference, so provide a definition
for OptionPrinter::MAX_DESCRIPTION_OFFSET constant.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

cmake: fix librbd and add src/journal

librbd was missing a recently added file, src/journal was missing
entirely - this brings the build back.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #6517 from kylinstorage/wip-trivial-bug

osd: fix bug in last_* PG state timestamps

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6552 from trociny/wip-journal

journal: incremental improvements and fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6462 from javacruft/wip/ec-modules

Build internal plugins and classes as modules

Reviewed-by: Loic Dachary <ldachary@redhat.com>

journal: update allocated tid when skipping committed entry in player

Otherwise, if on image open, there are no any uncommitted entries in
journal, allocated tid is not updated to the latest commited and
recording always starts from tid=0.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6539 from yehudasa/wip-cmake-fix

cmake: fix files list

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6316 from dillaman/wip-13494

rbd: dynamically generated bash completion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'origin/jewel'

cmake: fix files list

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls::journal: fixup: constify dump functions

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

journal: call metadata shutdown on journal remove

This fixes error observed when disabling journal:

  JournalMetadata: journal watch error: (107) Transport endpoint is not connected

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

journal: don't use object_number when comparing positions

Only tid does matter.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

journal: make commit and flush params configurable

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

journal: allow alternate pool for journal objects

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

journal: output operators for journal types

They are going to be used for diagnostic.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6471 from mslovy/wip-check-reply-guard-again

osd: avoid double-check for replaying and can_checkpoint() in FileStore::_check_replay_guard

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6426 from xiexingguo/xxg-wip-13642

librados: fix rare race where pool op callback may hang forever

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6407 from renhwztetecs/renhw-wip-13627

mon: initialize recorded election epoch properly even when standalone

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6385 from Vicente-Cheng/mkfs-does-not-decode-an-existing-superblock

osd: fix reactivate (check OSDSuperblock in mkfs() when we already have the superblock)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6373 from liewegas/wip-promoteon2ndread-ec

librados: fix PromoteOn2ndRead test for EC

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6520 from trociny/fix-rbd-apply_changes

rbd: make config changes actually apply

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6347 from xiexingguo/xxg-wip-13566

osd: fixes for several cases where op result code was not checked or set

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5168 from majianpeng/dynamic-control-optracker

common: allow enable/disable of optracker at runtime

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6089 from SUSE/wip-init-ceph-getopt

init-ceph: use getopt to make option processing more flexible

Reviewed-by: Kefu Chai <kchai@redhat.com>

src/init-ceph.in: remove unused variables

Signed-off-by: Nathan Cutler <ncutler@suse.com>

src/init-ceph.in: improve usage message

Signed-off-by: Nathan Cutler <ncutler@suse.com>

src/init-ceph.in: process command-line options using getopt

http://tracker.ceph.com/issues/3015 Fixes: #3015

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rbd: RBD::clone2: fix tracepoint

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6516 from trociny/wip-13719

librbd: perf section name: use hyphen to separate components

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: make config changes actually apply

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

tests: rbd/admin_socket: use xmlstarlet when parsing perf dump

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: perf section name: use hyphen to separate components

"/" and "@" characters make invalid xml format output.

Fixes: #13719
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

doc: Adding --cluster option to rbd man page.

Fixes #13457
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6519 from kylinstorage/wip-revise-organization

revise organization

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6525 from dillaman/wip-13636-jewel

WorkQueue: new PointerWQ base class for ContextWQ

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge branch 'wip-11286' of https://github.com/dillaman/ceph

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5987 from dillaman/wip-rbd-refactor

rbd: refactor cli command handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: dynamically generated bash completion

The rbd CLI bash completion is now dynamically generated by extracting
the available commands and command options from the CLI application.

Fixes: #13494
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: hidden 'bash-completion' command dumps all available commands

It also extracts all available options for a given command for use
which can be used by an improved bash completion script.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: updated rbd CLI --image-feature optional

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph_test_rados_api_tier: fix PromoteOn2ndRead for EC case

The ec and non-ec cases are copy&pasted code.  Yuck.  This duplicates the
non-ec fix from 347ac0f8.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: corrected handling of '--image-feature' optional

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6488 from xiexingguo/xxg-wip-rados-cleanup

librados: do cleanup

Reviewed-by: Sebastien Ponce <sebastien.ponce@cern.ch>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6523 from dillaman/wip-13740

librbd: perf counters might not be initialized on error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

WorkQueue: new PointerWQ base class for ContextWQ

The existing work queues do not properly function if added to a running
thread pool.  librbd uses a singleton thread pool which requires
dynamically adding/removing work queues as images are opened and closed.

Fixes: #13636
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: perf counters might not be initialized on error

Fixes: #13740
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

PendingReleaseNotes: document updated rbd CLI options

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6522 from branch-predictor/bp-temporary-mon-fix

mon: revert MonitorDBStore's WholeStoreIteratorImpl::get

Reviewed-by: Loic Dachary <ldachary@redhat.com>