os/bluestore: _do_write: fix _do_zero_tail_extent to handle shared extents

Also add a test case to reproduce this case.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_zero: simply truncate up if past eof

Avoid the _do_zero_tail_extent here; we're zeroing so this is no different
than a truncate up

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: prevent rename src from getting trimmed from lru

We were putting it at the end of the LRU *and* not holding a reference.

Fix by letting oldo carry a ref to the Onode.  This avoids a subsequent
lookup and also prevents it from getting trimmed.

Note that we could also put it at the front of the LRU, but given that it
is the rename source, I'm not sure that's appropriate.  For example,
temp recovery objects get renamed into place and we don't want to keep
negative onodes for those around.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_zero: handle unzeroed tail blocks

- _do_write_zero when handling tail extent if it is
  shared.
- handle tail extent we are also zeroing later (not in
  the tail handler).  THis mirrors how _do_write is
  structured.
- handle eof to offset zeroing if our offset starts in
  the tail extent.  just like _do_write's wal path.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_write_zero: preserve onode size

Instead of handling this in each caller, make the helper
keep onode.size intact.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_write: handle gap from eof to offset in wal path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_allocate: fix whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_truncate: no need to zero trailing block on truncate down

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_write: only use append case for aligned eof

The append case here only works if the EOF was aligned
and there is thus no partial-block zeroing we need to
do.  If the condition fails, we fall through to the
generic WAL path below.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_zero_tail_extent: do not assume tail block is zeroed

We will change this shortly.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_zero_tail_extent helper

We do this from both _do_truncate and _do_write.  No
functional change here.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: break _do_zero out of _zero

Seems useless now, but it maintains symmetry and we'll
need it later.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not do gift beyond the max ratio

Signed-off-by: Sage Weil <sage@redhat.com>

global/signal_handler: fix warning

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: release wal_cleaning extents in order

We need to order the freelist updates so that they match the
commit order of the actual transactions.  Otherwise we might, say,
set a key here, delete it in the _txc_update_fm, but commit in
the wrong order and end up with the key surviving.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: avoid temp released interval_map for wal cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/FreelistManager: detect overlapping extents on init

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: trim onode_map when adding new items

This ensures we maintain the cache size even for read workloads.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: keep onode_map max_size

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: wait for wal op that wrote cached tail before using

If the cached tail is written by a WAL op, we need to wait for it to
apply before we use the cached tail.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: note txc seq for tail cache

We'll need this soon...

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: assign per-Sequencer seq to each TransContext

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: add AppendWalVsTailCache test

This specifically exercises a WAL overwrite racing with a subsequent
append that uses the previous writes cached tail.  We need to make sure
the use of the cached_tail waits for the previous WAL write or else our
update of tail block will be overwritten with the prior (WAL) value.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix fsck vs enodes

- fix used_blocks accounting for shared extents
- make sure enodes only exist when objects follow

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: bluestore_inject_wal_apply_delay

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: reorder onode_map vs enode_set ctor/dtor

The onode_map should get declared after the enode_set so that
the enode_set is empty when it is destroyed.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: only cache tail in _pad_tail if we have the whole block

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: only use tail if tail_offset matches

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix off-by-one on caching tail block

We can cache the tail as long as we overwrite all prior content;
we don't have to extent the size.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: print buffered flag in debug line

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8368 from tchaikov/wip-cmake-fix-rbd

cmake: add StandardPolicy.cc to librbd

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8337 from jmunhoz/fix-15292

mon: remove unused variable

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #8352 from runsisi/wip-fix-missing-iter

rbd-mirror: fix missing increment for iterators

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8374 from trociny/fix-cls-rbd-read_peers

cls_rbd: read_peers: update last_read on next cls_cxx_map_get_vals

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7062 from wonzhq/scrub-prio-pool

osd: add the support of per pool scrub priority

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8210 from liewegas/wip-12814

osd: add 'proxy' cache mode

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8315 from mslovy/wip-clean-pg

osd: duplicated clear for peer_missing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8330 from xiexingguo/xxg-wip-race

common: fix race during optracker switches between enabled/disabled mode

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #8335 from liewegas/wip-13992

mon/MonClient: fix shutdown race

Reviewed-by: Kefu Chai <kchai@redhat.com>

cls::rbd: read_peers: update last_read on next cls_cxx_map_get_vals

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #7857 from tchaikov/wip-14844

ceph.spec.in: enable lttng and babeltrace explicitly

thanks to ktdreyer, we have lttng and babel in EPEL-7 now.

Reviewed-by: Nathan Cutler <ncutler@suse.com>

cmake: add StandardPolicy.cc to librbd

fixes the librbd build.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: fix missing increment of iterators

Signed-off-by: runsisi <runsisi@hust.edu.cn>

Merge pull request #8287 from dillaman/wip-15232

rbd-mirror: initial failover / failback support

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/pybind/test_rados: force setting readonly cache mode

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7712 from vitek/listbucket_encoding_type_fix

rgw: S3: set EncodingType in ListBucketResult

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8363 from cbodley/wip-cmake-vstart

vstart: fix up cmake paths when VSTART_DEST is given
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8362 from jdurgin/wip-ceph-python-path

ceph.in: fix python libpath for automake as well

Merge pull request #8364 from liewegas/wip-fix-update-op

os/ObjectStore: fix _update_op for split dest_cid

Reviewed-by: Samuel Just <sjust@redhat.com>

vstart: fix up cmake paths when VSTART_DEST is given

mstart.sh uses VSTART_DEST to tell vstart.sh where to create the
cluster. in the cmake case, it needs to get a relative path from there
to the built binaries and libraries - update these paths to use the new
'lib' and 'bin' directories

Signed-off-by: Casey Bodley <cbodley@redhat.com>

os/ObjectStore: fix _update_op for split dest_cid

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd: add basic failover/fallback test case

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: use 32bit image creation bid to avoid overflow

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: initial support for primary/non-primary handling

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: accessor for journal tag data

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: link primary mirror uuid to non-primary journal tag

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: update mirror peer sync state after completion

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track mirror peer replay state

This new state will protect against the case of an rbd-mirror crash
between registering with the remote and starting the image sync.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: allocate local journal tags from mirror remote peer tags

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: helper method for retrieving tag struct

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: use consistent "non-primary" verbiage for image mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: image demotion should record new demote journal event

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: blacklist journal error while attempting to acquire lock

The journal policy will return -EPERM if attempting to acquire the
exclusive lock when the image is non-primary.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new demote journal event

This event will be used as a placeholder to ensure that the demotion journal
tag is properly processed by rbd-mirror.  Without an event associated with
the demotion journal tag, the demotion event will be missed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: framework for replay allocating tags in local journal

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: journal reset should re-register the local client

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: cleanup debug log messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: cross-reference local image via global image id

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: mirroring directory reverse lookup from global id to image id

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: bootstrap requires the global image id

The global image id can be used to cross-reference images that
have been replicated amongst peers.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: fix state output stream formatter

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: support creating images with non-primary journals

When rbd-mirror initially creates a replicated image in the
local pool, the journal should be flagged as non-primary to
(1) prevent local ops againsts the image and (2) provide a
clean chain of replicated tags.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: don't acquire lock for primary local images

Abort the image replay for any local images that are marked as primary.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: tweaks to support creating mock test cases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: bootstrap now has initial support for existing images

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: properly initialize image copy start/end objects

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: only delete ImageCtx after open failure

The image is already closed -- it just needs to be destroyed
in a thread context outside of librbd.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_journal: disallow client registration against missing journal

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: added helper to query single client record from disk

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: convert ImageReplayer into templated class

This will facilitate create mock test cases.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper methods to allocate tags associated to remote journals

Allocating tags for the local journal will always be based off the local
commit position.  For remote journals, the local commit position doesn't
reflect actual events replayed by rbd-mirror, so it needs to provide the
remote journal commit position.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: initial integration with librbd lock/journal policies

For the first stage, acquiring the exclusive-lock will no longer
automatically result in the allocation of a journal tag.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: basic policy for journal handling

This will allow rbd-mirror to override the allocation of a new
tag and to restart rbd-mirror's image replayer when a watch error
forces the loss of the exclusive lock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: optional 'force' parameter for request lock payload

This will be used when force promoting an image to primary -- when
rbd-mirror receives the forced request, it will immediately abort
playback.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: basic policy for controlling the release of the exclusive lock

The rbd-mirror daemon will need to interlock with the request for the lock
from another client with the successful sync of remote journal events.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: journal type traits should be standalone

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8332 from rjfd/wip-15267

rbd-mirror: prevent enabling/disabling an image's mirroring when not in image mode

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph.in: fix python libpath for automake as well

Follow-on to a041e5c9413fd816bc2858509319fa469fa233e5
due to confusion from .pyc files hanging around.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #8320 from dachary/wip-virtualenv-jewel

Makefile-env.am: set a default for CEPH_BUILD_VIRTUALENV (part 2)

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #8360 from jdurgin/wip-ceph-python-path

ceph.in: correct dev python path for automake builds

Tested-by: Sage Weil <sage@redhat.com>

Merge pull request #8361 from liewegas/wip-test-port

test/test_pool_create.sh: fix port

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'origin/jewel'

Merge pull request #8356 from ceph/wip-cython-build-dir

pybind/Makefile.am: Prevent race creating CYTHON_BUILD_DIR

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/test_pool_create.sh: fix port

17108 is used by test/mon/test_pool_quota.sh

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8359 from athanatos/wip-15205

os/ObjectStore: try_move_rename in transaction append and add coverage to store_test

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.in: correct dev python path for automake builds

Accidentally broken by ad2e6f442df0fe302c09fc751fc9e12237789511

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

store_test: apply an appended transaction half the time

Signed-off-by: Samuel Just <sjust@redhat.com>

Transaction: handle OP_TRY_RENAME in dump

Signed-off-by: Samuel Just <sjust@redhat.com>

store_test: add a TryMoveRename test

Signed-off-by: Samuel Just <sjust@redhat.com>