ECBackend: integrate cache and rmw pipeline

Implements the rmw pipeline and integrates the cache.

HashInfo now maintains a projected size for use during the planning
phase of the pipeline.

(Doesn't build without subsequent patches, not worth stubbing out
the interfaces)

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: refactor PGLog a bit and add support for rolling back extents

It was hard to reason about the validity of the IndexedLog internal
pointers and iterators during updates, so this patch cleans that up
a bunch.  It also moves responsibility for doing rollbacks into
PGBackend.  Finally, it adds support for the new log entry format.

Signed-off-by: Samuel Just <sjust@redhat.com>

TestPGLog: fix bug with merge_log_split_missing_entries_at_head

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: add support for rolling back overwritten extents to pg_log_entry_t

Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend: remove unused hobject argument to read_request_t

Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend: deduplicate start_remaining_read_ops and start_read_ops

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedBackend: always set rollforward to head in submit

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: split rollback info trims into trims and rollforwards

Also, rollforward on activate() and adjust read_log debugging to
account for non-rollforward entries.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: 's/trim_rollback_to/roll_forward_to/g'

trim_rollback_to was a not terrible name before in that all
it ever did is (possibly) trim the stashed version of the
object.  However, now, it's going to encompass, in general,
the roll_forward part of a tpc (which will still be to
delete the stashed object in cases where that is
appropriate).

Signed-off-by: Samuel Just <sjust@redhat.com>

osd_types: allow non-aligned non-overwrites with ECOVERWRITES flag

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: add ExtentCache

Signed-off-by: Samuel Just <sjust@redhat.com>

src/test: update ceph_test_rados to support overwrites

Signed-off-by: Samuel Just <sjust@redhat.com>

osd,mon: add pool FLAG_EC_OVERWRITES flag

For now, this is a white box testing flag to allow
us to start testing the supporting features before
ec overwrites can actually be implemented.

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Tomy Cheru <tomy.cheru@sandisk.com>

osd,mon: remove FLAG_DEBUG_FAKE_EC_POOL

This was used in the past as scaffolding while the ec pools were being
developed.  There should be no legitimate users.

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Tomy Cheru <tomy.cheru@sandisk.com>

osd_types: remove unused fill_in_setattrs

Signed-off-by: Samuel Just <sjust@redhat.com>

osd_types::ObjectModDesc: remove claim_append

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: switch all users of PGTransaction to use the new structure

This patch removes ReplicatedBackend::PGTransaction and implemenations
and switches over all users.  Happily, do_osd_ops loses the mod_desc
cruft and OpContext::pending_attrs.  PGTransaction doesn't really
have a natural way to implement append, however.  In reality, I think
this is probably an improvement, but it does mean that copy_from's
final transaction is now filled in by a lambda rather than by
appending a transaction fragment.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: introduce PGTransaction

ECBackend is going to need a transaction representation which reduces
the operational representation from the OSDOp to a descriptive one
which makes questions like "what is the largest offest written" and
"does this transaction delete the object?" simple to answer.  At the
same time, we're going to eliminate the PGBackend::PGTransaction
interface since I don't think writing directly to an
ObjectStore::Transaction is buying us enough to offset the irritation
of having to update both implemenations.

A happy consequence of this design will be that we can fill in the
pg_log_entry_t::mod_desc member after submission in the backend
rather than inline in do_osd_ops.  We can also dispense with having
to maintain OpContext::pending_attrs separately from the ongoing
PGTransaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

common/: add interval_map

Signed-off-by: Samuel Just <sjust@redhat.com>

inline_variant: simplify it a lot, enable perfect forwarding

The previous implementation was a bit more baroque than it
needed to be.  Also, it made copies of the lambdas in a
few places.  Finally, it caused segfaults.  Not actually
sure why.

Signed-off-by: Samuel Just <sjust@redhat.com>

common/: add match() utilities for boost::variant

C++ doesn't have a sum type with nice pattern matching syntax.
Fortunately, someone on stack overflow fixed that.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd_types: update_snaps should take a const argument

Signed-off-by: Samuel Just <sjust@redhat.com>

Context: add [Gen]LambdaContext and some related helpers

Signed-off-by: Samuel Just <sjust@redhat.com>

hobject: add helper typedefs

Signed-off-by: Samuel Just <sjust@redhat.com>

PGBackend: add DoutPrefixProvider to parent interface

Signed-off-by: Samuel Just <sjust@redhat.com>

FileStore::_do_copy_range: tolerate short reads on replay

Consider a sequence like:

0. foo object size is 15
1. clone_range foo -> foo.0 5~5
2. write foo 5~5
3. clone_range foo -> foo.1 10~5
4. write 10~5 foo
5. rename foo -> foo.1
6. remove foo.0
7. remove foo.1
8. remove foo.2

If this sequence is interupted after 8 and replayed from 1, by the time
it gets to 3 the object will only have size 10 and no replay guard
(since 1 was skipped and 2 recreated the object with size 10 resulting
in a short read.  This should only happen if the replay guard is
missing, which should only happen if the object gets deleted later
in the sequence.

Signed-off-by: Samuel Just <sjust@redhat.com>

store_test::col_split_test: send bounded size transactions

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart: ratchet down the osd_copyfrom_max_chunk to make multiple chunks likely

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12046 from dachary/wip-16014-cot

tests: use shorter directories for tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12036 from tchaikov/wip-osd-scrub-repair-cleanup

test/osd-scrub-repair.sh: use repair() instead of "ceph pg repair"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12039 from xiexingguo/xxg-wip-bluestreo-1117

os/bluestore: fix bufferspace stats leak due to blob splitting

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12042 from ifed01/wip-bluestore-drop-blob_depth

os/bluestore: remove garbage collector staff

Reviewed-by: Sage Weil <sage@redhat.com>

tests: use shorter directories for tests

So that jenkins can use longer directories. We can't have both otherwise
the limit UNIX domain socket path length limit triggers errors such as:

... client.admin.12750.asok is too long! The maximum length on this system is 107

Fixes: http://tracker.ceph.com/issues/16014
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #11961 from tianshan/fix-RGWSimpleRadosLockCR

rgw: fix RGWSimpleRadosLockCR set_description()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11965 from ceph/wip-rgw-fix-period-commit

rgw: fix missing master zone for a single zone zonegroup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12020 from leseb/glance-mitaka

doc: openstack glance mitaka uses show_multiple_locations

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: fix missing master zone for a single zone zonegroup

Fixes: http://tracker.ceph.com/issues/17364
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore: remove garbage collector staff

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11424 from renhwztetecs/renhw-wip-mgr

mgr: init() return when connection daemons failed && add some err info

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12015 from xiexingguo/xxg-wip-bluestore-init-csum

os/bluestore: avoid unnecessary call to init_csum()

Reviewed-by: Sage Weil <sage@redhat.com>

test/osd-scrub-repair.sh: use repair() instead of "ceph pg repair"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12038 from tchaikov/wip-ceph-objectstore-tool

test: ceph-objectstore-tool: should import platform before using it

Reviewed-by: Owen Synge <osynge@suse.com>

doc: openstack glance mitaka uses show_multiple_locations

As of the Mitaka release show_image_direct_url is not needed, but
instead show_multiple_locations should be used.
Adding the necessary guidance for Mitaka release.

Signed-off-by: Sébastien Han <seb@redhat.com>

test: ceph-objectstore-tool: should import platform before using it

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9613 from dzafman/wip-16064

common osd: Improve scrub analysis, list-inconsistent-obj output and osd-scrub-repair test

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10507 from wjwithagen/wip-wjw-tests-ceph_objectstore_tool

ceph_objectstore_tool.py: Don't use btrfs on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11890 from liupan1111/wip-code-clean-ups

os/filestore: handle EINTR returned by io_getevents()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11765 from bassamtabbara/wip-crushtool-fix

mon: if crushtool config is empty use internal crush test

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: fix bufferspace stats leak due to blob splitting

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11926 from dzafman/wip-test-17830

test: fix test-erasure-eio and osd-scrub-repair races (17830)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12030 from branch-predictor/bp-rmxattr-bl

librados: remove unused bufferlist from rados_write_op_rmxattr

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

test: Add sleep 1 so that we look for 60 seconds

This failed because test blew through 60 requests in 8 seceons before
the scrub repair even started on Jenkins.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix duplicate port in test-erasure-code-plugins.sh

Add comments about uniqueness of port number required

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #12032 from liewegas/wip-bluestore-warning

os/bluestore: fix warning and uninit variable

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mon: Fix compiler warnings, missed _dump() change

Caused by: af720cc87b4d631695c1549c31d57bfbc82bc6ba

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Re-enable osd-scrub-repair.sh

Signed-off-by: David Zafman <dzafman@redhat.com>

test: In osd-scrub-repair.sh fix use of wait_for_clean()

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #11996 from dachary/wip-17849-bootstrap-osd

mon,ceph-disk: add lockbox permissions to bootstrap-osd

Reviewed-by: Sage Weil <sage@redhat.com>

include/mempool: silence warnings

/home/sage/src/ceph4/src/include/mempool.h:421:68: warning: no return statement in function returning non-void [-Wreturn-type]
   void *operator new[](size_t size) { assert(0 == "no array new"); } \

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: initialize oldest

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12029 from batrick/mon-_dump-bugfix

mon: remove utime_t param in _dump

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

librados: remove unused bufferlist from rados_write_op_rmxattr

rados_write_op_rmxattr declares the bl bufferlist variable, then doesn't make
any use of it. Get rid of it, as it's probably an effect of copy+paste.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

Merge pull request #12028 from batrick/client-redundant-init

client: remove redundant initialization

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

test: Re-enable test-erasure-eio.sh

Signed-off-by: David Zafman <dzafman@redhat.com>

test: activate_osd() doesn't need to set crush

Tests use objectstore_tool() which stops and starts OSDs,
but may assume consistency of object locations.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: wait_for_clean() add sleep backoff

Reduce size of log on timeout by doing a backoff so that
we don't log 3000 loops at 1/10 second sleeps.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: CLEANUP to later enable TEST_rados_get_with_subreadall_eio_shard_*

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: remove utime_t param in _dump

Missing in af720cc87b4d631695c1549c31d57bfbc82bc6ba.

Found in warning:

In file included from /home/pdonnell/ceph/src/mds/MDSRank.h:18:0,
                 from /home/pdonnell/ceph/src/mds/MDBalancer.cc:18:
/home/pdonnell/ceph/src/common/TrackedOp.h:153:16: warning: ‘virtual void TrackedOp::_dump(ceph::Formatter*) const’ was hidden [-Woverloaded-virtual]
   virtual void _dump(Formatter *f) const {}
                ^
In file included from /home/pdonnell/ceph/src/mon/mon_types.h:23:0,
                 from /home/pdonnell/ceph/src/mon/MonMap.h:22,
                 from /home/pdonnell/ceph/src/mon/MonClient.h:20,
                 from /home/pdonnell/ceph/src/mds/MDBalancer.cc:19:
/home/pdonnell/ceph/src/mon/MonOpRequest.h:106:8: warning:   by ‘void MonOpRequest::_dump(utime_t, ceph::Formatter*) const’ [-Woverloaded-virtual]
   void _dump(utime_t now, Formatter *f) const {
        ^

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon,ceph-disk: add lockbox permissions to bootstrap-osd

ceph-disk --dmcrypt needs to put a config-key and authorize
the OSD to get it back. The corresponding permissions are
added to the bootstrap-osd profile in the monitor.

When preparing the OSD lockbox, use the bootstrap-osd profile instead of
implicitly requiring admin permissions to perform the initial config-key
and auth get-or-create operations.

Fixes: http://tracker.ceph.com/issues/17849
Signed-off-by: Loic Dachary <loic@dachary.org>

client: remove redundant initialization

messenger is already initialized in the constructor member
initialization list.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #12025 from dachary/wip-17923-tox

tests: no python3 tests for ceph-disk

Reviewed-by: Samuel Just <sjust@redhat.com>

tests: no python3 tests for ceph-disk

2.5.0 published today made it so an environment with no action fails
where it previously did not.

Fixes: http://tracker.ceph.com/issues/17923
Signed-off-by: Loic Dachary <loic@dachary.org>

doc, test: Add schemas for list-inconsistent-* rados command output

If jsonschema cmd is available use it to test output against schema

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Update testing for new list-inconsistent-obj output

Signed-off-by: David Zafman <dzafman@redhat.com>

rados, osd: Improve attrs output of list-inconsistent-obj

Persist the user_version and shard id of scrubbed obj
Rados command dump inconsistent obj's version and shard-id
    so they can be passed to repair command
Rados list-inconsistent-obj output of attrs
    Make attrs an array since there are more than one
    Use base64 encode for values with non-printable chars
    Add indication if base64 encoding used
Add checking for ss_attr_missing and ss_attr_corrupted
    Rename attr errors to attr_key_mismatch and attr_value_mismatch
Add missing size_mismatch_oi scrub checking
For erasure coded pools add ec_size_error and ec_hash_error not just read_error
Use oi_attr_missing and oi_attr_corrupted just like list-inconsistent-snap does
Pick an object info based on version and use that to find specific shards in error
    Check for object info inconsistency which should be rare
Make all errors based on comparing shards to each other object errors
    We don't want give the impression that we've picked the correct one

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #12024 from tchaikov/wip-ceph-erasure-code-corpus-arch

tests: sync ceph-erasure-code-corpus for using 'arch' not 'uname -p'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: sync ceph-erasure-code-corpus for using 'arch' not 'uname -p'

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12011 from ceph/wip-git-archive-all

git-archive-all.sh: use an actually unique tmp dir

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12007 from jcsp/wip-trackedop-now

common/TrackedOp: remove unused 'now' in _dump()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12016 from dachary/wip-17919-osd-crush

tests: osd-crush.sh must retry crush dump

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12012 from tchaikov/wip-sepia-bt

script/sepia_bt.sh: a script to prepare for debugging on teuthology@sepia

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11167 from jcsp/wip-17308

mon: make MDSMonitor tolerant of slow mon elections

Reviewed-by: John Spray <john.spray@redhat.com>

tests: osd-crush.sh must retry crush dump

After ceph osd setcrushmap, the script must retry because ceph osd crush
dump races against the update of the crushmap.

Fixes: http://tracker.ceph.com/issues/17919
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #11979 from ceph/wip-disable-osd-scrub-repair

test: disable osd-scrub-repair and test-erasure-eio

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/bluestore: more readability improvements

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: avoid unnecessary call to init_csum()

We have to initiate CSumType from 1, which represents CSUM_NONE,
to be aligned with OSDMnitor's pool_opts_t handling.

So we have to explicitly check against CSUM_NONE to skip init_csum(),
which will set FLAG_CSUM and alloc memory for csum_data and thus
shall be avoided whenever it is possible.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

script/sepia_bt.sh: a script to prepare for debugging on teuthology@sepia

Signed-off-by: Kefu Chai <kchai@redhat.com>

git-archive-all.sh: use an actually unique tmp dir

git archive into $TMPDIR/$(basename "$(pwd)").$FORMAT is not unique;
if two runs are running simultaneously, this will collide.  Make
TMPDIR actually unique, and then the cleanup can just remove the whole
directory as well.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #12003 from athanatos/wip-17899

OSDMonitor: only reject MOSDBoot based on up_from if inst matches

Reviewed-by: Sage Weil <sage@redhat.com>

mon: make MDSMonitor tolerant of slow mon elections

Previously MDS daemons would get failed incorrectly
when they appeared to have timed out due to
delays in calling into MDSMonitor that were
actually caused by e.g. slow leveldb writes leading
to slow mon elections.

Fixes: http://tracker.ceph.com/issues/17308
Signed-off-by: John Spray <john.spray@redhat.com>

mon: clean up MDSMonitor interface

This was almost all public.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11836 from batrick/client-new-20161108

 client: remove unnecessary bufferptr[] for writev

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11837 from batrick/client-unique-20161108

client: use unique_ptr

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11904 from stiopaa1/mds_scrubheader_passStringByConstRef

mds/ScrubHeader.h: pass string by const reference

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11651 from stiopaa1/mds_mdsrank_protectedDestructor

mds/MDSRank.h: make destructor protected

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11338 from stiopaa1/mds_sessionmap_unneededCount

mds/SessionMap.cc: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11957 from batrick/mds-deactivate-doc

doc: clarify mds deactivate purpose

Reviewed-by: John Spray <john.spray@redhat.com>

common/TrackedOp: remove unused 'now' in _dump()

Greg pointed this out in PR #11985

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11985 from batrick/op-tracker-mds-event

mds: revert "mds/Mutation: remove redundant _dump method"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11958 from batrick/cls-lua-lua_next

lua: use simpler lua_next traversal structure

Reviewed-by: John Spray <john.spray@redhat.com>