Merge remote-tracking branch 'gh/next'

FileJournal: introduce start_seq header entry

FileStore::header_t::start_seq now encodes the op seq which may be
written at FileStore::header_t::start.  This way, FileStore::open()
can pass a valid sequence number to read_entry for validation.
Otherwise, read_entry has no way of knowing whether a failure of a
read at header.start indicates that the journal was empty, or that
the entry is corrupt.  With start_seq, read_entry can assume
corruption if start_seq <= committed_up_to.

Fixes: #4527
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

FileJournal: fill in committed_up_to for old headers

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #176 from dachary/wip-4597

fix nspace assignment in LFNIndex::lfn_parse_object_name

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #196 from ceph/wip-mon-crush2

Wip mon crush2

Reviewed-by: Dan Mick <dan.mick@inktank.com>

mon: fix crush unit tests for idempotency

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #197 from ceph/wip-3266

mds: verify mds tell 'dumpcache <filename>' target does not exist

Reviewed-by: Greg Farnum <greg@inktank.com>

mds: verify mds tell 'dumpcache <filename>' target does not exist

Open target with O_CREAT|O_EXCL to ensure we don't overwrite some other
important file (like, say, /etc/passwd).  This is irritating because there
is not c++ ofstream equivalent for O_EXCL; kludge around it using
ostringstream instead.

Fixes: #3266
Signed-off-by: Sage Weil <sage@inktank.com>

mon: make 'osd crush unlink ..' idempotent

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #191 from ceph/wip-4582b

Fixes #4582.

Reviewed-by: Greg Farnum <greg@inktank.com>

mds: do not go through handle_mds_failure for oneself

A standby MDS can attempt the handle_mds_failure paths for itself, if
it sees the transition from up to down. This leads it to insert itself
into the resolve_gather set, which is bad. So check if the failed MDS
is the same as whoami, and abort if so. This fixes #4637.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge pull request #194 from ceph/wip-rbd-diff

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #195 from dalgaaf/wip-da-fix-make

Makefile.am: fix build of ceph_test_cors

Reviewed-by: Sage Weil <sage@inktank.com>

Makefile.am: fix build of ceph_test_cors

Fix build of ceph_test_cors: use $(CRYPTO_LIBS) instead of -lcryptopp.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #192 from ceph/wip-mon-disk-warn

mon: limit warnings about low mon disk space

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #193 from ceph/wip-mon-crush

mon: make 'osd crush rm|unlink ...' idempotent

Reviewed-by: Dan Mick dan.mick@inktank.com

Merge remote-tracking branch 'gh/wip-4623'

client: Kick waiters for max size

If the mds restarts without successfully logging a max size
cap update, the client waits indefinitely in Client::get_caps
on the waitfor_caps list.  So when the client gets an mds map
indicating a new active mds has replaced a down mds, we need to
kick the caps update request.  This patch mimics the behavior
in the kernel by setting the wanted_max_size
and requested_max_size to 0 and wakes up the waiters.

Fixes #4582.
Signed-off-by: Sam Lang <sam.lang@inktank.com>

client: log inode when waiting on max size

Logging the specific inode that the get_caps()
is waiting for helps during debugging of stuck waiters.

Signed-off-by: Sam Lang <sam.lang@inktank.com>

mon: limit warnings about low mon disk space

Only warn once per percentage point per epoch.

Signed-off-by: Sage Weil <sage@inktank.com>

librados: Fixes to list_snaps test changes

Add back assert for overlap
Fix snap size and remove unnecessary comments
Fix failing test cases

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph-disk:  CalledProcessError has no output keyword on 2.6

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>

librados: test for EINVAL on bad list_snaps() ioctx snapid

Signed-off-by: Sage Weil <sage@inktank.com>

librados: set SNAP_DIR on listsnaps command

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: fail with EINVAL if bad snapid set on list_snaps

Signed-off-by: David Zafman <david.zafman@inktank.com>

librados: fix list_snaps() tests

The ioctx must be set to SNAP_DIR, and the head does not include any
clones.

Bug #4623

Signed-off-by: David Zafman <david.zafman@inktank.com>

mon: make 'osd crush rm|unlink ...' idempotent

Return 0 if the item is not there, not -ENOENT.

Fixes: #4624
Signed-off-by: Sage Weil <sage@inktank.com>

librbd: simplify diff_iterate calls to list_snaps

We don't need the size.  Use the simpler API call.

Signed-off-by: Sage Weil <sage@inktank.com>

Fix comment typos

Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge pull request #185 from dalgaaf/wip-da-fix-misc-2

Bunch of fixes for issues from SCA

Merge pull request #186 from dalgaaf/wip-da-pylint

Fix smaller python issues

Merge pull request #187 from imjustmatthew/imjustmatthew_docs2

Adds "mds fail 0" command to operations commmand reference.

Merge pull request #188 from dmick/wip-test-config-key

test_mon_config_key.py: fix 'del' to clean up correctly internally

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

test_mon_config_key.py: fix 'del' to clean up correctly internally

Merge remote-tracking branch 'origin/wip-4619'

Reviewed-by: Greg Farnum <greg@inktank.com>

mds: initialize tableservers/clients on mds creation

The handle_mds_recovery(who) path initializes the anchorclients by having
the server send a 'ready' message on recovery when the server is active
and a peer becomes active.  Similarly, recovery_done() does the same when
the server becomes active.  However, this misses the creation path.  Handle
that explicitly in boot_create.

Fixes: #4619
Signed-off-by: Sage Weil <sage@inktank.com>

Revert "mds: trigger tableserver active/recovery hook even for self"

This reverts commit 968c6c0c9408b33904041e5ddbd9ea738e831713.

This will trigger the 'ready' message twice when we restart, because we
will trigger in both recovery_done() and handle_mds_recovery().

Signed-off-by: Sage Weil <sage@inktank.com>

README: fix dependency lists

1) automake appeared twice
2) make apt-get command match the list

Adds "mds fail 0" command to operations commmand reference.
Partially fixes #2206, though better documentation will eventually be needed.

Merge pull request #184 from dachary/wip-4617

explain what an inline xattr is and how it relates to omap

mds: trigger tableserver active/recovery hook even for self

The tableserver now sends a READY message to clients when they go active;
we need to do this even for our own local tableclients, or else they do
not initialize and hang on first use after bringing up a fresh cluster.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

ceph-disk: fix some (local) variable names

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: fix naming of local variable in is_mounted()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: merge twice defined function is_mounted(dev)

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: define exception type

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: fix Redefining name 'uuid' from outer scope

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: add missing space after comma

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.cc: reduce scope of variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.h: remove twice initialized purge_keys

Remove twice initialized purge_keys from RGWUserAdminOpState();

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_cors.cc:  prefer prefix ++operator for iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_cors.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_cors.cc: use %u to format unsigned in sprintf()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.cc: use 'true' directly instead of variable

Instead of passing 'true' via bool defer_user_update variable
in RGWUser::execute_modify() to keys.add() use it directly.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.cc: reduce scope of same_email in execute_modify()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.cc: remove some unused std::string variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_cors_swift.h: prefer prefix ++operator for iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_op.cc: prefer prefix ++operator for iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_op.cc: remove unsused variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_op.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_cors_s3.cc: prefer prefix ++operator for iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_cors_s3.cc: remove unused variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_cors_s3.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

explain what an inline xattr is and how it relates to omap

The logic of the configuration flags related to xattr is clarified to define what an inline xattr is and when storing in the object map is preferred.

http://tracker.ceph.com/issues/4617 refs #4617

Signed-off-by: Loic Dachary <loic@dachary.org>

rgw/rgw_cors.cc: fix inefficient usage of string::find()

Fix warning from cppcheck:
 [src/rgw/rgw_cors.cc:70]: (performance) Inefficient usage of
 string::find() in condition; string::compare() would be faster.

Instead of string::find() use boost::algorithm::starts_with().

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_cors.cc: prefer prefix ++operator for iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_cors.cc: use empty() instead of size() == 0

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_bucket.cc: prefer prefix ++operator for iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_bucket.cc: remove unused variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.h: initialize some bool variables in constructor

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.h: move initialization in initialization list

Move initialization of some variables from constructor body to
the initialization list.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rbd.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/Migrator.cc: prefer prefix ++operator for iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

fix nspace assignment in LFNIndex::lfn_parse_object_name

The unused ns variable is assigned to the nspace data member of the
parsed hobject_t. It would be nicer to use a constructor accepting
the namespace in argument.

http://tracker.ceph.com/issues/4597 refs #4597

Signed-off-by: Loic Dachary <loic@dachary.org>

rgw: Create RESTful endpoint for user and bucket administration.

Expose the following operations through a RESTful endpoint:
    user create
    user modify
    user remove
    subuser create
    subuser modify
    subuser remove
    key create
    key remove
    bucket list
    bucket stats
    bucket link
    bucket unlink
    bucket check
    bucket remove
    remove object

building on the existing /{admin} endpoint.

Signed-off-by caleb miles <caleb.miles@inktank.com>

doc/release-notes: v0.60

Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'next'

Merge pull request #181 from ceph/wip_4510

Scrub/repair should correctly handle truncation and EIO

Fixes #4510
Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

PG::_scan_list: assert if error is neither -EIO nor -ENOENT

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore: rename debug_delete_obj to debug_obj_on_delete

This should make the method intent less confusing.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: _scan_list can now handle EIO on read, stat, get_omap_header

Signed-off-by: Samuel Just <sam.just@inktank.com>

ObjectStore: add allow_eio to read, stat, get_omap_header

This will allow enlightened callers to handle EIO.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #183 from ceph/wip-4313-b

qa: workunits: mon: test 'config-key' store

Reviewed-by: Sage Weil <sage@inktank.com>

librados: test empty ObjectWriteOperation

Tests that #2673 is fixed.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #182 from ceph/wip-no-cors-without-rgw

Makefile.am: disable building ceph_test_cors when radosgw is not enabled

Makefile.am: disable building ceph_test_cors when radosgw is not enabled

This test depends on radosgw. Trying to build it without radosgw will
result in a compile error.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

v0.60

Merge remote-tracking branch 'gh/next'

Merge pull request #169 from ceph/wip-rbd-diff

rbd incremental backup/restore

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

librados: don't use lockdep for AioCompletionImpl

This is a quick workaround for the next branch. A more complete fix
will be done for the master branch. This does not affect correctness,
just what qa runs with lockdep enabled do.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Sage Weil <sage.weil@inktank.com>

test: fix signed/unsigned comparison in test_cors

Signed-off-by: Greg Farnum <greg@inktank.com>
Acked-by: Sage Weil <sage@inktank.com>

PG: don't compare auth with itself

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: pass authoritative scrub map to _scrub

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: read_error should trigger a repair in _compare_scrub_objects

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore,OSD: add mechanism for injecting EIO, truncating obj

This will be used in testing repair.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG::_select_auth_object: prefer a peer which did not hit a read error

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: make _select_auth_object smarter

Previously, we just picked the first one to have the object in
question.  Now, we will attempt to choose one that has as
much of the following as possible:
1) has the object (there must be one)
2) has an object_info attr
3) has a valid object_info attr
4) has an object_info whose size matches the scrubbed size

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge branch 'wip-mds'

mds: bump the protocol version.

We've changed quite a lot of the restart behavior, as well as one
of the message encodings. This is cheaper and easier than using feature bits,
and CephFS is still a tech preview or whatever, so let's cover them using this.

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: don't roll back prepared table updates

When table server is recovering, it re-sends 'agree' messages for
prepared table updates. It is possible table client receives an
'agree' messages before it commits the corresponding update. Don't
send 'rollback' message back to the server in this case.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: clear scatter dirty if replica inode has no auth subtree

This avoids sending superfluous scatterlock state to recovering MDS

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>