Merge pull request #191 from ceph/wip-4582b

Fixes #4582.

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #192 from ceph/wip-mon-disk-warn

mon: limit warnings about low mon disk space

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #193 from ceph/wip-mon-crush

mon: make 'osd crush rm|unlink ...' idempotent

Reviewed-by: Dan Mick dan.mick@inktank.com

Merge remote-tracking branch 'gh/wip-4623'

client: Kick waiters for max size

If the mds restarts without successfully logging a max size
cap update, the client waits indefinitely in Client::get_caps
on the waitfor_caps list.  So when the client gets an mds map
indicating a new active mds has replaced a down mds, we need to
kick the caps update request.  This patch mimics the behavior
in the kernel by setting the wanted_max_size
and requested_max_size to 0 and wakes up the waiters.

Fixes #4582.
Signed-off-by: Sam Lang <sam.lang@inktank.com>

client: log inode when waiting on max size

Logging the specific inode that the get_caps()
is waiting for helps during debugging of stuck waiters.

Signed-off-by: Sam Lang <sam.lang@inktank.com>

mon: limit warnings about low mon disk space

Only warn once per percentage point per epoch.

Signed-off-by: Sage Weil <sage@inktank.com>

librados: Fixes to list_snaps test changes

Add back assert for overlap
Fix snap size and remove unnecessary comments
Fix failing test cases

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph-disk:  CalledProcessError has no output keyword on 2.6

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>

librados: test for EINVAL on bad list_snaps() ioctx snapid

Signed-off-by: Sage Weil <sage@inktank.com>

librados: set SNAP_DIR on listsnaps command

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: fail with EINVAL if bad snapid set on list_snaps

Signed-off-by: David Zafman <david.zafman@inktank.com>

librados: fix list_snaps() tests

The ioctx must be set to SNAP_DIR, and the head does not include any
clones.

Bug #4623

Signed-off-by: David Zafman <david.zafman@inktank.com>

mon: make 'osd crush rm|unlink ...' idempotent

Return 0 if the item is not there, not -ENOENT.

Fixes: #4624
Signed-off-by: Sage Weil <sage@inktank.com>

Fix comment typos

Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge pull request #185 from dalgaaf/wip-da-fix-misc-2

Bunch of fixes for issues from SCA

Merge pull request #186 from dalgaaf/wip-da-pylint

Fix smaller python issues

Merge pull request #187 from imjustmatthew/imjustmatthew_docs2

Adds "mds fail 0" command to operations commmand reference.

Merge pull request #188 from dmick/wip-test-config-key

test_mon_config_key.py: fix 'del' to clean up correctly internally

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

test_mon_config_key.py: fix 'del' to clean up correctly internally

Merge remote-tracking branch 'origin/wip-4619'

Reviewed-by: Greg Farnum <greg@inktank.com>

mds: initialize tableservers/clients on mds creation

The handle_mds_recovery(who) path initializes the anchorclients by having
the server send a 'ready' message on recovery when the server is active
and a peer becomes active.  Similarly, recovery_done() does the same when
the server becomes active.  However, this misses the creation path.  Handle
that explicitly in boot_create.

Fixes: #4619
Signed-off-by: Sage Weil <sage@inktank.com>

Revert "mds: trigger tableserver active/recovery hook even for self"

This reverts commit 968c6c0c9408b33904041e5ddbd9ea738e831713.

This will trigger the 'ready' message twice when we restart, because we
will trigger in both recovery_done() and handle_mds_recovery().

Signed-off-by: Sage Weil <sage@inktank.com>

README: fix dependency lists

1) automake appeared twice
2) make apt-get command match the list

Adds "mds fail 0" command to operations commmand reference.
Partially fixes #2206, though better documentation will eventually be needed.

Merge pull request #184 from dachary/wip-4617

explain what an inline xattr is and how it relates to omap

mds: trigger tableserver active/recovery hook even for self

The tableserver now sends a READY message to clients when they go active;
we need to do this even for our own local tableclients, or else they do
not initialize and hang on first use after bringing up a fresh cluster.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

ceph-disk: fix some (local) variable names

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: fix naming of local variable in is_mounted()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: merge twice defined function is_mounted(dev)

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: define exception type

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: fix Redefining name 'uuid' from outer scope

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: add missing space after comma

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.cc: reduce scope of variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.h: remove twice initialized purge_keys

Remove twice initialized purge_keys from RGWUserAdminOpState();

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_cors.cc:  prefer prefix ++operator for iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_cors.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_cors.cc: use %u to format unsigned in sprintf()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.cc: use 'true' directly instead of variable

Instead of passing 'true' via bool defer_user_update variable
in RGWUser::execute_modify() to keys.add() use it directly.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.cc: reduce scope of same_email in execute_modify()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.cc: remove some unused std::string variables

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_cors_swift.h: prefer prefix ++operator for iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_op.cc: prefer prefix ++operator for iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_op.cc: remove unsused variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_op.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_cors_s3.cc: prefer prefix ++operator for iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_cors_s3.cc: remove unused variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_cors_s3.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

explain what an inline xattr is and how it relates to omap

The logic of the configuration flags related to xattr is clarified to define what an inline xattr is and when storing in the object map is preferred.

http://tracker.ceph.com/issues/4617 refs #4617

Signed-off-by: Loic Dachary <loic@dachary.org>

rgw/rgw_cors.cc: fix inefficient usage of string::find()

Fix warning from cppcheck:
 [src/rgw/rgw_cors.cc:70]: (performance) Inefficient usage of
 string::find() in condition; string::compare() would be faster.

Instead of string::find() use boost::algorithm::starts_with().

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_cors.cc: prefer prefix ++operator for iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_cors.cc: use empty() instead of size() == 0

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_bucket.cc: prefer prefix ++operator for iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_bucket.cc: remove unused variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.h: initialize some bool variables in constructor

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_user.h: move initialization in initialization list

Move initialization of some variables from constructor body to
the initialization list.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rbd.cc: use static_cast instead of C-Style cast

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/Migrator.cc: prefer prefix ++operator for iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw: Create RESTful endpoint for user and bucket administration.

Expose the following operations through a RESTful endpoint:
    user create
    user modify
    user remove
    subuser create
    subuser modify
    subuser remove
    key create
    key remove
    bucket list
    bucket stats
    bucket link
    bucket unlink
    bucket check
    bucket remove
    remove object

building on the existing /{admin} endpoint.

Signed-off-by caleb miles <caleb.miles@inktank.com>

doc/release-notes: v0.60

Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'next'

Merge pull request #181 from ceph/wip_4510

Scrub/repair should correctly handle truncation and EIO

Fixes #4510
Reviewed-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

PG::_scan_list: assert if error is neither -EIO nor -ENOENT

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore: rename debug_delete_obj to debug_obj_on_delete

This should make the method intent less confusing.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: _scan_list can now handle EIO on read, stat, get_omap_header

Signed-off-by: Samuel Just <sam.just@inktank.com>

ObjectStore: add allow_eio to read, stat, get_omap_header

This will allow enlightened callers to handle EIO.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #183 from ceph/wip-4313-b

qa: workunits: mon: test 'config-key' store

Reviewed-by: Sage Weil <sage@inktank.com>

librados: test empty ObjectWriteOperation

Tests that #2673 is fixed.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #182 from ceph/wip-no-cors-without-rgw

Makefile.am: disable building ceph_test_cors when radosgw is not enabled

Makefile.am: disable building ceph_test_cors when radosgw is not enabled

This test depends on radosgw. Trying to build it without radosgw will
result in a compile error.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

v0.60

Merge remote-tracking branch 'gh/next'

Merge pull request #169 from ceph/wip-rbd-diff

rbd incremental backup/restore

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

librados: don't use lockdep for AioCompletionImpl

This is a quick workaround for the next branch. A more complete fix
will be done for the master branch. This does not affect correctness,
just what qa runs with lockdep enabled do.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Sage Weil <sage.weil@inktank.com>

test: fix signed/unsigned comparison in test_cors

Signed-off-by: Greg Farnum <greg@inktank.com>
Acked-by: Sage Weil <sage@inktank.com>

PG: don't compare auth with itself

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: pass authoritative scrub map to _scrub

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: read_error should trigger a repair in _compare_scrub_objects

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore,OSD: add mechanism for injecting EIO, truncating obj

This will be used in testing repair.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG::_select_auth_object: prefer a peer which did not hit a read error

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: make _select_auth_object smarter

Previously, we just picked the first one to have the object in
question.  Now, we will attempt to choose one that has as
much of the following as possible:
1) has the object (there must be one)
2) has an object_info attr
3) has a valid object_info attr
4) has an object_info whose size matches the scrubbed size

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge branch 'wip-mds'

mds: bump the protocol version.

We've changed quite a lot of the restart behavior, as well as one
of the message encodings. This is cheaper and easier than using feature bits,
and CephFS is still a tech preview or whatever, so let's cover them using this.

Signed-off-by: Greg Farnum <greg@inktank.com>

mds: don't roll back prepared table updates

When table server is recovering, it re-sends 'agree' messages for
prepared table updates. It is possible table client receives an
'agree' messages before it commits the corresponding update. Don't
send 'rollback' message back to the server in this case.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: clear scatter dirty if replica inode has no auth subtree

This avoids sending superfluous scatterlock state to recovering MDS

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: don't replicate purging dentry

open_remote_ino is racy, it's possible someone deletes the inode's
last linkage while the MDS is discovering the inode.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: eval inodes with caps imported by cache rejoin message

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: try merging subtree after clear EXPORTBOUND

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: clear dirty inode rstat if import fails

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: don't open dirfrag while subtree is frozen

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: notify bystanders if export aborts

So bystanders know the subtree is single auth earlier.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: fix export cancel notification

The comment says that if the importer is dead, bystanders thinks the
exporter is the only auth, as per mdcache->handle_mds_failure(). But
there is no such code in MDCache::handle_mds_failure().

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: unfreeze subtree if import aborts in PREPPED state

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: check MDS peer's state through mdsmap

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: avoid double auth pin for file recovery

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: add dirty imported dirfrag to LogSegment

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: send lock action message when auth MDS is in proper state.

For rejoining object, don't send lock ACK message because lock states
are still uncertain. The lock ACK may confuse object's auth MDS and
trigger assertion.

If object's auth MDS is not active, just skip sending NUDGE, REQRDLOCK
and REQSCATTER messages. MDCache::handle_mds_recovery() will take care
of them.

Also defer caps release message until clientreplay or active

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: issue caps when lock state in replica become SYNC

because client can request READ caps from non-auth MDS.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: share inode max size after MDS recovers

The MDS may crash after journaling the new max size, but before sending
the new max size to the client. Later when the MDS recovers, the client
re-requests the new max size, but the MDS finds max size unchanged. So
the client waits for the new max size forever. This issue can be avoided
by checking client cap's last_sent, share inode max size if it is zero.
(reconnected cap's last_sent is zero)

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

mds: take object's versionlock when rejoinning xlock

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>