qa: pjd must run as root

Merge remote branch 'origin/mds_xsyn'

Merge remote branch 'origin/stable'

librbd: fix block_completion race condition

If block completions finished before all of them were scheduled, the
pending_count could reach 0, and the callback could be called more
than once. This caused the reference counting to be off, so the next
call to the callback passed an invalid pointer, resulting in crashes
like this (or more mysterious ones):

common/Mutex.h: In function 'void Mutex::Lock(bool)', in thread '0x7f5f37e62700'
common/Mutex.h: 118: FAILED assert(r == 0)
 1: (librbd::RBD::AioCompletion::get_return_value()+0x19c) [0x7f5f408835bc]
 2: /usr/bin/kvm() [0x4629fd]
 3: (librbd::AioCompletion::complete_block(librbd::AioBlockCompletion*, long)+0x13c) [0x7f5f4088ab5c]
 4: (librbd::rados_aio_sparse_read_cb(void*, void*)+0x8d) [0x7f5f4088b40d]
 5: (librados::RadosClient::C_aio_sparse_read_Ack::finish(int)+0x12e) [0x7f5f4050beee]
 6: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x783) [0x7f5f40517863]
 7: (librados::RadosClient::_dispatch(Message*)+0x3c) [0x7f5f404ffe8c]
 8: (librados::RadosClient::ms_dispatch(Message*)+0x33) [0x7f5f404fff53]
 9: (SimpleMessenger::dispatch_entry()+0x69d) [0x7f5f405bf5bd]
 10: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x7f5f4050052c]
 11: (()+0x6d8c) [0x7f5f40c9dd8c]
 12: (clone()+0x6d) [0x7f5f3e0b904d]

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: add AioCompletion debugging

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

uclient: path_walk should notice when it gets absolute paths.

Previously it assumed everything was relative, but we need to handle
absolute paths on occasion, and already have the code to do so!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

vstart: put .journal outside of osd data dir

This prevents it from getting cleaned up during mkfs.  Then we can make it
a symlink, or at the very least avoid deleting + reallocating it.

Signed-off-by: Sage Weil <sage@newdream.net>

qa: Make snaptest-git-ceph.sh clean up after itself.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

auth: remove CephCryptoManager

It doesn't do anything.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librbd: fix AioCompletion race condition

If block completions finished before all of them where scheduled, the
pending_count would reach 0, and the AioCompletion would be destroyed
since the refcount would also reach 0. To fix this, hold a reference
to the AioCompletion while scheduling more block completions.

This caused a crash like this when running qemu:

1: ceph::__ceph_assert_fail (assertion=<value optimized out>, file=<value optimized out>, line=<value optimized out>, func=<value optimized out>) at common/assert.cc:86
2: Lock (ictx=0x7f83d0001f00, off=5368705024, len=0, buf=0x2fac000 "", c=0x2cebe00) at common/Mutex.h:118
3: add_block_completion (ictx=0x7f83d0001f00, off=5368705024, len=0, buf=0x2fac000 "", c=0x2cebe00) at librbd.cc:159
4: librbd::aio_read (ictx=0x7f83d0001f00, off=5368705024, len=0, buf=0x2fac000 "", c=0x2cebe00) at librbd.cc:1368
5: rbd_aio_rw_vector (bs=<value optimized out>, sector_num=10485752, qiov=<value optimized out>, nb_sectors=<value optimized out>, cb=<value optimized out>, opaque=<value optimized out>, write=0) at block/rbd.c:626
6: qemu_rbd_aio_readv (bs=<value optimized out>, sector_num=<value optimized out>, qiov=<value optimized out>, nb_sectors=<value optimized out>, cb=<value optimized out>, opaque=<value optimized out>) at block/rbd.c:649
7: bdrv_aio_readv (bs=0x2ce92e0, sector_num=10485752, qiov=0x2de08f0, nb_sectors=8, cb=<value optimized out>, opaque=<value optimized out>)

Fixes: #998
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

monmaptool: implement --help

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/bufferlist: don't alloc large array on stack

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

.gitignore man/Makefile

blech

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: do not use automake for workunit makefiles

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa/workunits/direct_io: fit .gitignore

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

radosgw_admin: fix clitest help

crbdnamer: man page

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crbdnamer: +x

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: radosgw_admin: fix clitest

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: direct_io: fix warnings

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

configure: check for boost statechart header

Partially fixes: #1124
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

radosgw_admin: log show can dump info in xml, json

radosgw_admin: fix log show (with new poolid param)

radosgw_admin: pool create command

Merge branch 'wip-workunits'

librados: use empty(), not size()

Signed-off-by: Sage Weil <sage@newdream.net>

qa: test_sync_io: zero buffers prior to read

Be extra paranoid to ensure we read back the correct data and didn't
just find it in undefined memory.

Signed-off-by: Sage Weil <sage@newdream.net>

qa: test_sync_io improvements

Exit (without removing foo) on error.

Allow read- or write-only operation.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: log pool id, and store/retrieve pool id info

librados: expose pool id

Conflicts:

src/librados.cc

qa: fix test_shorT_dio_read

No args, hardcode filename.

Signed-off-by: Sage Weil <sage@newdream.net>

PG: clear scrub_received_maps in scrub_clear_state

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

qa: fix loopall, runallonce scripts

Run anything executable.

Signed-off-by: Sage Weil <sage@newdream.net>

qa: move compiled workunits into organized directory

No need for a separate qa/src/ dir; just put them where we want them.  We
need to ignore non-executable files when doing the bulk runs anyway.

Signed-off-by: Sage Weil <sage@newdream.net>

qa: add short directio read test

This tests for the bug fixed by linux commit:c3cd628.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: add henry's direct io tests

Not sure what the second one is testing, actually, but including it anyway.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: include $basedir/src/ (qa/src/) in path

This lets the workunits find stuff build in qa/src/, in case you are
running using these (old) scripts.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: add direct and sync io workunits

Just run the test programs in ../src

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: expand test_sync_io

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: organize workunits a bit

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: move compiled items into qa/src/

osd: fix head_exists fix

Needs to default to false, or else deletions don't work.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: don't update snapset head_exists unless full transaction succeeds

Make the update at the end of make_writeable, along with the other snapset
updates.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix make_writeable

Look at the ObjectState to determine if the head exists, not the snapset
(which doesn't reflect the prior object state).

Signed-off-by: Sage Weil <sage@newdream.net>

monc: protect msg source check with monc_lock

Push the locking up into ms_dispatch, so that we protect the source check
and don't race with session reopens.  Also simplifies things a bit.

Fixes: #1151
Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'mds_issue_caps'

Merge remote branch 'origin/stable'

Conflicts:
src/dumpjournal.cc

Merge remote branch 'origin/deb-pybind'

debian: Move python-ceph into section python.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: Strip and do -dbg magic for librgw too.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: Fix copy-pasto in librgw1 control paragraph.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: Depend on python-ceph in obsync.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: Depend on librgw1 in python-ceph.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: Package librgw.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

pybind: Open shared libs by their major version.

The *.so files are only in the -dev packages, and normal
operation should not require those. The major version
numbers represent incompatible API/ABI changes anyway.
The debian dependencies were already correctly including
the major version.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: Make git ignore obsync, gceph* package dirs.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: Prevent git ignore from recursing on *.substvars etc.

Should never have triggered but this is faster simpler
and correcter.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: Make git ignore all of the *.debhelper files.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: Dev packages no longer have soname in their names.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: Properly package the python bindings.

Build-depend on python-support. Add binary package
python-ceph, making it contain all the ceph python
packages, regardless of their name; the modules are
too small to deserve their own debs.

Make python-ceph depend only on librados2 for now.
librgw is not packaged yet.

Dropping unnecessary build-dep on python-dev, that's
only needed for compiling C extensions, and we're using
ctypes.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rgw: list bucket display the correct objects owners

boto_tool: add put_bucket_acl

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: don't allow handing out object ownership

librados: remove useless reference holding

librados: get reference to the io context for the pending async ops

heap_profiler: if log_dir is empty, don't try and log to root dir!

If log_dir was empty then the prefix would be of the form /mon.a
That isn't good, so fill it in so it writes to the current dir.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

coverage.sh: use .lcov instead of .info to avoid confusion

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge remote branch 'origin/wip_coverage'

Conflicts:
src/mon/Monitor.cc

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: remove required indexes when modifying user info

Conflicts:

src/rgw/rgw_admin.cc

mon: Follow the Message rules when forwarding tells

My initial commit was naughty, fix that up here by
creating a new message for each mon and just copying
the command.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: clear correct state bit in do_file_recover

The RECOVERING bit is set, not NEEDSRECOVER.  This way we match
_recovered().

Signed-off-by: Sage Weil <sage@newdream.net>

Thread: remove globals. Thread create must succeed

Remove the references to global variables from Thread.h. They are really
unecessary. In every case, the printout is followed by an assert which
will deliver the exact same information.

Assert that thread creation succeeds. Nobody was checking the return
value of Thread::create() previously. Added a new function,
Thread::try_create(), which programmers can use if they do want to check
the value of Thread::create() and handle it appropriately.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

LogClient: pass in CephContext explicitly

Again, the Messenger passed to LogClient may be NULL, so pass in the
CephContext explicitly.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Monitor: pass in CephContext explicitly

Sometimes we create a Monitor without a Messenger. So we can't pull the
CephContext out of the Messenger, because it may be NULL. Just specify
it explicitly in the Monitor constructor.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds, mon, osd: with coverage enabled, exit(0) when SIGTERM is received

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Revert "mon: add all_exit and exit commands"

This reverts commit 69092735c4b25c88ded58b17b1c198fafcf5931b.

Revert "mds: allow mds to 'exit immediately'"

This reverts commit c01961ff4b712a12e25e437adceae264273d7428.

Revert "osd: add command to exit cleanly"

This reverts commit 1cc78b6a864d36cbe1eda45890d7f361adb8201d.

mds: fix up MDCache::path_is_mine to remove a totally bogus assert.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mon: add new "tell" infrastructure.

This lets you issue commands
like "ceph mon tell 0 heap start_profiler". Hurray!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

CephContext: initialize module_type in ctor

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: be slightly more careful about checking if lock has caps

issued_caps_need_gather==true implies it, but we can skip the work if the
lock has no caps.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix xlock_finish do_issue checks

Should default to false, and only get set to true if there are caps for
this lock.  Among other things this means we don't set it for dentry
locks (which have no caps).

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix eval_gather callers passing pissue_gather as wrong arg

Whoops!

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix xlock_finish issue flag check

We were sometimes setting do_issue but not *pneed_issue.  Simplify by
setting do_issue internally to the function and then either issuing or
setting *pneed_issue at the end.

Also fix bug with second argument to eval_gather().

Signed-off-by: Sage Weil <sage@newdream.net>

mds: make pneed_issue required for eval(SimpleLock*)

Signed-off-by: Sage Weil <sage@newdream.net>

mds: explicitly issue_caps after file_recover

Signed-off-by: Sage Weil <sage@newdream.net>

mds: make pneed_issue required for *lock_finish()

Signed-off-by: Sage Weil <sage@newdream.net>

mds: efficiently issue_caps for OOO or extra locks in acquire_locks

Signed-off-by: Sage Weil <sage@newdream.net>

mds: explicitly issue_caps for rdlock_finish_set

Signed-off-by: Sage Weil <sage@newdream.net>

mds: explicitly issue_caps for slave xlock release

Signed-off-by: Sage Weil <sage@newdream.net>

mds: efficiently issue_caps for scatterlocks unlock during refragment

Signed-off-by: Sage Weil <sage@newdream.net>

mds: efficiently drop_rdlocks and drop_xlocks too

Signed-off-by: Sage Weil <sage@newdream.net>

mds: set or issue caps on lock state changes

Set pneed_issue (or issue ourselves) whenever we jump directly to the
target lock state.  Make sure we only do it if there are caps (cap shift)
for this particular lock.

Part of fix for #1084.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: make issue_caps from file_update_finish smarter

We do one funky thing in file_update_finish that only issues caps on a
single cap when max_size changes.  This is more commonly we see.  However,
if a lock changes state and we need to issue on the whole inode (for all
clients), avoid doing the cap-specific issue by checking the issue set.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: issue caps from drop_locks

In drop_locks, build a set of inodes we need to issue caps on.  Then do it
all at once.  This does two things:

 - it fixes the fact that currently a dropped lock leading to an eval and
   lock state change will not issue caps _at_all_
 - it ensure we only issue_caps once for each inode, even when we are
   dropping multiple locks on it.

This should fix #1084.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: pass pissue_caps through *lock_finish()

This allows *lock_finish() callers to handle the issue_caps themselves.
None of them do yet (this arg is still optional) so this is patch has no
functional change (yet!).

Signed-off-by: Sage Weil <sage@newdream.net>

CephContext: remove silly debug code

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Dispatcher should take a CephContext argument

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: fix tcmalloc check.

We can run commands to do things like start the profiler even if
it's not already running!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>