qa: drop rgw.py from test_python.sh workunit

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph.spec.in: drop rgw.py

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: drop rgw.py

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: fix full/nearfull ratio initialization

create_pending() is called before create_initial(), so just put these in
the first incremental.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: allow subuser creation on radosgw-admin user create

This fixes issue #2312: allow the creation of both user
and a subuser in a single radosgw-admin command.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

osd: dump old ops singly rather than all at once.

Fixes #2269. Convert the OpTracker::check_ops_in_flight interface
to take a vector<string> and create a separate warning for each old
Op, and dump those singly to the clog in the OSD.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
Reviewed-by: Sage Weil <sage@newdream.net>

osdmap: restructure flow in OSDMap::decode

Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

ceph-object-corpus: prune archive

Signed-off-by: Sage Weil <sage@newdream.net>

test/encoding/import.sh: fix

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/wip_journal'

Reviewed-by: Sage Weil <sage@newdream.net>

osdmap: remove bad cct reference

Not in OSDMap

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-2286-ratio-a'

Reviewed-by: Sage Weil <sage@newdream.net>

osdmap: workaround old bug that zeroed pool_max

Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

python-ceph: remove rgw bindings

We shouldn't expose these internal apis.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

pgmap: allow Incrementals to specify [near]full_ratios of 0

This commit isn't entirely safe: old monitors used 0 to mean "no change".
We can revert this (and the PGMonitor.cc portion of
841f2885318d1bcf37aab3f2947b1f40fee772a9) if we don't want to allow
0 as a valid ratio setting, and to maintain perfect backwards
compatibility.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

pgmon: add "set_full_ratio x" and "set_nearfull_ratio x" commands

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

pgmon: convert PGMonitor::prepare_command to our standard infrastructure.

So that I can add another command.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

pgmon: remove the PGMonitor update_full_ratios stuff

Making it a config watcher is just a huge mess in terms of consistently
updating it appropriately.
The next commit will add a monitor command for changing it.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mon: only fill in full/nearfull sets if the ratio > 0

This avoids putting all OSDs in both sets when the ratios are 0, as they
are with a fresh cluster and pgmap.  This also makes setting the ratio to
0 effectively disable the full/nearfull feature.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: unconditionally encode PGMap full ratios in the Incremental

This properly spreads the real value to peon monitors -- they weren't
seeing the right values at all before.
Initialize all related values to zero so that it's obvious if they
somehow avoided becoming set properly.
This doesn't require any kind of protocol revision, luckily -- mixing
monitors from before and after this change might result in extra work
recalculating full sets, but it won't spread bad values or anything.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw-admin: user create is idempotent

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>
Reviewed-by: Sage Weil <sage@newdream.net>

osdmap: dump pool_max

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-2289-2'

FileJournal: let peek_write return a reference

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

common: add blkdev.{h,cc}

Signed-off-by: Sage Weil <sage@newdream.net>

rbd: fix probe of source when block device

Fixes: #2304
Signed-off-by: Sage Weil <sage@newdream.net>

filestore: factor block device size probe out into helper

Signed-off-by: Sage Weil <sage@newdream.net>

FileJournal: clarify locking in header

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

OpRequest: mark_event output is now at debug 5

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

librados: add methods to check and wait for safe/complete -and- callback

Allow user to check for (safe|complete) -and- callback return, and wait
for that condition.

Fixes: #2301
Signed-off-by: Sage Weil <sage@newdream.net>

librbd: 'rbd cache enabled' -> 'rbd cache'

'enabled' is useless verbiage.  We should fix the rgw option too,
protably...

Signed-off-by: Sage Weil <sage@newdream.net>

qa: comment out xfstest 232

Test 232 in the xfstests suite produces an XFS error in the log
when run over an RBD device.  This is most likely an XFS problem
that will be tracked separately (in tracker 2302).

My original plan with getting this checked in was to have it run a
baseline set of the tests--all known to pass on rbd devices--with
the intention of doing ongoing work to add back missing tests (at
least from the "auto" group) as we understand and fix whatever
makes them produce failures.

So just comment out test 232 so the xfstests script is able to
run to completion without error.

Signed-off-by: Alex Elder <elder@dreamhost.com>

objecter: recalc pgid in case it wasn't a precalculated pgid

We had an issue where we didn't call it since we relied on the
existance of the object name and we were passed an empty one.
However, we still need to handle the case of list_objects, where
we don't want to recalc pgid.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

cls_rgw: rgw_dir_suggest_changes move cur_disk_bl inside loop

Otherwise we'd fail in case the cls_cxx_read_key failed, since
we rely on its size.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Merge branch 'wip-oc-perf'

librados: test get_pool_name

Signed-off-by: Sage Weil <sage@newdream.net>

librados: C bindings for get_pool_name()

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'upstream/wip-guard'

Filestore: fix guard errors in COLL_REMOVE and COLL_MOVE

Use object rather than collection check in COLL_REMOVE.  Adjust
COLL_MOVE check for -1/1 distinction.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: skip empty entries when listing a bucket

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

cls_rgw: rgw_dir_suggest_changes() don't try to decode cur_disk

don't try to decode cur_disk if bufferlist is zero sized.

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

cls_rgw: don't zero out error code before reading value

We were zeroing the return code before reading it.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Merge branch 'stable'

test_librbd: fix write sizes

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test_idempotent_sequence: no config file

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

run_xfstests.sh: ensure cleanup on errors

Because we exit on any error (due to 'set -e'), the cleanup call was
never getting made in the event of an error.  The net effect of that
was that a filesystem could be left mounted, and rbd cleanup then
couldn't complete because the module was in use.

Fix the trap call so it calls cleanup on exit as well as error.
Switch to using the capitalized signal names in the call.

Signed-off-by: Alex Elder <elder@dreamhost.com>

run_xfstests.sh: pass test result via exit status

It turns out that xfstests *does* exit with non-zero status
when a test fails.  Its exit status is the number of tests
that failed (which, now that we have over 255 tests could be
an issue...)

Save the exit status and make it be the result of the run.

Signed-off-by: Alex Elder <elder@dreamhost.com

objclass: fix cls_cxx_map_remove_key()

didn't set the correct osd op, didn't do anything.

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

rgw: call dir_suggest_changes with correct bufferlist

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

objectcacher: name them

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librados: add IoCtx::get_pool_name() to c++ api

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ObjectCacher: use "objectcacher" instead of the CephContext::name for perfcounter init

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ObjectCacher: group all the function definitions properly

Some of the ObjectCacher functions were separate, which works poorly
if you're trying to do things like use ldout.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ObjectCacher: add a bunch of perfcounters

Hurray, we can collect cache statistics!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ObjectCacher: move functions out of the header.

These all set off my "big-functions-in-the-header" alarm.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

objectcacher: perf skeleton

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>
Conflicts:

src/osdc/ObjectCacher.cc
src/osdc/ObjectCacher.h

qa: add run_xfsests.sh script

Add a script that runs xfstests over a pair of devices that are
specified using command line arguments.  The tests are run using
a specified filesystem type (xfs, ext4, or btrfs).

A default set of tests is run if none is specified on the command
line.  Normally there's an "auto" group used for this purpose, but
for now I've laid out a (large) subset of them that I know pass on
rbd devices.  These can be updated as we find they work reliably.

Signed-off-by: Alex Elder <elder@dreamhost.com>

test/rbd/fsx: fix signed/unsigned warnings

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librbd: implement discard

Implement sync and async discard.  Embed an ObjectWriteOperation in the
BlockCompletion struct.

The sync version does a sync op on every block, just like write()... very
stupid.  Both of these should fixed.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip-librbd-caching'

Reviewed-by: Sage Weil <sage.weil@dreamhost.com>

test_librbd_fsx: clean up properly

Delete and close everything on exit.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: flush pending writes when a new snapshot is created

This makes sure the state is as consistent as librbd can make it
before the snapshot is actually created.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

test: port fsx to run on an rbd image instead of a file

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

test: copy fsx to rbd directory

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

.gitignore: clean up to not match so many things recursively

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ObjectCacher: include object when outputting a BufferHead

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

finisher: add indent settings header

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

test_librbd: flush before opening another handle to the image

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ObjectCacher: remove useless conditional

We assert this condition at the beginning of the function.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ObjectCacher: always delete callback in commit_set

The GatherBuilder used later always deletes it. To make
the interface sane, delete it in every return path.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados: debug logging for aio writes

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ObjectCacher: remove unused/unimplemented methods

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ObjectCacher: remove redundant 'public:'

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Remove unused Completion.h header

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ObjectCacher: add some asserts about when the lock is held

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: flush cache before creating a snapshot

This is a temporary workaround until the ObjectCacher
is smarter about snapshots.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: fix bytes read accounting in read_iterate

ObjectCacher will never do short reads, and always returns 0.
librados may do short reads at the end of an object.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: check for writes to snapshots

librados does this for us normally, but caching does not check for this.
We might as well check early to avoid scheduling a bunch of aios anyway.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: allow writeback caching

This uses the existing infrastructure of ObjectCacher for
buffer management and expiry.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: remove writeback window

This is superseded by a full-fledged writeback cache.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ObjectCacher: remove dependency on Objecter

Abstract out how writeback is done with a WritebackHandler object.
For RBD caching, this will be done by librados, but the Client uses
the Objecter directly.

This also requires different locks, since librbd does not have access
to the lock the underlying Objecter uses. Thus, both lock and the
writeback handler are parameters of the ObjectCacher constructor.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Filer: make file_to_extents static

Now the ObjectCacher has no need for a Filer instance (not that
it had a good reason for one before).

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ObjectCacher: check for read/write errors

The objecter doesn't fix every error for us.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados: do aio callbacks in async thread

Call user completions in an async thread.  This allows callers to call back
into librados from the callback, and allows them to take locks in their
callbacks that they hold when queuing requests (making their life much
easier).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librados: unconditonally associate IoCtxImpl with AioCompletionImpl

We were only setting the pointer if we were on the aio_write_list.  Set
this pointer for any aio.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: flush log on failure injection

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip-2226b'

Conflicts:
src/common/config_opts.h
src/os/FileStore.cc
src/os/FileStore.h

run_seed_to: assume tool is in path, not cwd

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

run_seed_to_range.sh: run filestore idempotent sequence tests over a range

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

run_seed_to.sh: set -e

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: use our assert

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test_idempotent_sequence: simple test script

Takes seed and kill point, then verifies the result.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

DeterministicOpSequence: writing to object being cloned in the same tx.

We write to the object being cloned prior to the clone to ensure we are
cloning a valid range of bytes.

The write and the clone were being done in two distinct transactions,
which would trigger a diff mismatch if a failure happened to occur within
the write tx. By not differentiating the transactions when building a
pristine copy, we were executing one more transaction (the clone_range one)
than the transactions that were executed in the failed filestore, thus
triggering the mismatch (one more object in the pristine filestore than on
the failed filestore).

Now we issue a single transaction, containing both the clone_range()
preceded by a write().

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>

test_idempotent_sequence: outputting collection being diff'ed.

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>

test_idempotent_sequence: Add more info on diff's dout's.

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>

FileStore: don't return ENOENT from object_map getters

ENOENT in those contexts means that the object_map does not know
about the object, not that the object does not exist.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

test_idempotent_sequence: make get-last-op return 0 if no txns applied

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test_idempotent_sequence: return true from diff if different

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test_idempotent_sequence: include object name in diff output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test_idempotent_sequence: fix diff return value, mount behavior

No mkfs!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>