Merge remote branch 'gh/wip-wrkldgen-throughput'

workloadgen: time tracking using ceph's utime_t's instead of timevals.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

workloadgen: forcing the user to specify a data and journal.

These default arguments, although handy when we just want to run the test,
just mess things up when we don't actually need them. If we don't specify
them on the CLI, we'll end up using the default ones, and that is just
annoying.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

workloadgen: add option to specify the max number of in-flight txs.

Use '--test-max-in-flight VAL' (default: 50) or check '--help' for more.
Also, allow the test to work even if we don't specify a conf file.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

workloadgen: Allow finer control over what the generator does.

  Allow the user to have more control on:
    - the sizes of the data being written by the operations;
    - which operations are suppressed from execution;
    - view the throughput;
    - specify the periodicity of throughput output.

For the CLI options, '--help' should suffice.

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>

Merge branch 'wip-rgw-bench'

Conflicts:
debian/rules

libs3: trailing / does strange things to EXTRA_DIST

drwxr-xr-x 1031/1031         0 2012-05-07 11:15 ceph-0.46/src/libs3/inc/
drwxr-xr-x 1031/1031         0 2012-05-04 15:28 ceph-0.46/src/libs3/inc/inc/
-rw-r--r-- 1031/1031      2343 2012-05-04 15:28 ceph-0.46/src/libs3/inc/inc/simplexml.h

etc.  Freaking autotools!

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/wip-osd-peering'

Reviewed-by: Sam Just <sam.just@inktank.com>

Makefile: drop librgw.so unittests

Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec: kill librgw

Signed-off-by: Sage Weil <sage@newdream.net>

debian: kill librgw.so

Signed-off-by: Sage Weil <sage@newdream.net>

osd: reset last_peering_interval on replica activate

There was a silent bug in the activate 'acks' that go from the replica back
to the primary.  Prior to 86aa07d7a91ac23074e76551c3a6db3a5736cffa, we
were passing same_interval_since to the callback, which mean that
sometimes _activate_committed() would ignore it and we wouldn't update
last_epoch_started.  This was mosty invisible; the next peering event would
just, in some cases, look at more past intervals than it needed to.

In 86aa07d7a91ac23074e76551c3a6db3a5736cffa we fixed this so that the check
is correct.  (We noticed because now we aren't setting the pg CLEAN flag
until after last_epoch_started is updated.)  That, in turn, revealed a
similar bug that we're fixing here: the replica's last_peering_reset could
be lower than the primary's, such that the activate 'ack' info is ignored.

To fix this, simply set last_peering_reset to the current epoch when the
replica activates; this will always be greater than the primary's.

Signed-off-by: Sage Weil <sage@inktank.com>

libs3: dist and distdir make targets

Signed-off-by: Sage Weil <sage@inktank.com>

Makefile: include libs3/ contents in dist tarball

Signed-off-by: Sage Weil <sage@inktank.com>

Makefile: osdc/Journaler is only used by the mds

Signed-off-by: Sage Weil <sage@inktank.com>

Makefile: librgw.la -> librgw.a; and use it

The various rgw tools were all recompiling my_libradosgw_src files over
again.  Instead build a single .a (not .la!) and link that in.

Signed-off-by: Sage Weil <sage@inktank.com>

Makefile: libos.la -> libos.a

There is a -laio associated with this, so use a var instead of referring to
it by name.

Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: libosd.la -> libosd.a

Faster build.

Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: libmon.la -> libmon.a

Builds >2x as fast.

Signed-off-by: Sage Weil <sage@newdream.net>

libs3: added 'make check' target

Signed-off-by: Sage Weil <sage@inktank.com>

debian: build-depend on libxml2-dev

Signed-off-by: Sage Weil <sage@inktank.com>

mon: add safety checks for 'mds rm <gid>' command

- make sure the gid exists
- only remove it if it's inactive (state < 0)

Fixes: #2188
Signed-off-by: Sage Weil <sage@inktank.com>

osd: do not mark pg clean until active is durable

Do not mark a PG CLEAN or set last_epoch_clean until after the PG activate
is stable on all replicas.

This effectively means that last_epoch_clean will never fall in an interval
that follows last_epoch_started's interval.  It *can* be >
last_epoch_started when it falls within the same interval.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: check against last_peering_reset in _activate_committed

We are checking against last_peering_reset in _activate_committed(), so we
need to pass in that value to compare against; last_peering_reset may be
greater than same_interval_since, e.g. on a replica that learns about the
PG after the initial creation epoch.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: tweak slow request warnings

- always include 'slow request' in the warning string
- only summarize if we warn about anything (they all may have backed off)
- be more concise

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

keyring: clean up error output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

keyring: catch key decode errors

Return EINVAL on decoding errors.

Other decode_base64() callers are already guarded.

Fixes: #2124
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

debian: depend on uuid-runtime

We use uuidgen for osd creation.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

safe_io: int -> ssize_t

int is 32-bit on 64-bit archs, but ssize_t is 64-bits.  This fixes overflow
when reading large (>2GB) extends.

Fixes: #2275
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: fix call to get_uuid() on non-existant osd

Didn't catch this with vstart.sh testing.

Signed-off-by: Sage Weil <sage@newdream.net>

debian: add rules for rest-bench

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest-bench: build conditionally

added configure --with-rest-bench, and configure --with-system-libs3

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

obj_bencher: changed interface

Passing bufferlist and not const bufferlist in aio_write(). We assign
it to another object which is not const, and it doesn't work too
well.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest-bench: change thread context for libs3 calls

Apparently S3_put_object() and S3_get_object() need to
run on the same thread as S3_runall_request_context() (at least
per context). So We now call them in the workqueue thread.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest-bench: change command line arg for seconds

seconds should be a param, not a command.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

obj_bencher: fix data encoding

There was a bug when doing a read with multiple threads, when
one of the threads was left behind; when it returned the compared
data string might have been cluttered by newer strings that
were longer.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

obj_bencher: use better round robin for completion slot scan

Start where left last time, don't start from zero.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest-bench: reuse libs3 handle

This is necessary for keep-alive to be useful. Otherwise a new
connection will be created for each request.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

obj_bencher: fix param order

seq benchmark was broken, passed params in wrong order.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest-bench: use refcount for req_state life cycle

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest-bench: multiple fixes

write seems to work

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest-bench: cleanups, initialization

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest-bench: create workqueue for requests dispatching

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest_bench: cleanups, implement get and put

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest_bench: some more implementation

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest_bench: initial work

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rados_bencher: abstract away rados specific operations

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rados_bencher -> obj_bencher

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rados_bencher: fix build

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rados_bencher: restructure code, create RadosBencher class

Preparing for different benchmark backend.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rados_bencher: restructure code (initial work)

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

librados: call safe callback on read operation

This avoids confusion for the user who isn't sure if they should wait for
complete or safe on a read aio.  It also means that you can always wait
for safe for both reads or writes, which can simplify some code.

Dup the roundtrip functional tests to verify this works.

Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Yehuda Sadeh <yehuda.sadeh@inktank.com>

crush: note that tree bucket size is tree size, not item count

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/wip-crush-forcefeed'

Reviewed-by: Sam Just <sam.just@inktank.com>

OpRequest: ignore all ops while the oldest one is still young.

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
Reviewed-by: Sage Weil <sage@newdream.net>

crush: update_item() should pass an error back to the caller

If you give it a nonsensical loc, it will fail check_item_loc() (false) and
then error out on insert_item().

Reported-by: Sam Just <sam.just@inktank.com>
Signed-off-by: Sage Weil <sage@newdream.net>

crush: improve docs/comments for check_item_loc and insert_item semantics

We don't adjust the internal hierarchy structure (currently).  This is a
bit confusing, so describe the semantics in some detail.

Signed-off-by: Sage Weil <sage@newdream.net>

crush: comment and clean up checks for check_item_loc and insert_item

- drop useless cur for check_item_loc
- comment the checks we're doing so the code is understandable
- use name_exists instead of broken get_item_id != 0 check

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-crush-update'

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge branch 'wip-osd-uuid'

Reviewed-by: Greg Farnum <greg@inktank.com>

Makefile: fix $shell_scripts substution

No spaces here, apparently!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: simplify 'osd create <uuid>' command

Make the flow clearer for the three cases (exists, about to exist, new).

Signed-off-by: Sage Weil <sage@newdream.net>

crushtool: another simple test for update

If the weight doesn't change it should be a no-op.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crush: document return values

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crush: compare fixed-point weights in update_item

This is less ugly than converting the quantized value back to a float and
comparing that.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

thread: remove get_num_threads() static

This looks in /proc to count threads.  Kludgey and no longer needed.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

global_init: do not count threads before daemonize()

We were verifying that there was only 1 thread (the presumably main()) when
we call daemonize.  However, with the new logging code, we stop a thread
right before the check, and /proc apparently updates asynchronously such
that our attempt to count running threads gives us a bad answer.

Just remove this kludgey check; we'll have to catch this class of bugs
the hard way.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

crush: clean up check_item_loc() comments

Thanks Greg!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

OpRequest: only show a small set of the oldest messages, instead of all.

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: update cache interface for put_obj_meta

This fixes issue #2381.
The method interface was different than the one needed in order
to override the one in RGWRados.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

doc: fix some underscores

Signed-off-by: Sage Weil <sage@newdream.net>

osd: drop unused CEPH_OSDMAP*VERSION* #defines

It's easier to manage/rev/grok these inline.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-doc-rebase-2'

Fixed link to blog.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

Fixed another link to the blog.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

Fixed link.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

Clean up. Changed ceph.newdream.net to ceph.com.
Removed {ARCH} references.
Added link to Source.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

doc: more fonts

Signed-off-by: Sage Weil <sage@newdream.net>

doc: new theme

Signed-off-by: Ross Turk <ross.turk@inktank.com>

doc/install/debian: simplify more

Signed-off-by: Sage Weil <sage@newdream.net>

doc/install: reorg, simplify

Signed-off-by: Sage Weil <sage@newdream.net>

doc: clarify path names for mkcephfs

Signed-off-by: Sage Weil <sage@newdream.net>

doc: simplify quick start

Remove dup items

Signed-off-by: Sage Weil <sage@newdream.net>

doc: Whitespace cleanup.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Rename to use dashes not underscores in URLs.

This makes the-separate-words in the url match as separate words in
searches, where this_way only matches an explicit "this_way" search.
http://www.mattcutts.com/blog/dashes-vs-underscores/

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Remove leading whitespace, that syntax means it's a definition list.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Updated URLs for the new ceph.com site.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

objecter: ping connections with lingering ops every tick()

Also, make sure that we initialize linger_op.session once
we have it.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Removed "Ceph Development Status" per Bryan
Modified title syntax per Tommi
Modified paragraph width to 80-chars per Dan
Moved "Build from Source" out of Install
Renamed create_cluster to config-cluster
Added config-ref with configuration reference tables
Added a toc ref for man/1/obsync per Dan
Removed redundant sections from Ops
Deleted "Why use Ceph" and "Introduction to Storage Clusters"

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

Merge branch 'stable'

Conflicts:
src/common/config_opts.h

config: $cluster-whatever for admin_socket, log files

This is more consistent with /var/lib/ceph/$type/$cluster-$id.

Signed-off-by: Sage Weil <sage@newdream.net>

config: Admin socket path had literal "name" in it instead of $name.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

config: Admin socket path had literal "name" in it instead of $name.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mon: make 'osd map <poolname> <objectname>' return pg and osd mapping

This keeps coming up and is annoying to do manually.

Fixes: #2141
Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

ceph-object-corpus: a few instances of the newly encoded types

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: ignore trailing goop after OSDMap and OSDMap::Incremental

All users pass around bufferlists and avoid encoding these structures
inline, but the dencoder tests are picky.  Disable that for these types so
that we can add new fields without noise.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: drop 'osd crush add ...'

'osd crush set ...' is better, us that instead.

Signed-off-by: Sage Weil <sage@newdream.net>

vstart.sh: use 'osd crush set ...'

Signed-off-by: Sage Weil <sage@newdream.net>

mon: 'osd crush set ...' do an add or update

This operation will add/update/move the item to the specified location.
It is idempotent and much more useful than 'osd crush add ...'.

Signed-off-by: Sage Weil <sage@newdream.net>

crushtool: extent cli test to include --remove-item and --update-item

Signed-off-by: Sage Weil <sage@newdream.net>