rest-bench: cleanups, initialization

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest-bench: create workqueue for requests dispatching

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest_bench: cleanups, implement get and put

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest_bench: some more implementation

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rest_bench: initial work

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rados_bencher: abstract away rados specific operations

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rados_bencher -> obj_bencher

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rados_bencher: fix build

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rados_bencher: restructure code, create RadosBencher class

Preparing for different benchmark backend.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rados_bencher: restructure code (initial work)

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

librados: call safe callback on read operation

This avoids confusion for the user who isn't sure if they should wait for
complete or safe on a read aio.  It also means that you can always wait
for safe for both reads or writes, which can simplify some code.

Dup the roundtrip functional tests to verify this works.

Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Yehuda Sadeh <yehuda.sadeh@inktank.com>

crush: note that tree bucket size is tree size, not item count

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/wip-crush-forcefeed'

Reviewed-by: Sam Just <sam.just@inktank.com>

OpRequest: ignore all ops while the oldest one is still young.

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
Reviewed-by: Sage Weil <sage@newdream.net>

crush: update_item() should pass an error back to the caller

If you give it a nonsensical loc, it will fail check_item_loc() (false) and
then error out on insert_item().

Reported-by: Sam Just <sam.just@inktank.com>
Signed-off-by: Sage Weil <sage@newdream.net>

crush: improve docs/comments for check_item_loc and insert_item semantics

We don't adjust the internal hierarchy structure (currently).  This is a
bit confusing, so describe the semantics in some detail.

Signed-off-by: Sage Weil <sage@newdream.net>

crush: comment and clean up checks for check_item_loc and insert_item

- drop useless cur for check_item_loc
- comment the checks we're doing so the code is understandable
- use name_exists instead of broken get_item_id != 0 check

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-crush-update'

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge branch 'wip-osd-uuid'

Reviewed-by: Greg Farnum <greg@inktank.com>

Makefile: fix $shell_scripts substution

No spaces here, apparently!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: simplify 'osd create <uuid>' command

Make the flow clearer for the three cases (exists, about to exist, new).

Signed-off-by: Sage Weil <sage@newdream.net>

crushtool: another simple test for update

If the weight doesn't change it should be a no-op.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crush: document return values

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crush: compare fixed-point weights in update_item

This is less ugly than converting the quantized value back to a float and
comparing that.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

thread: remove get_num_threads() static

This looks in /proc to count threads.  Kludgey and no longer needed.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

global_init: do not count threads before daemonize()

We were verifying that there was only 1 thread (the presumably main()) when
we call daemonize.  However, with the new logging code, we stop a thread
right before the check, and /proc apparently updates asynchronously such
that our attempt to count running threads gives us a bad answer.

Just remove this kludgey check; we'll have to catch this class of bugs
the hard way.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

crush: clean up check_item_loc() comments

Thanks Greg!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

OpRequest: only show a small set of the oldest messages, instead of all.

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: update cache interface for put_obj_meta

This fixes issue #2381.
The method interface was different than the one needed in order
to override the one in RGWRados.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

doc: fix some underscores

Signed-off-by: Sage Weil <sage@newdream.net>

osd: drop unused CEPH_OSDMAP*VERSION* #defines

It's easier to manage/rev/grok these inline.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-doc-rebase-2'

Fixed link to blog.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

Fixed another link to the blog.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

Fixed link.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

Clean up. Changed ceph.newdream.net to ceph.com.
Removed {ARCH} references.
Added link to Source.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

doc: more fonts

Signed-off-by: Sage Weil <sage@newdream.net>

doc: new theme

Signed-off-by: Ross Turk <ross.turk@inktank.com>

doc/install/debian: simplify more

Signed-off-by: Sage Weil <sage@newdream.net>

doc/install: reorg, simplify

Signed-off-by: Sage Weil <sage@newdream.net>

doc: clarify path names for mkcephfs

Signed-off-by: Sage Weil <sage@newdream.net>

doc: simplify quick start

Remove dup items

Signed-off-by: Sage Weil <sage@newdream.net>

doc: Whitespace cleanup.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Rename to use dashes not underscores in URLs.

This makes the-separate-words in the url match as separate words in
searches, where this_way only matches an explicit "this_way" search.
http://www.mattcutts.com/blog/dashes-vs-underscores/

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Remove leading whitespace, that syntax means it's a definition list.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Updated URLs for the new ceph.com site.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

objecter: ping connections with lingering ops every tick()

Also, make sure that we initialize linger_op.session once
we have it.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Removed "Ceph Development Status" per Bryan
Modified title syntax per Tommi
Modified paragraph width to 80-chars per Dan
Moved "Build from Source" out of Install
Renamed create_cluster to config-cluster
Added config-ref with configuration reference tables
Added a toc ref for man/1/obsync per Dan
Removed redundant sections from Ops
Deleted "Why use Ceph" and "Introduction to Storage Clusters"

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>

Merge branch 'stable'

Conflicts:
src/common/config_opts.h

config: $cluster-whatever for admin_socket, log files

This is more consistent with /var/lib/ceph/$type/$cluster-$id.

Signed-off-by: Sage Weil <sage@newdream.net>

config: Admin socket path had literal "name" in it instead of $name.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

config: Admin socket path had literal "name" in it instead of $name.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mon: make 'osd map <poolname> <objectname>' return pg and osd mapping

This keeps coming up and is annoying to do manually.

Fixes: #2141
Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

ceph-object-corpus: a few instances of the newly encoded types

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: ignore trailing goop after OSDMap and OSDMap::Incremental

All users pass around bufferlists and avoid encoding these structures
inline, but the dencoder tests are picky.  Disable that for these types so
that we can add new fields without noise.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: drop 'osd crush add ...'

'osd crush set ...' is better, us that instead.

Signed-off-by: Sage Weil <sage@newdream.net>

vstart.sh: use 'osd crush set ...'

Signed-off-by: Sage Weil <sage@newdream.net>

mon: 'osd crush set ...' do an add or update

This operation will add/update/move the item to the specified location.
It is idempotent and much more useful than 'osd crush add ...'.

Signed-off-by: Sage Weil <sage@newdream.net>

crushtool: extent cli test to include --remove-item and --update-item

Signed-off-by: Sage Weil <sage@newdream.net>

crushtool: add --update-item command

Similar to --add-item, except it will move, rename, or reweight the item if
it is already present in the map.

Signed-off-by: Sage Weil <sage@newdream.net>

crush: do some docs

Signed-off-by: Sage Weil <sage@newdream.net>

crush: implement update_item()

This is similar to insert_item(), except it will succeed if the item is
already there, and will move an item to the specified location if it is
not.  It returns 0 for no change, 1 if a chance was made.  It also makes
sure the weight and name match.

Signed-off-by: Sage Weil <sage@newdream.net>

crush: add check_item_loc

The check_item_loc() method will take an item and position and tell you if
it matches the items current location.  The matching is identical to that
used for insert_item, in that a specific location constraint match means
success, even if a less specific one does not match (e.g., rack=wrongrack,
host=correcthost will return true).

Signed-off-by: Sage Weil <sage@newdream.net>

crush: fix weights when removing items

Reweight an item to 0 before removing it, so that the parent weights are
adjusted accordingly.

Signed-off-by: Sage Weil <sage@newdream.net>

rbd.py: add new error for deleting an image with snapshots

Forgot to update the python bindings in
88dda3be5aed38c8f587af9b27cb54a0fcca7530

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

config: no -C shorthand for --cluster

ceph-authtool uses it!

Fixes: #2376
Signed-off-by: Sage Weil <sage@newdream.net>

Edits.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Corrections.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Significant updates to introduction, getting packages, building from source, installing packages, and creating a cluster.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: document NOIN, NOOUT, NOUP, NODOWN flags and flapping

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Removed this. As part of restructuring of install to include admin host.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Minor edits. Still WIP.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

The new files contain S3 APIs and a build from source doc.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Minor cleanup.

Signed off by: John Wilkins <john.wilkins@dreamhost.com>

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Added a new landing page graphic, and made some minor edits on the landing page.

Submitted by: John Wilkins <john.wilkins@dreamhost.com>

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Removed some files for reorg.

Submitted by: John Wilkins <john.wilkins@dreamhost.com>

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Added a plug for commercial support. ;)

Submitted by: John Wilkins <john.wilkins@dreamhost.com>

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Building out information architecture. Modified getting involved, why use ceph, etc.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Remove reference to Introduction to RADOS OSDs

Submitted by: John Wilkins <john.wilkins@dreamhost.com>

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Added introduction to clustered storage and deleted older files that have been moved.

Submitted by: John Wilkins <john.wilkins@dreamhost.com>

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Restructuring documentation.

Submitted by: John Wilkins <john.wilkins@dreamhost.com>

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Initial cut of introduction, getting started, and installing. More to do on installation. RADOS gateway to follow.

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Put change to rbd manpage in .rst instead, update man/ from output

doc: move documentation build instructions to doc/dev section

Signed-off-by: John Wilkins <john.wilkins@dreamhost.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Treat rename across pools as an error
Fixes: #2370
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

doc: add warning about multiple monitors on one machine.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: normalize bucket/obj before updating cache

Fixes bug #2369. The problem was that sometimes we send the
notification with the un-normalized bucket/obj pair. We
should make sure that we use the caonical name before doing
any cache update.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

vstart.sh: explicitly specify uuids during startup

This exercises all the new per-osd uuid code.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: --get-{osd,journal}-uuid synonyms for --get-{osd,journal}-fsid

Signed-off-by: Sage Weil <sage@newdream.net>

osd: allow uuid to be fed to mkfs with 'osd uuid' setting

E.g., ceph-osd --mkfs --osd-uuid <uuid> -i 123 ...

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: allow fsid to be fed in for mkfs

Mkfs currently always generates a new uuid.  Allow the caller to feed one
in.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: 'osd create <uuid>'

Make the osd create command idempotent by providing a uuid.  If you call it
multiple times with the same (or some other existing) uuid you'll get back
the osd id that is already using it.

Drop support for 'osd create <id>', which was mostly useless and
non-idempotent anyway.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: fill in osd uuid in map on boot

We may want to make this more strict, so that if it is defined it has to
match the map, and only fill it in when the map's uuid is still zeroed
(for legacy clusters)...

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: store a uuid for each osd

Rev the extended section of the map to store it.  Dump it when the osd
exists.  Zero it out if an osd is destroyed.  Provide some accessors to
identify an osd given a uuid (linear search).

Signed-off-by: Sage Weil <sage@newdream.net>

osd: make output less ugly

Signed-off-by: Sage Weil <sage@newdream.net>

osd: create a 'ready' file on mkfs completion

Signed-off-by: Sage Weil <sage@newdream.net>

osd: use fsync+rename when writing meta files (during mkfs)

It's overkill to do the dir fsync on each file, but not worth making
efficient.

Signed-off-by: Sage Weil <sage@newdream.net>

PG: Add probe set to HB peers during GetInfo

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: check_new_interval now handles adding new maps to past intervals

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

filestore: allow flusher+sync_flush enable/disable via injectargs

This only affects the decision to queue or do things inline, so it is safe
to change while the filestore is up and running.

Also adjust the #ifdef so that there we share a single path through the
code when sync_file_range() is missing.

Fixes: #2368
Signed-off-by: Sage Weil <sage@newdream.net>