Merge branch 'wip-flock'

rados tool: fix rados df formatting

rgw: catch buffer::error exceptions thrown at decode

mds: flock: remove auto-insertion of waiting locks.

I don't remember precisely why I had the locking do this (maybe
just for self-containment), but with the messages being re-processed
anyway it's just redundant, and was leading to lots of erroneous
EINTR results.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: user info dumps json/xml

doc: Deb install should work now.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Write about deb installation.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rgw: log of nonexsistent bucket config option

qa: test false

mds: flock: tidy up comments

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

flock: move implementation into a separate c++ file

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

OSD: Fix flipped error codes in do_convertfs

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: poll allocation thread

rgw: user removal delets user index last

and also stop if failed to remove anything, so that
we're not left out with partial user info scattered around.

rgw: multipart meta object uses tmap

doc: Architecture, placeholder in install, and first appendix.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Make object store diagram fit in 1024 pix wide browser.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Move internals into a new section, /dev.

Most of the doc is user-oriented, let's isolate internals more.

Split into multiple files. Clean up RST.

Use Sphinx's graphviz plugin for graphs.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

objectcacher: write vs truncate races are ok

It is possible to write a buffer, truncate it away, and then not have it
when we get the commit.

Fixes: #1467
Signed-off-by: Sage Weil <sage@newdream.net>

mds: truncate sanity checks

Hunting #1472.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: configurable thread pool size

cfuse: initialize Fh to NULL

Avoid debugging confusion.

Signed-off-by: Sage Weil <sage@newdream.net>

client: clean up _{create,release}_fh

Signed-off-by: Sage Weil <sage@newdream.net>

mon: fix 'osd create ...' error paths

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librbd: cosmetic changes to progress stuff

Signed-off-by: Sage Weil <sage@newdream.net>

Finishing moving src/doc/object_store.dot to doc/

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add object store architecture overview picture

It requires graphviz / dot to be installed. It generates a nice graph
of the dependencies between classes in the object store.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'wip-librbd-async-copy'

qa: add multiple_rsync.sh workunit

For #1318

Signed-off-by: Sage Weil <sage@newdream.net>

osd: one more flush before collection_list

Signed-off-by: Sage Weil <sage@newdream.net>

mds flock: make get_overlapping_locks and share_space handle 0 lengths

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: flush previous operations to fs before collection list + destroy

We need to flush any prior ops to the fs before we can rely on
collection_list to return all the objects we need to delete.  If we miss
any, we will crash shortly after this when the rmdir(2) fails with
-ENOTEMPTY (as with #1471).

Signed-off-by: Sage Weil <sage@newdream.net>

osd: whitespace

architecture.rst: describe debug logs, cephcontext

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

MOSDSubOpReply/MSDSubOp: No need to update encoding version.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FlatIndex: lfn_get needs to set *exist for short filenames

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

OSD: Fix encoding versions affected by hobject switch

PG log did not previously store the object locator.  To get the hash for
the hobject, scan the collection for the object during read_log if we
encouter an old style log entry.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

librbd: improve copy_with_progress

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge remote branch 'origin/wip-scrub-oi-size'

Makefile: os/CollectionIndex.h

Signed-off-by: Sage Weil <sage@newdream.net>

assert: work around libatomic_ops vs assert in a less lame way

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: reorder includes

Signed-off-by: Sage Weil <sage@newdream.net>

assert: use our assert

Signed-off-by: Sage Weil <sage@newdream.net>

crypto: reinclude our assert to clobber system one

Signed-off-by: Sage Weil <sage@newdream.net>

global_init: make startup line consistent, less ugly

Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: add os/*Index.h to dist tarball

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

doc/architecture: describe lib arch, config arch

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: fix osd reply message

cosd,OSD: Improve filestore upgrade path

Previously, fsconverter was required to update an osd filestore to the
most recent version.  cosd will now handle that automatically on
startup.  cosd --convert-filestore will also update the FileStore
to the most recent version.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ObjectStore: Support older format on journal replay

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

librados: don't hide error in case of failed pool ctx creation

Merge branch 'stable'

client: plug leak of inode refs on seekdir(0)

Signed-off-by: Sage Weil <sage@newdream.net>

client: fix iterator adjustment in readdir assimilation

We need to move past the item we already had or else we'll remove it again
during the next iteration of the loop.

Signed-off-by: Sage Weil <sage@newdream.net>

client: drop mostly-useless relink()

Just use unlink() and then link().  Carry an inode ref to avoid badness.
The relink() is left over from a simpler time when we didn't do proper
refcounting.

Signed-off-by: Sage Weil <sage@newdream.net>

doc: Hide empty and broken "module index" page.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Disable static file copying, we don't use it.

Avoids a warning from sphinx on every run.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Flesh out librados API example, to show doxygen & sphinx integration.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

JournalingObjectStore: journal_replay committed_seq fix

Previously, committed_seq was set to op_seq at the end of
replay_journal.  committed_seq should not actually be set until the
sync thread runs.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

doc: Adjust how table of contents is presented.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

client: additional sanity checks on link/unlink

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'vapre/stable'

client: simplify Inode operator<<

Signed-off-by: Sage Weil <sage@newdream.net>

client: fix readder result merge

When merging readdir results into the cache, we want to remove any names
_preceeding_ the current item before updating it.  Then, at the end, we
clean up the trailing items.

This fixes a cfuse crash on workunits/snaps/snaptest-2.sh.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: relink_inode -> relink

and some comments.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: add some asserts

Add sanity checks in close_dir(), and make sure we aren't replacing a dn
with the same name in link().

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: fix readder result merge

When merging readdir results into the cache, we want to remove any names
_preceeding_ the current item before updating it.  Then, at the end, we
clean up the trailing items.

This fixes a cfuse crash on workunits/snaps/snaptest-2.sh.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

os/hashindex: fix #include

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip_colindex'

osd/,os/,osdc/: Convert collection_list_handle_t to a struct

Previously, we stored the collection_list handle as an opaque
uint64_t.  Now, collection_list_handle_t is a struct defined
in types.h.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

fsconverter.cc: Added fsconverter

fsconverter updates an existing FileStore to the most recent FileStore
format

Signed-off-by: Samuel Just <rexludorum@gmail.com>

FileStore: On mount, scan collections for unstable state

CollectionIndex implementations may perform compound operations
leaving invalid state if interrupted.  index->cleanup() gives
the implementation an oportunity to cleanup any in-progress
operation.  For HashIndex, split and merge fall in this
category

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileStore: Added update_version_stamp

update_version_stamp may be used to refresh the filestore version
stamp when all collections have been brought to the current
version.

Signed-off-by: Samuel Just <rexludorum@gmail.com>

FileStore: Add filestore version stamp

A filestore will now be tagged with a version stamp during
mkfs.  If on mount the version stamp detected lags the current
version, the mount will fail unless filestore_update_collections
is set in gconf.  If it is set, opening a collection will cause
the version stamp on the collection to be read and the
appropriate indexing implmentation to be used.  This will allow
for conversion from old collection indexing schemes to new
ones.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd_types.h: Fix support for old object_info_t

The object locator in the old encoding is necessary
to get the hash for soid.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileStore: Integrates HashIndex/IndexManager into FileStore

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

os/FlatIndex: FlatIndex encapsulates previous FileStore behavior

FlatIndex will be used to read and convert the previous on disk
format.

Signed-off-by: Samuel Just <rexludorum@gmail.com>

FileStore: CollectionIndex, HashIndex, IndexManager

Adds ColletionIndex, an interface for collection indexing
systems, and HashIndex, a mechanism for organising a prehashed
collection.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: find_object_context now takes an hobject_t

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd/: fix hobject_t construction

sobject_t requires only an object_t and a snapid_t.  hobject_t also
requires the hash which should be used for the object.  In most cases,
the osd must fill this in using the op message.  In cases where the hash
used does not matter (as in the metadata collection), the explicit
hobject_t(const sobject_t &) constructor supplies a hash.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

dupstore: s/sobject_t/hobject_t

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

src/messages/: s/sobject_t/hobject_t

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

src/osd/: s/sobject_t/hobject_t/g

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

src/os/: s/sobject_t/hobject_t/g

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

object.h: hobject_t definition

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ceph/src/test/store_test.cc: added store_test.cc

Meant to test basic file store functionality

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ObjectStore.h: const cleanup in transaction method params

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileStore.cc: Fix error in collection_rename

Previously, the rename operated on the collection names.  Now, rename
operations on the full paths to the collections.

Signed-off-by: Samuel Just <rexludorum@gmail.com>

JournalingObjectStore: journal_replay committed_seq fix

Previously, committed_seq was set to op_seq at the end of
replay_journal.  committed_seq should not actually be set until the
sync thread runs.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Integrate Doxygen into Sphinx docs.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

First draft of the documentation overhaul.

To build the docs, run ./admin/build-doc. To browse them, either get
them on any static website, or just run ./admin/serve-doc to serve
them quickly off of port 8080.

build-doc sets up a virtualenv to avoid needing Sphinx installed
system-wide. serve-doc needs thttpd installed.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

test: make escaping unitest happy

json escaping fix

radosgw_admin, rados tool: don't use camel casing for json output

fix json escaping

testlibrbdpp: test copy_with_progress

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librbd: add copy_with_progress API

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pg: warn about inconsistent object_info/on-disk size

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

osd_types: add standard header so spacing is correct

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

pg: remove useless line

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>