os/hashindex: fix #include

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip_colindex'

osd/,os/,osdc/: Convert collection_list_handle_t to a struct

Previously, we stored the collection_list handle as an opaque
uint64_t.  Now, collection_list_handle_t is a struct defined
in types.h.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

fsconverter.cc: Added fsconverter

fsconverter updates an existing FileStore to the most recent FileStore
format

Signed-off-by: Samuel Just <rexludorum@gmail.com>

FileStore: On mount, scan collections for unstable state

CollectionIndex implementations may perform compound operations
leaving invalid state if interrupted.  index->cleanup() gives
the implementation an oportunity to cleanup any in-progress
operation.  For HashIndex, split and merge fall in this
category

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileStore: Added update_version_stamp

update_version_stamp may be used to refresh the filestore version
stamp when all collections have been brought to the current
version.

Signed-off-by: Samuel Just <rexludorum@gmail.com>

FileStore: Add filestore version stamp

A filestore will now be tagged with a version stamp during
mkfs.  If on mount the version stamp detected lags the current
version, the mount will fail unless filestore_update_collections
is set in gconf.  If it is set, opening a collection will cause
the version stamp on the collection to be read and the
appropriate indexing implmentation to be used.  This will allow
for conversion from old collection indexing schemes to new
ones.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd_types.h: Fix support for old object_info_t

The object locator in the old encoding is necessary
to get the hash for soid.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileStore: Integrates HashIndex/IndexManager into FileStore

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

os/FlatIndex: FlatIndex encapsulates previous FileStore behavior

FlatIndex will be used to read and convert the previous on disk
format.

Signed-off-by: Samuel Just <rexludorum@gmail.com>

FileStore: CollectionIndex, HashIndex, IndexManager

Adds ColletionIndex, an interface for collection indexing
systems, and HashIndex, a mechanism for organising a prehashed
collection.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: find_object_context now takes an hobject_t

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd/: fix hobject_t construction

sobject_t requires only an object_t and a snapid_t.  hobject_t also
requires the hash which should be used for the object.  In most cases,
the osd must fill this in using the op message.  In cases where the hash
used does not matter (as in the metadata collection), the explicit
hobject_t(const sobject_t &) constructor supplies a hash.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

dupstore: s/sobject_t/hobject_t

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

src/messages/: s/sobject_t/hobject_t

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

src/osd/: s/sobject_t/hobject_t/g

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

src/os/: s/sobject_t/hobject_t/g

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

object.h: hobject_t definition

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ceph/src/test/store_test.cc: added store_test.cc

Meant to test basic file store functionality

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ObjectStore.h: const cleanup in transaction method params

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileStore.cc: Fix error in collection_rename

Previously, the rename operated on the collection names.  Now, rename
operations on the full paths to the collections.

Signed-off-by: Samuel Just <rexludorum@gmail.com>

JournalingObjectStore: journal_replay committed_seq fix

Previously, committed_seq was set to op_seq at the end of
replay_journal.  committed_seq should not actually be set until the
sync thread runs.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Integrate Doxygen into Sphinx docs.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

First draft of the documentation overhaul.

To build the docs, run ./admin/build-doc. To browse them, either get
them on any static website, or just run ./admin/serve-doc to serve
them quickly off of port 8080.

build-doc sets up a virtualenv to avoid needing Sphinx installed
system-wide. serve-doc needs thttpd installed.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

test: make escaping unitest happy

json escaping fix

radosgw_admin, rados tool: don't use camel casing for json output

fix json escaping

client: debug link/unlink and dn_set parent relationship

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

heartbeatmap: fix reset_timeout with mixed-used threads

If you have a ThreadPool used by multiple WorkQueues, and some of them are
setting a suicide timeout, we need to clear it when a suicide timeout is
not set.

Signed-off-by: Sage Weil <sage@newdream.net>

rados tool: df dumps formatted data

rgw: log show dumps some information about bucket owner

bandaid for gitbuider

Merge remote branch 'vapre/wip-pgid'

osd: remove dead lookup_lock_pg()

Signed-off-by: Sage Weil <sage@newdream.net>

osd: set suicide timeouts on some workqueues

OpWQ: timeout * 10
RecoveryWQ: this does no io; it if stalls we're probably stuck in an
  infinite loop.  timeout * 10.
ScrubFinalizeWQ: this is cpu only. we're probably stuck in a loop, or
  swapping.  timeout * 10.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: health not ok when up < in osds

We were warning if there were any not up or in osds.  Instead, warn if
there are any osds that are in but not up.  That means if a node fails
and successfully marks the node out and retracts onto remaining nodes, the
ceph cluster is healthy again.

Presumably the fact that the nodes failed should raise other alerts,
because those specific daemons/nodes are not healthy.

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: add suicide grace to workqueue

The main thing we are worried about here are kernel BUGs that kill off our
threads.

Signed-off-by: Sage Weil <sage@newdream.net>

heartbeatmap: add suicide grace

Add a second suicide grace period that will make us kill ourselves if
we are sufficiently catatonic.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: don't check for time skew when Expires was set

client: simplify Client::remove_cap() args

Signed-off-by: Sage Weil <sage@newdream.net>

client: clean up Client::remove_session_caps() args

Signed-off-by: Sage Weil <sage@newdream.net>

client: don't falsely populate mds_sessions on bad MClientSession msg

If from isn't in the map, we don't want to add it via operator[].

Fixes: #1367
Signed-off-by: Sage Weil <sage@newdream.net>

testlibrbd: remove useless sizeofs

Should be sizeof(*result), if anything.

Signed-off-by: Sage Weil <sage@newdream.net>

testlibrbd: fix assert side effect

Coverity cid 1, 2

Signed-off-by: Sage Weil <sage@newdream.net>

FileJournal: Also classify kernel major version 3 as safe with disk-cache on

Signed-off-by: Wido den Hollander <wido@widodh.nl>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: decode pg_t in old PG::Info

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: requrie PGID64 of mons, other osds

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: require PGID64 of mons

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: require PGID64 of osds and other mons

Signed-off-by: Sage Weil <sage@newdream.net>

pgmap: decode old pg_t

Signed-off-by: Sage Weil <sage@newdream.net>

mdsmap: int64_t pool ids

Encode old-format maps for clients without PGID64 feature bit.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: add 64-bit pools to ObjectLocator

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: encode/decode old + new version

In MOSDMap, reencode map payloads using old format if the target doesn't
have the PGID64 feature bit.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: decode old pg_stat_t

Signed-off-by: Sage Weil <sage@newdream.net>

osd: make MOSDOp[Reply] encoding backwards compatible

Move away from struct-based encoding of MOSDOpReply while we're at it.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: add PGID64 feature bit

Signed-off-by: Sage Weil <sage@newdream.net>

osd: int pool -> int64_t pool

Internal interfaces.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: redo pg_t encoding with 64-bit pool id

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix pg_pool_t::get_pg_num return types

Unsigned!

Signed-off-by: Sage Weil <sage@newdream.net>

ceph_context: whitespace

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge commit 'v0.34'

v0.34

rgw: use formatter->dump_string() instead of dump_format()

rgw: get bucket info only if bucket name is not empty

Makefile.am: install coverage files for libtool outputs as well

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados: watch request should wait for safe response

objectcacher: only want for commit

There was some old, weird stuff going on here where we would wait for the
ACK and COMMIT separately.  This is just wrong.  Writeback does not
complete until the data is committed on disk.

Simplify by waiting only for commit, removing all the 'ack' code, and
going back to a single callback (flush_set).

I didn't notice this for 05063867e2a54176ffc9bbc73391f52766ab403f; both of
these cleanups are needed to fix this.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix shadow

Coverity cid 29

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

fix utime.h #include recursion

Coverity cid 11, 13

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crush: comment fall-thru

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: remove unused Mutation in do_rmdir_rollback()

Signed-off-by: Sage Weil <sage@newdream.net>

workunits: make file_layout test setting only one parameter

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cephfs: use strtol instead of atoi; handle 0 properly

Besides being generally better, this means we can accept pool 0
as the pool to store stuff in.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cephfs: fix the stripe_count documentation.

Whoops!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: server: should apply new layout settings on top of old layout

This way, the MDS can handle updates of some values without needing
the user to specify the entire layout (ie, they can just switch pools).
This brings the behavior more in line with setting the dir layout.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: allow setting all options on a file layout.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

objectcacher: remove uncommitted xlist<>; fix broken purge() cleanup

There was a problem where:

 - we would dirty some buffers on an object
   - bump dirty_tx count
 - flush()
   - this adds the Object to ObjectSet::uncommitted
 - truncate
   - client clears FILE_BUFFER cap_ref
   - Object::purge()
     - clear dirty_tx count
 - client puts last inode
   - Object::uncommitted is not empty in ~ObjectSet

(This was triggered after several runs of workunits/suites/blogbensh.sh
on sepia.)

It turns out the uncommitted xlist<> is pretty useless, though: the same
information is captured in the dirty_tx counter.  We add a separate
counter to the Object itself (for the benefit of Object::can_close()).

We also clean up Object::purge() to call truncate(0), a small
simplification.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: remove some compilation warnings

rgw: don't override ret value in do_read_permisions

rgw: return correct bucket missing error when needed

rgw: fix 500 error on bucket creation

rgw: fix clitest

MonClient: fix possible null dereference

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds/Server.c: fix memory leak in err path

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: don't check for zero content when creating a bucket

rgw: rgw_admin bucket stats dumps owner

client: drop useless insert_trace timestamp argument

This is reachable via request.  It was also misnamed in the header :/.

Signed-off-by: Sage Weil <sage@newdream.net>

client: be careful about replacing dentries during readdir assimilation

When we are assimilate readdir results into our cache, we need to be more
careful about replacing existing dentries.  We were calling
insert_dentry_inode(), which would replace a name if it already exists,
which might include pd->first, an active iterator.

Move the dentry link/relink into the caller (where we already have an
iterator pointing to the existing item, if any).  Then update the dentry
lease information separately.

Fixes: #1391
Signed-off-by: Sage Weil <sage@newdream.net>

rgw: add radosgw_admin bucket stats

client: factor update_dentry_lease() out of insert_dentry_inode()

The first bit of insert_dentry_inode() handles the details of checking
whether an item still exists, un/relinking it, etc.

The second bit just updates the dentry lease information.

Signed-off-by: Sage Weil <sage@newdream.net>

librados: fix memory leak in error path

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/confutils.cc: fix resource leak

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

client: carry inode reference across calls to _flush

Also drop the unused _flush Context* argument.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: fix prepare_get_obj when ENOENT

rbd: fix shared_ptr usage

shared_ptr calls the disposal function even when the pointer is null
that is being disposed of.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

do_kernel_showmapped: fix resource leak

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge remote branch 'origin/next'

client: fix readdir + cap recovation/reissue race

The current race:
 - we start readdir
 - mds revokes dir cap
 - some file gets removed
 - mds reissues dir cap
 - we finish readdir and set I_COMPLETE

We should only set it if there have been no FILE_SHARED reissues during
the readdir.

Note that we still set I_COMPLETE even if we don't have the cap; that's
useless but harmless, since it is undefined without FILE_SHARED being
set.

Signed-off-by: Sage Weil <sage@newdream.net>

rados_sync: fix resource leak on err

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>