Merge pull request #5 from homac/master

Minor fix for init files and cleaned up spec file. Please pull

man: clarify mount.ceph auth options

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

man: update rule definition for ceph-rbdnamer

This is the rule we install since 891025e539a92b5d75011e2e75c475fc0c272042.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

authx -> cephx everywhere it's used

The term authx was in the mount.ceph man page, and got accidentally
copied into rbd help.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ceph.spec: Clean up and fix spec file and build for a couple of distributions

Clean up and fix the spec file. This includes cleaning up of build and
installed system dependencies, LSB compliance fixes, splitting up into
several sub-packages (lib*) and so on. It now builds fine for the
following distributions in the Open Build Service and should be
considered as a starting point for further fixes:

 - CentOS 6
 - Fedora 15
 - RedHat Enterprise Linux 6
 - openSUSE 11.4
 - openSUSE 12.1
 - openSUSE Factory
 - SUSE Linux Enterprise 11 (SP1 and SP2)

Signed-off-by: Holger Macht <hmacht@suse.de>

init-ceph/init-radosgw: Don't use unspecified runlevel 4

Don't use runlevel 4 in init scripts. AFAIK, no distribution is using it
and at least the Open Build Service complains about it.

Signed-off-by: Holger Macht <hmacht@suse.de>

client: fix logger deregistration

Only unregister logger if it is non-NULL (and thus registered) to avoid
running afoul of the cct assertions.

Signed-off-by: Sage Weil <sage@newdream.net>

doc: Make overview.rst valid reStructuredText, so I can stop seeing warnings.

It's still wrong, but now it won't clutter the output.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

pybind: Add a description to docstring.

This avoids a Sphinx warning like this:

.../src/pybind/rbd.py:docstring of rbd.RBD.version:2: WARNING: Field list ends without a blank line; unexpected unindent.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: reStructuredText syntax fix.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Merge branch 'wip-messenger'

doc: "ceph -w" clarification.

Stop saying "watch cluster state" so many times.

Don't say stdout, that's the assumption.

Don't call showing things on terminal "printing".

Don't show -C option here.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Fix sentence that ended too abruptly.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Split intro to talk about the DFS separately. Mention petabytes.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Add more missing commands to control.

This is too unstructured, that will have to be fixed later.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Add misc explanations of Ceph internals from email.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Use todo directive, rescue list of missing commands from wiki.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Link to manpage when command is mentioned.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: More consistency on formatting placeholder names.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

msgr: unset did_bind in stop().

We use did_bind as a flag on whether or not to stop the Accepter thread
and we should clear it when we do the stopping.
This fixes #1829.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

objecter: fix use-after-free

messenger consumes the m reference.  Yay valgrind.

Signed-off-by: Sage Weil <sage@newdream.net>

client: move PerfCounter into Client

globals are evil.

Fixes: #1826
Signed-off-by: Sage Weil <sage@newdream.net>

swift: auth response returns X-Auth-Token instead of X-Storage-Token

osd: fix build_incremental_map_msg

We keep both the inc and the full for our oldest osdmap.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: trivial_libceph test

This currently fails... see #1827

Signed-off-by: Sage Weil <sage@newdream.net>

libceph: add missing #includes

Signed-off-by: Sage Weil <sage@newdream.net>

libceph: catch errors from Client::init()

And clean up error paths.

Signed-off-by: Sage Weil <sage@newdream.net>

client: return errors from init

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: dump transaction to log if we hit an error

This will let us see which operation in the transaction failed.

Signed-off-by: Sage Weil <sage@newdream.net>

objectstore: implement Transaction::dump()

Signed-off-by: Sage Weil <sage@newdream.net>

objectstore: create Transaction::iterator class

Remove iterator state from Transaction itself.

Signed-off-by: Sage Weil <sage@newdream.net>

qa: rados load-gen workunits

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/rados: add asynchronous write,append,read,write_full operations

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

doc: Clarify documentation of reweight command.

This caused some discussions on the mailing list, so let's try to be clear
about the meaning of an OSD weight.

Signed-Off-By: Andre Noll <maan@systemlinux.org>

doc: Add missing documentation for osd pool get.

"osd pool set" was already documented, but the corresponding "get"
command was not. This patch adds the list of valid fields for this
command, together with short descriptions.

Thanks to Gregory Farnum for pointing out a flaw in the description
of lpl_num and lpgp_num in an earlier version of the patch.

Signed-Off-By: Andre Noll <maan@systemlinux.org>

doc: Update the list of fields for the pool set command.

This list was lacking a few fields: crash_replay_interval, pg_num,
pgp_num and crush_ruleset. Include these fields and add add short
descriptions.

Signed-Off-By: Andre Noll <maan@systemlinux.org>

doc: Document pause and unpause osd commands.

These two commands were undocumented so far. This patch adds a short
description.

Signed-Off-By: Andre Noll <maan@systemlinux.org>

doc: Add documentation of missing osd commands.

The set of OSD commands which added by the previous commit is
incomplete. This patch adds documentation for the following
OSD commands which were previously missing: dump, tree, crush,
cluster_snap, lost, create, rm.

Signed-Off-By: Andre Noll <maan@systemlinux.org>

doc: Import the list of ceph subcommands from wiki.

This adds the content of the wiki page at

http://ceph.newdream.net/wiki/Monitor_commands

to doc/control.rst in order to make it available at the new official
location for the ceph documentation. This first patch is just the
result of a cut-and-paste operation. There are no changes in content,
but the text was converted to rst format.

Signed-Off-By: Andre Noll <maan@systemlinux.org>

messenger: fix up fault()'s "onconnect" parameter.

We should be setting this true when calling fault() from connect().
And rename it in the header -- it does produce less output when set,
but it's actually useful because it changes the shutdown logic
slightly.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: mark_disposable when closing a Client connection.

This is causing issues since the Client's ack of the MClientSession
is somehow not getting back to the MDS. We should make the
Client/SimpleMessenger be better-behaved about shutdown, but for
now this solution matches how we're working elsewhere in our codebase.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

SimpleMessenger: remove void send_keepalive.

Nobody uses this; they all call the version that returns an int.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge branch 'wip-mon-stats'

mon: add mds, mon info to cluster_logger

Signed-off-by: Sage Weil <sage@newdream.net>

mdsmap: rename get_num_*_mds() methods

Signed-off-by: Sage Weil <sage@newdream.net>

mon: report basic cluster stats via perfcounters

These are basic point-in-time cluster stats.

Signed-off-by: Sage Weil <sage@newdream.net>

crush: drop unused label

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'gh/stable'

crush: fix force to behave with non-root TAKE

If the (first) TAKE in the crush rule is not the root, see if they picked
a point somewhere beneath the appropriate part of the hierachy.  If not,
continue with a normal mapping.

Previously we would simply error out in this case and map nothing.

Signed-off-by: Sage Weil <sage@newdream.net>

crush: simplify force argument check

force isn't used past this point, only force_pos.  Collapse the if
conditions.

Signed-off-by: Sage Weil <sage@newdream.net>

workunits: print tests pjd runs

This will tell us which ones actually failed within a test suite.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge branch 'wip_pgls'

pybind: add object locator support to pybind pool listing

list_objects returns Object().  Object therefore now has an optional
locator_key parameter which will set up the object locator on Object()
methods to allow objects returned from list_objects with locator keys to
be used normally.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rados.cc: add --object-locator and object locator output to ls

--object-locator locator causes io to use the specified locator.  For
objects with non-empty locators, rados pool ls will output the locator
as well.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: collection_list_handle_t is now an hobject_t

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd: delay pg list on a snapid until missing is empty

We cannot determine from the missing set whether an object existed
at a given snap.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

msgr: turn up socket debug printouts

These shouldn't be too common and will help in debugging
socket leaks.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

udev: drop device number from name

The device number depends on how many rbd images have been
mapped. Removing it makes the name determined solely by the name,
image, and snapshot that are mapped, for ease of scripting or persistence
across reboots.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

pybind: trivial fix of missing argument

Signed-off-by: Henry C Chang <henry.cy.chang@gmail.com>

crush: whitespace

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: initialize cluster_snapshot_epoch

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crush: set max_devices=0 for map with empty buckets

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crush: fix stepping on unallocated memory

If size is 0 we can't write here.

Reported-by: pankaj singh <psingh.ait@gmail.com>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: trim lru after flushing dirty data

Shouldn't matter, but it would be interesting to see if this affects
#1737.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: unmount cleanup

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: wait for sync writes even with cache enabled

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: send umount warnings to log, not stderr

stderr isn't usually open anyway.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ObjectStore,ReplicatedPG: remove old collection_list_partial

No need for the old collection_list_partial instance: it's cleaner to
just use an hobject_t as the collection list handle.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

librados,Objecter,PG: list objects now includes the locator key

Previously, there was no way to recover the locator key used to create
and object.  Now, rados_objects_list_next and ObjectIterator will return
the key as well as the object name.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

hobject_t: encode max properly

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

hobject_t: make filestore_hobject_key_t 64 bits

So we can return 0x100000000 when max=true.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

os/HashIndex: some minimal debug output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ObjectStore: Add collection_list_partial for hash order

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

hobject_t: define explicit hash, operator<<; drop implicit sobject_t()

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

object.h: Sort hobject_t by nibble reversed hash

To match the HashIndex ordering, we need to sort hobject_t by the nibble
reversed hash.  We store objects in the filestore in a directory tree
with the least significant nibble at the top and the most at the bottom
to facilitate pg splitting in the future.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

hobject_t: sort by (max, hash, oid, snap)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

hobject_t: define max value

Create a max value that is greater than all other values.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

gitignore: Ignore src/keyring, as created by vstart.sh

Commit 86c34ba9ee8c883b71a8449c3c261154365c35ae changed
the filename but not .gitignore.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ReplicatedPG: don't crash on empty data_subset in sub_op_push

If data_subset is empty (i.e., the data we pulled is no longer useful),
we should mark complete false and continue rather than fail the
assert in range_end().

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

workunits: fix secret file and temp file removal for kernel rbd

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ReplicatedPG: do not ->put() scrub messages when adding to a WorkQueue.

This function is passing a reference from PG::active_rep_scrub to
the req_scrub_wq, not eliminating the reference (and the WorkQueue
doesn't grab a new reference itself, either).
The other alternative is to convert the WorkQueue to grab a
reference, but since they can cycle through the WorkQueue more than
once, and need to be ->put() outside the WorkQueue, I don't like
that option.
This should fix #1758.

Also add an assert to PG::_request_scrub_map to check on the other
possible cause of this bug (and fix the indentation).

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

workunits: make rbd kernel workunit executable

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

doc: Reorganize pip calls to use a requirements file.

The conditional before running pip install was unnecessary,
"pip install" on already installed packages is fast (as long
as it's not --upgrade), and --quiet makes it not spam the
console.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Switch diagram tools from dia to ditaa.

Now you can create diagrams easily with the ".. ditaa::"
directive in the Sphinx documents.

admin/build-doc now checks for debs required for building
the documentation, or just lists commands missing for hosts
not using dpkg.

For more on Ditaa, see http://ditaa.sourceforge.net/

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

filestore: send back op error to log, not stderr

Signed-off-by: Sage Weil <sage@newdream.net>

doc: fix typo

Signed-off-by: Sage Weil <sage@newdream.net>

workunits: add some tests for kernel rbd

This covers some snapshot and resize functions that aren't tested by fs benchmarks.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: the showmapped command shouldn't connect to the cluster

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ceph-rbdnamer: include snapshot name if present

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd, mount.ceph: use pre-stored secret if available

If a secret is specified, store and use it, but otherwise
check for a pre-existing secret to use.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

secret: add is_kernel_secret function

This will let us know whether we can add a key mount option
if no secret is specified.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

secret: fix error check

add_key will return -1 when an error occurs, which should be handled at a higher level and not printed here.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: allow snapshots to be mapped

unmap and showmapped already support snapshots. map should too.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

test_rados.py: clean up after EEXIST test

This extra pool caused subsequent pool tests to fail.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge remote branch 'gh/stable'

doc: fix rst syntax

Signed-off-by: Sage Weil <sage@newdream.net>

doc: document monitor cluster expansion/contraction

Pretty sure my rst syntax is wrong.

Signed-off-by: Sage Weil <sage@newdream.net>

cephtool: fix shutdown

Fix 'ceph -w' brokenness from commit ad13d0b7.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filejournal: make FileJournal::open() arg slightly less weird

Pass in fs_op_seq (last_committed_seq), not the next expected seq, so we
can avoid subtracting and adding 1 in odd places.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'stable'