mon: limit size of each logm paxos event

Limit the number of log events we cram into a single paxos event.

Fixes: #2518
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: clear osd_stat on osd creation/destruction

Reported-by: Vladimir Bashkirtsev <vladimir@bashkirtsev.com>
Signed-off-by: Sage Weil <sage@inktank.com>

doc: Added S3 examples to the toctree.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: adding code samples for S3 API usage (thanks, DH!)

Signed-off-by: Ross Turk <ross@inktank.com>

Makefile.am: explicitly mention that -Wl,--as-needed is location-sensitive.

doc: Added ${lsb_release -sc} based on Sam's feedback.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

rgw: shutdown init_timer

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: try to create fcgi socket through open() first

FCGX_OpenSocket might just exit() without any warning if it fails
to create the socket.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

admin_socket: only init if path is defined

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote branch 'gh/chef-3'

upstart: simplify start; allow group stop via an abstract job

Use a 'ceph-mds' or 'ceph-mon' event to start instances instead of
explicitly calling start.  This avoids the ugly is-this-already-running
check.  [Thanks Guilhem Lettron for that!]

Make the -all job abstract (which means it stays started and can be
stopped).  Trigger a helper task (-all-starter) to trigger instance
start.  Make instances stop with the -all task.  This allows you to do

 start ceph-mds-all
 stop ceph-mds-all
 start ceph-mds id=foo
 start ceph-mds-all
 stop ceph-mds id=bar
 stop ceph-mds-all

but not

 start ceph-mds id=foo
 stop ceph-mds-all

because ceph-mds-all isn't running.  Not quite as flexible in sysvinit in
that regard, but good enough for me.

Fixes: #2414
Signed-off-by: Sage Weil <sage@inktank.com>

paxos: warn on extreme clock skew

This would have helped us diagnose #2480.

Signed-off-by: Sage Weil <sage@inktank.com>

omapbench: fix warning

Signed-off-by: Sage Weil <sage@inktank.com>

omapbench: fix misc warnings

Signed-off-by: Sage Weil <sage@inktank.com>

ReplicatedPG: fix pgls listing, add max listing size

Previously, a client requesting a large pgls could tie up the
osd for an unacceptable amount of time.  Also, it's possible
for the osd to return less than the requested number of
entries anyway, so we now return 1 when we have completed the
listing.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

objecter: fix pgls

First problem: if the osd returns more entries than we ask for, max_entries
was going negative, and we were requesting (u64)(-small number) on the
next iteration, slamming the OSD when the PG was big.  We fix that by
finishing response_size >= max_entries.

Second problem: AFAICS we were not requesting the second chunk on a large
PG at all, here, if the OSD returned less than what we wanted.  Fix this
by asking for more in that case.

That means we detect the end of a PG in two ways:

 * if the OSD sets the return value to 1 (instead of 0)
 * if we get 0 items in the response

Another patch will change the OSD behavior to return 1, and all will be
well.  If we run against an old OSD, we'll send an extra request for each
PG and get nothing back before we realize we've hit the end and move on.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Sam Just <sam.just@inktank.com>

Merge remote-tracking branch 'gh/wip-admin'

mon: fix slurp latest race

It is possible for the latest version to get out in front of the
last_committed version:

 a- start slurping
 a- slurp a bunch of states, through X
 a- get them back, write them out
 b- monitor commits many new states
 a- slurp latest, X+100 say, but only get some of those states due to the
    slurp per-message byte limit
 a- write latest + some (but not all) prior states
 a- call back into slurp(), update_from_paxos(), trigger assert

This fix ensures that we make note of the source's new latest, so that on
the next pass through slurp() we will grab any missing states.

We *also* explicitly require that we get everything up through what we have
stashed, in defense against some future kludging that might only require we
nearly (but not completely) in sync before finishing the slurp.

Fixes: #2379
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Makefile: include ceph-mds upstart bits in dist tarball

Signed-off-by: Sage Weil <sage@inktank.com>

test/: Added object map benchmarking tool

omap_bench writes configurable objectmaps to a configurable number
of objects and generates latency statistics.

Signed-off-by: Eleanor Cawthon <eleanor.cawthon@inktank.com>

doc: fix autobuild debian source line

Signed-off-by: Sage Weil <sage@inktank.com>

mon: throttle client msgr memory

Limit the amount of memory that can be consumed by client messages, similar
to the OSD.  Do not limit inter-mon messages.

Fixes: #2495
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Merge remote-tracking branch 'origin/wip-2491'

Objecter: tone down linger op messages on tick

Signed-off-by: Samuel Just <sam.just@inktank.com>

test_stress_watch: exercise watch/unwatch/close from second client

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

qa: stress_watch.sh workunit runs test_stress_watch

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/wip-mon-doc'

doc: Added preliminary rbd and ko info.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

osd: Adding const to methods in OSDMap

Signed-off-by: Adam Crume <adamcrume@gmail.com>
Signed-off-by: Sage Weil <sage@inktank.com>

vstart.sh: initialize keyring even if not using cephx

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge branch 'master' of ssh://github.com/ceph/ceph

perf_counters: use bufferlist instead of vector<char>

bufferlist allocates page-sized chunks and avoids realloc, which'll be
more efficient than vector resize doubling just about always.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: fix admin socket lock dependency

Signed-off-by: Sage Weil <sage@inktank.com>

admin_socket: fix unit test

Signed-off-by: Sage Weil <sage@inktank.com>

osd: ack pending notify when unwatching

If an unwatch crosses paths with a notify, we currently force the notifier
to keep waiting.  Instead, implicitly ack any notify when we unwatch.

Fixes: #2491
Signed-off-by: Sage Weil <sage@inktank.com>

admin_socket: pass args separately

This avoids making the callback parse off the command portion on their own.
It also lets them assert that the command portion is in the set of
registered commands.

Signed-off-by: Sage Weil <sage@inktank.com>

admin_socket: streamline 'version' commands

 version     -- ceph version
 git_version -- git sha1
 0           -- proto version (for backward compat)

Signed-off-by: Sage Weil <sage@inktank.com>

admin_socket: streamline command set

 log flush
 log dump
 log reopen
 config show
 config set <var> <val>
 perf dump
 perf schema

Signed-off-by: Sage Weil <sage@inktank.com>

admin_socket: register command prefixes

Allow any command prefix to be registered, provided it is separated by a
space.  Previously, we always matched against the first word.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote branch 'gh/wip-admin'

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

librbd: Simplify timing init

Remove possibility of set_start_time before set_ictx error

Signed-off-by: Dan Mick <dan.mick@inktank.com>

librbd: Add latency (elapsed-time) stats for rbd operations

Fixes: #2408
Signed-off-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

rgw: put_bucket_info does not override attrs

This fixes #2487. When writing bucket info we just
wrote the object content, and were overriding any
attrs that object contained (that is -- corrupted
the ACLs).

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: return default policy if policy attr is broken

In case the policy attr on object/bucket is broken
return a default policy, in which the owner is set
as the bucket owner.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

log: add missing .cc file

Signed-off-by: Sage Weil <sage@newdream.net>

log: uninline SubsystemMap mutators

This lets you do something like

 p g_ceph_context->_conf->subsys.set_log_level(ceph_subsys_mon, 20)

from gdb.

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: fix my broken ifdef flow

Signed-off-by: Sage Weil <sage@newdream.net>

doc: ceph.conf cleanup for cephx authentication.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: fix broken link.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added -a to service ceph stop.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Consolidated mkcephfs deployment files.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

osd: fix null dereference

The pg arg is optional!

Signed-off-by: Sage Weil <sage@inktank.com>

doc: Added pools and authentication.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

mon: fix reweight_by_utilization waiting

Propose if we changed something, but not if we don't.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix rewewight_by_utilization

Update the incremental, not the in-memory OSDMap!

Fixes: #2454
Signed-off-by: Sage Weil <sage@newdream.net>

filestore: log about syncfs(2) etc support

Fixes: #2479
Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/stable'

doc: Updated authentication with header canonicalization.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge remote-tracking branch 'gh/wip-2349'

Reviewed-by: Sage Weil <sage@newdream.net>

workloadgen: Fix the statistics output and a bug triggered by coll destruction

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

filestore: initialize dummy xattr buffer

Shut up valgrind.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix misdirected op check

The sense of this check was wrong from the get-go, back in a5d99add.  Fix
it.  This may be silencing a lot of misdirected op noise!

Signed-off-by: Sage Weil <sage@inktank.com>

doc: Add Paxos call chain to doc/mon/

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: add some info on can_mark_out function

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Do some aesthetic reformatting on the Elector class.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: add doxygen-format comments to the Elector class.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Do some aesthetic reformatting on the Paxos class.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: add doxygen-format comments to the Monitor class.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: add doxygen-format comments to the PaxosService class.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: add doxygen-format comments to the Paxos class.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

test_filestore_workloadgen: less noise

Signed-off-by: Sage Weil <sage@newdream.net>

test_filestore_workloadgen: write zeros, not random bytes

The random byte generation is sufficiently slow that it skews the results.
Zeros are good enough!

Signed-off-by: Sage Weil <sage@newdream.net>

admin_socket: fix unittest

Signed-off-by: Sage Weil <sage@inktank.com>

log: add log flush, dump_recent, reopen admin socket commands

Signed-off-by: Sage Weil <sage@inktank.com>

config: add 'set_config <var> <val>' admin socket command

Signed-off-by: Sage Weil <sage@inktank.com>

rados.cc: added listomap and getomap commands

Signed-off-by: Eleanor Cawthon <eleanor.cawthon@inktank.com>

config: add 'show_config' admin socket command

Signed-off-by: Sage Weil <sage@inktank.com>

admin_socket: initialize explicitly on startup; disallow changes

There is an annoying dependency between the config lock and the admin
socket lock due to the fact that we initialize (or reinitialize) the socket
via a config observer.

Instead, explicitly initialize on startup.  Do not allow the admin socket
location to be changed at runtime.

Signed-off-by: Sage Weil <sage@inktank.com>

librados.h: update rados_object_list_next documentation for key/locator

*entry and *key don't need to be freed, and are only valid until the
next call to rados_objects_list_*.

Signed-off-by: Samuel Just <sam.just@inktank.com>

cephcontext: generalize admin socket hook

We'll process non-perfcounter commands too.

Signed-off-by: Sage Weil <sage@inktank.com>

doc: needed carriage returns between headings, code tag and syntax.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: removed redundant install.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Filled in descriptions for object operations.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

ceph.spec.in: buildrequires libuuid-devel

Reported-by: Guido Winkelmann <guido-ceph@thisisnotatest.de>
Signed-off-by: Sage Weil <sage@inktank.com>

ceph.spec.in: drop unnecessary rmdirs

This hunk was inadvertantly included in a330c64.

Reported-by: Guido Winkelmann <guido-ceph@thisisnotatest.de>
Signed-off-by: Sage Weil <sage@inktank.com>

doc: Restored install for dev/unstable packages.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

filestore: make SNAP_DESTROY ioctl check more informative

Tell user about user_subvol_rm_allowed mount option.

Signed-off-by: Sage Weil <sage@inktank.com>

filestore: fix btrfs subvol ioctl tests

Create a temp subvol that *we own* to test snaps against.  Otherwise, we
won't be able to remove them if we weren't the owner of the source subvol
we were snapshotting when we are not root.

Signed-off-by: Sage Weil <sage@inktank.com>

filestore: verify that op_seq is nonzero

If we get zero from commit_op_seq something is very wrong, and we should
not continue.  Broken mkfs or something deleted commit_op_seq file.

Signed-off-by: Sage Weil <sage@inktank.com>

vstart.sh: wipe out btrfs subvols in osd dirs

For testing on btrfs.

Signed-off-by: Sage Weil <sage@inktank.com>

filestore: fix initial btrfs snap creation (on mkfs)

Create the snap_1 snap of current *after* we write the op_seq.  Otherwise
we will mistakingly believe the snapshot is of op seq 0.

Signed-off-by: Sage Weil <sage@inktank.com>

rgw: replace dump_format() with dump_int()

Cleanup, fixes issue #2469.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge branch 'stable'

librados, ObjectCacher: include limits.h for INT_MAX

This was required to build on CentOS 6.2. I'm not sure why it worked
without it on Debian.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

rbd.py: Fix seemingly-random segfaults

Avoid resetting Image.closed until image is successfully opened.

Fixes: #2475
Signed-off-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

librados: Remove errant sha1

Signed-off-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

rbd.py doc: Fix markup on :class: references

Signed-off-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge branch 'osd-mkfs'

Reviewed-by: Sam Just <sam.just@inktank.com>

Merge remote-tracking branch 'gh/wip-mon-auth'

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

Merge branch 'stable'