OpRequest: mark_event output is now at debug 5

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileJournal: allow write_thread and write_finisher to run in parallel

Previously, the write_thread and the write_finisher serialized on
write_lock limiting performance.  aio_lock has been introduced to
protect data moving from the write_thread to the write_finisher.
Similarly, submit_entry (called while holding journal_lock in
the filestore) also tended to hit contention on the write lock.
The queues and related structures are now protected by queue_lock.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

OSD: improve information and format of OSDTracker messages

Also, Message now has a timestamp indicating when the message
was fully recieved for use by OSDTracker.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge remote branch 'gh/wip-defaults'

osdmaptool: fix clitest for lpg_num 0

Broke this in 11b93d3af2c0f1bca3e48fe773a14f07f479ccc3

Signed-off-by: Sage Weil <sage@newdream.net>

FileStore: don't return ENOENT from object_map getters

ENOENT in those contexts means that the object_map does not know
about the object, not that the object does not exist.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

DBObjectMap: check in clone and link that source is not destination

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'wip-2223'

Reviewed-by: Sage Weil <sage@newdream.net>

filestore: use config observer to start/stop dumping/tracing

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objectstore: tweak dump() a bit

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: dump transactions sets, include seq and trans_num

Lists of transactions commit as a unit; reflect that in the dump.  Also
include seq and trans_num.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: remove commented code, fix whitespace

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ObjectStore: Remove code duplication when dumping transactions.

By using OStreamFormatter, we can have a single function responsible for
dumping a transaction. We keep the same old functions for outputting
directly to a Formatter and to an ostream, but these are only wrappers
for the function that will really handle the dumping.

The "real" dump() function will now take only a Formatter as an argument,
to which we will output. We keep the 'dump(std::ostream& out)' function,
although now it simply creates an OStreamFormatter and passes it to the
dump() function.

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>

filestore: dump transaction in json on error

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filejournal: dump injson

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mkcephfs: update man page

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-authtool: update man page

Signed-off-by: Sage Weil <sage@newdream.net>

mkcephfs: note that btrfs (and --mkbtrfs) are optional and experimental

And that --mkbtrfs will be deprecated soon.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-authtool: add warning to man page

- data is not encrypted over the wire
- intended for trusted environments

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: disable localized pgs by default

Signed-off-by: Sage Weil <sage@newdream.net>

mon: alloc pgp_num adjustment up and down

Signed-off-by: Sage Weil <sage@newdream.net>

debian: python-support -> dh_python2

I followed the instructions on

http://wiki.debian.org/Python/TransitionToDHPython2

Signed-off-by: Sage Weil <sage@newdream.net>

COPYING: doc/ CC BY-SA

Signed-off-by: Sage Weil <sage@newdream.net>

README: update

- refer to COPYING, SubmittingPatches
- a word about dependencies
- building packages
- drop the list of built binaries... it's incomplete and not very
  interesting.

Signed-off-by: Sage Weil <sage@newdream.net>

debian: drop unnecessary conflicts on librgw

Cut and paste baggage from libcephfs, it looks like.

Signed-off-by: Sage Weil <sage@newdream.net>

init-radosgw: start in runlevel 4

Fixes lintian error

W: radosgw: init.d-script-missing-start etc/init.d/radosgw 4

Signed-off-by: Sage Weil <sage@newdream.net>

obsync: include man page in tarball, packages

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-rbdnamer: include in dist tarball and debs/rpms

Signed-off-by: Sage Weil <sage@newdream.net>

init-ceph: start at all runlevels

This fixes lintian error:

W: ceph: init.d-script-missing-start etc/init.d/ceph 4

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'stable'

FileStore: dumping transactions to a file

Dump each queued transaction to a predefined file, specified with
--filestore-dump-file, in JSON format.

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>

osd: fix heartbeat set_port()

set_port() fails an assert if it isn't an in4 or in6 address, which a
default entity_addr_t is not.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: reenable clone on recovery

This hasn't turned up problems in QA.

Fixes: #2002
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: allow users to specify the osd heartbeat server address.

Reported-by: Nick Bartos <nick@pistoncloud.com>
Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
Reviewed-by: Sage Weil <sage.weil@dreamhost.com>

osd_types: fix off by one error in is_temp

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Reviewed-by: Gregory Farnum <gregory.farnum@dreamhost.com>

rgw: list multipart response fix

LastModified was formatted outside of the Part block.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Makefile.am: remove some clutter

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

v0.45

Merge branch 'next'

configure: --with-system-leveldb

Default to bundled leveldb.  Optionally check.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: fix leveldb includes

Signed-off-by: Sage Weil <sage@newdream.net>

cephfs: fix uninit var warning

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: add missing .h to tarball

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: fix object name with slashes when vhost style bucket used

Fixes issue #2259. The problem was that we were initializing the
object name, then in the case of a virtual host bucket we were
setting it as the first string before the slash instead of setting
it to the entire string.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

OSD: use per-pg temp collections, bug #2255

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

encoding: fix iterator use for struct_len copy_in

The end() iterator position does not record an offset when the list is
modified.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

buffer: allow advance() to move an iterator backward

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'gh/stable' into next

configure: HAVE_FALLOCATE -> CEPH_HAVE_FALLOCATE

/usr/include/linux/fs.h defines this on CentOS 5, even though it does not
in fact compile.  This stupid workaround avoids the problem.

Reported-by: Nick Couchman <Nick.Couchman@seakr.com>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

init-ceph: manage pid_file from init script

With upstart the daemon shouldn't manage the pid file itself.  Move this
out of the default config and into the legacy sysvinit script.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

config: move /var/run and /var/log defaults to config_opts.h

This flips the sense of the common_init defaults.  Before, the alternate
defaults were filled in if it was a daemon.  Now, they're cleared out if
we are not a daemon.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

config: {osd,mon}_data default to /var/lib/ceph/$type/$cluster-$id

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'stable'

dencoder, rgw: make ceph-dencoder load much faster

by avoiding linking with unneeded shared objects.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

encoding: use iterator to copy_in encoded length

This gives us a pointer to the position into the list where the final
length value will be copied.  Previously we used bl.copy_in(), which takes
a byte offset and needs iterator over the bufferlist to seek to the
correct position, resulting in O(n^2) encoding time for large structures.

Fixes: #2161
Reported-by: Jim Schutt <jaschut@sandia.gov>
Diagnosed-by: Ake van der Meer <petrabbit@xs4all.nl>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

v0.44.2

FileStore: do not check dbobjectmap without option set

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

config: include /etc/ceph/$cluster.keyring in keyring search path

mkcephfs and the docs etc still write to /etc/ceph/keyring.

Signed-off-by: Sage Weil <sage@newdream.net>

config: expand metavariables for --show-config, --show-config-value

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-cluster'

Reviewed-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

config: parse fsid uuid in config, not ceph_mon

Use the new OPT_UUID type.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

config: add distinct UUID type

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

global: add -C or --cluster early args to specify cluster name

This will let you specify which cluster to talk to on the command line
(e.g., 'ceph -C foo ...' or when starting a daemon).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

config: add cluster name as metavariable; use for config locations

Add a cluster name (default "ceph") to the config structure, and expand
$cluster in all config values.

Make the default config be /etc/ceph/$cluster.conf, etc.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

config: implement --show-config and --show-config-value <option>

Dump internal config value(s) to stdout and then exit.

Signed-off-by: Sage Weil <sage@newdream.net>

test_workload_gen: fix logging

Signed-off-by: Sage Weil <sage@newdream.net>

config: remove obsolete bdev_* options

These were part of ebofs.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/wip-log'

README: update instructions

Needed to add submodule instructions.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rados: fix rados import

This fixes issue #2253. Wrong param order to fread().

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

config: drop loud ERROR prefix

This makes gitbuilder sad.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: allow row, room, datacenter, pool in conf for initial crush map

These work just like host and rack, except that they are optional.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crush: don't warn on skipped types

It's perfectly okay to skip some.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: set 'default' pool type correctly

Got this wrong in e85961167eb1f37f80f263257799e4e901d17e74

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-fuse: fix log reopen when -f is specified

Don't restart if it wasn't stopped.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: define more crush types

We don't use these by default, but this way they are there should someone
want to use them.

Signed-off-by: Sage Weil <sage@newdream.net>

test_rewrite_latency: check return value

Fixes warning

warning: test/test_rewrite_latency.cc:27:36: ignoring return value of ‘ssize_t pwrite(int, const void*, size_t, __off64_t)’, declared with attribute warn_unused_result [-Wunused-result]

Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: add mssing header

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: throttle at num_threads * 2

If we throttle at num_threads, then nothing gets into the workqueue until
a worker thread is idle, which means you pay the latency of setting it up
and queueing it.  This way we keep some requests ready to go.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/msgr-api-changes'

Reviewed-by: Sage Weil <sage@newdream.net>

filestore: print Sequencer name in debug output

And clean it up just a bit.

Signed-off-by: Sage Weil <sage@newdream.net>

msgr: clean up Pipe::do_sendmsg.

Document it as with the tcp stuff, remove an if(0)'d debugging block,
and remove the useless "sd" parameter since it's always the same as
the Pipe's sd member.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

msgr: write minimal documentation for the tcp functions.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

msgr: make a bunch of stuff private.

Why were all these data members public? They're accessed by Pipes
and the Accepter and stuff, so maybe that's why...but that's all
internal interface stuff.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

msg: update the Dispatcher and Messenger documentation

Clarify what mark_down() and mark_down_on_empty() actually do.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

dispatcher: fix documentation for ms_handle_reset

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

msgr: rename set_ip() -> set_addr_unknowns()

The generic interface shouldn't reference specifics like that.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

msgr: Remove _my_name and ms_addr, replace with direct access to my_inst.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

msgr: store the entity_inst_t in the Messenger.

Convert ms_addr and _my_name to be references to their fields in
the entity_inst_t my_inst.
This way we can use const references for accessing all of them,
instead of the bizarre distinction we had before for get_myinst().

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

buffer: implement a contents_equal function on bufferlists

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

msgr: change the signature of get_myaddr()

Return a const reference to the actual address, instead of copying it.
All current users are happy with this, and I can't see a good reason
to copy it instead.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

msgr: get_connection() is required to establish a connection if none exists.

Making an allowance for lossy server connections is silly. Just don't
ask for the Connection in that case. (There aren't any users who
rely on the previous behavior.)
Document that requirement in Messenger.h!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

test: fix monmaptool help text

Broken by commit:15f0a3270fdcf09acce554313f2d0c0814a511e4

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cls_rgw: guard decode

thee were few cases where decode wasn't guarded.

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

cls_rgw: reset return code in some cases

Beforehand the return code was ignored, so fixed the cases
where we erroneously return error instead of success.

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

librados: fix exec test

Return for read operations is now returned correctly.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

doc: disable broken 'doxygenclass' class in librados c++ doc

This is the last remaining gitbuilder error.  Add it back when the C++
docs actually build.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/stable'

test_workload_gen: fix Sequencer ctor

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/wip-name-sequencers'