librados, librbd: remove selfmanaged_snap_rollback_object

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge branch 'librados_api_cpp' into librados_api

Conflicts:
src/include/rbd/librbd.hpp
src/librbd.cc
src/rbd.cc

testlibrbdpp: use new librbd api

rbd: update for librbd api changes

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: tweak C++ API

- rename image_open to open and make it return an int
- remove Image::close, replace with destructor
- make Image constructor private

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados: C++ API rework

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rollback rename

rbd: use new librbd C++ api

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: make C++ api nicer

Adds Image class and replaces aio_create_completion with a constructor.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Some tweaks for the librados C API

rados_reopen_log: should take a cluster parameter.

Add rados_pool_list, rados_pool_list_free.

rados_snap_set_read -> rados_pool_snap_set_read

rados_snap_set_write_context -> rados_pool_selfmanaged_snap_set_write_ctx

write/write_full/etc: re-arrange parameter order to be the same as
pwrite(2).

Change interface of rados_pool_list a bit

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librbd, rbd: fill in the rest of image_info_t

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

More fixes, additions for config API

Add test of the librados configuration API to testrados.c

rados_reopen_log should return void since it can't encounter errors.

Create new rados_conf_get_alloc function that allocates memory, but has
a simpler interface.

Create rados_set_conf_defaults, a place to put librados-specific
configuration defaults.

Implement rados_conf_read_file by re-calling common_init. This isn't the
best way to do it, but it will get the function implemented for now.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Update testrados, rename functions in librados.cc

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_create: add id parameter

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Fold common_set_defaults into common_init

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Splt rados_init into rados_create + rados_connect

Splt rados_init into rados_create and rados_connect.  The pattern will
be for users to call create, set configuration, and then connect. Rename
rados_release to rados_destroy, to be more symmetrical with
rados_create. You can't reconnect after calling destroy.

Don't create the messenger inside the RadosClient constructor. Instead,
wait until RadosClient::connect().

Rename rados_conf_apply to rados_reopen_log. Add comment about SIGHUP.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add rados_conf_apply, comments

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

md_config_t::set_val/get_val

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common: more include and copyright fixes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common: Fix some missing includes, copyrights

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cconf: remove second argument to cconf --lookup

Everyone uses get_conf to get configuration values. So the logic for
defaulting to some value if we can't find the requested key should live
there. Also fix a case in cconf where we could encounter a usage error
and keep on going.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ceph_common.sh: remove get_val, get_val_bool

get_val and get_val_bool are unused.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Rename config.h -> common/config.h

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config.cc: doesn't depend on ceph_ver.h

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

testlibrbdpp: update for new librados and librbd APIs

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: update for new librados and librbd APIs

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: implement stacking on top of librados

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados: add constructor to allow client re-use

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados: switch to noun_verb function names

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: stack on top of librados

pybind: update rados python bindings for new API

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: more API cleanup; rados_conf_ stubs

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

move g_default_file_layout into ceph_fs.cc

It's defined in ceph_fs.h.

Fix a bunch of headers that use types without including the headers that
define those types.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: add cluster handle to C API

Had to add a layer of indirection to the list context handles.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: improve up_thru request behavior

There is some epoch the OSD wants for up_thru, based on when the PG mapping
last changed.  However, once the monitor gets to the point where it must
update the map, it should set up_thru to the most recent epoch the OSD has
seen (i.e. the epoch it is known to be "up thru"!).  This will hopefully/
frequently avoid any subsequent up_thru requests.

MOSDAlive already has a separate field (in PaxosServiceMessage) to hold the
latest epoch; just fix the constructor to set it properly, and make the
monitor use it.  No protocol change, yay!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: set correct epoch for up_thru osd->mon request

Put the epoch we need for up_thru in the request.  Putting the most recent
epoch causes incorrect osdmap churn.

Fixes: #824
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

PGMap: make osd_full and nearfull ratios configurable.

These were previously set by #defines. Pretty stupid
when we have a nice config system already!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Makefile: include ceph_argsparse.h in dist tarball

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: fix clone_range

This was broken by the safe_write() switchover; the success return value
is now 0, not the number of bytes written.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

common: Split argument parsing into ceph_argparse

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

keyring_init: don't print error when explicit key/keyfile is specified

e.g. when I am non-root and specify a key explicitly, no need to complain
about not being able to read root's /etc/ceph/keyring.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Revert "keyring_init: g_conf.keyring is not a list"

This reverts commit 2fb6036aa53f5eb3173b80fd17b7240bd3daf156.

keyring_init: g_conf.keyring is not a list

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Revert "Makefile.am: remove unused libs from linking with librbd tests and rbd"

Same problem as 38f38a99149e88f18072fcbdbee316ac21f6f30f.

This reverts commit e5db46cea0997f3f959b2ae896c980585f079ac0.

Clock: remove unused mutex

We don't use a mutex in g_clock any more, so let's not construct one any
more.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'pool_memory'

test: Add new memory tests, move to own subdir.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

vstart: remove directories, too.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

OSD: convert waiting_for_pg from hash_map to map.

This doesn't need to be a hash_map; there will only be an entry
for each PG that gets a message request while it's not active.
Shouldn't be too many PGs that that happens too, right?

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

PG: remove the object locking stubs and some dead code.

These are unused (#if 0'd, so no way to use them!) and require
a memory-hogging hash_map. Goodbye!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

PG: convert hash_maps to maps, remove unused.

waiting_for_[missing|degraded]_object don't need to be
hash_maps, and we don't use stat_object_temp_rd at all.
Swap to map and remove to reduce per-PG memory consumption!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

debug.h: cleanup includes

Shouldn't need to include DoutStreambuf.h; that's all implementation.
Don't include Mutex.h, since we don't use it.
*Do* include config.h, since we need it.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common: Move hex dump functions into hex.h

Move hex dump functions into hex.h. Remove unecessary includes from
debug.cc

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Makefile: version.cc should depend on ceph_ver

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

debug.h: move Ceph version stuff into version.h

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Revert "Makefile.am: remove unused libs from linking with librbd"

librados doesn't export ceph::buffer_total_alloc

This reverts commit 9bbd6c32a59ce0a2e4cc21a498e0b04bcd4781ed.

testlibrbdpp: fix off by one error in read test

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Makefile.am: remove unused libs from linking with librbd tests and rbd

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Makefile.am: remove unused libs from linking with librbd

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge remote branch 'origin/max_commit_size'

pybind/rados: write_full: remove silly extra param

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind/rados: implement Pool.write_full

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librbd: hold image context lock minimally

Holding the image context lock during snapshot removal prevented the
client from responding to a notify, causing a deadlock. This could be
triggered by removing a snapshot while concurrently adding more to the
same image.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

pybind/rados: implement Pool::change_auid

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind/rados: add rados.version

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind/rados: Add Rados.pool_exists

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind/rados: Snap.name should be a py string

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind/rados: add snapshots

Add snapshot lookup, iteration, creation, destruction interface.

Add test.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add Pool::list_objects

Add a Pool::list_objects method. Add a test for this to pybind-test.py

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add pybind-test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados-python bindings: Fix pool deletion a bit

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mkcephfs: fix premature tmp directory deletion

Previously, the temp directory would be deleted after the first daemon
on a host was started leaving the second one to fail.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

.gitignore: ignore testsnaps

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

.gitignore: ignore debian packaging outputs

monmaptool: fix command-line output

Don't check errno if it isn't set.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

dout: don't print version when forcing fg logging.

dout: don't print version when forcing fg logging.

Fix tests that were assuming us to spew errors about /var/log, which we
no longer do.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

dout initialization: remove unecessary flush()

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

dout: properly output ceph version on opening dout

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

monmaptool: set_foreground_logging

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

simplemessenger: Fix num_threads bug printout.

Also add documentation to get_num_threads since its contract
changed significantly.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

msgr: complain if there are > 1 threads, not 1

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

init-ceph: use do_cmd for pid_file dir creation

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix population on unconnected_watchers on obc load

Fixes: #807
Signed-off-by: Sage Weil <sage@newdream.net>

common: thread: get number of threads from /proc

The kernel knows how many threads we have; just ask it. One less atomic
variable to carry around.

We will eventually have to avoid doing this check for non-daemon code,
but that's a separate issue.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Remove ExportControl (we have better auth now)

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

init-ceph: status: use daemon_is_running

daemon_is_running does some nice things like check /proc/$pid/cmdline.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

init-ceph: fix status for multi-node clusters

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/osd: Fix indentation on RadosModel.h and TestSnaps.cc

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

testsnaps: add snapshot test

Uses RadosModel.h to check the results of a randomized sequence of
writes, reads, snapshots, snapshot removals, and rollbacks for errors.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'dout2'

Journaler: add some checks for expire_pos.

I don't think these are necessary checks, but the expire_pos >= trim_pos
invariant got broken somehow by johnl, and these checks won't hurt!

Signed-off-by: Greg Farnum <gregf@hq.newdream.net>

Journaler: call set_layout after init_headers.

set_layout modifies last_committed, but then init_headers
uses operator= and overwrites those changes. In this case
it doesn't matter as they're both writing the same changes,
but make the ordering explicit for the future.

Signed-off-by: Greg Farnum <gregf@hq.newdream.net>

OSD: ignore osd_max_write_size if it's set to 0.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

common/lockdep.cc: don't use dout unlocked

Lockdep should use the regular dout() interfaces, rather than going
around them. In particular, we shouldn't output to dout without taking
dout_lock.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

assert: allow assertions inside calls to dout()

We should handle the situation where we assert() while already holding
the dout() lock. At the same time, we want to get the dout lock if we
can, because it makes the logs look nicer. pthread_mutex_trylock solves
the dilemma.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

dout: Convert _dout_lock to plain pthread_mutex_t

Convert _dout_lock to plain pthread_mutex_t. This way, we don't have to
depend on the order of global constructor initialization. It should also
be slightly more efficient. The dout_lock was never subject to lockdep
anyway, so that's not an issue.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

assert.cc: some cleanup

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common/debug.h: use std::string rather than string

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common: Remove common/tls.cc

Using ELF TLS via the __thread keyword is much faster than using
pthread_getspecific and pthread_setspecific. It's also much nicer
looking syntactically. Finally, the __thread keyword is going to be
standardized in C++0x. So there's no reason to have an infrastructure
dependent on pthread_getspecific.

There were no users so this shouldn't affect anything negatively.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

dout: use DoutLocker rather than Mutex::Locker

Use DoutLocker rather than Mutex::Locker, in preparation for making the
dout_lock a plain old pthread_mutex_t.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>