OSD: convert waiting_for_pg from hash_map to map.

This doesn't need to be a hash_map; there will only be an entry
for each PG that gets a message request while it's not active.
Shouldn't be too many PGs that that happens too, right?

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

PG: remove the object locking stubs and some dead code.

These are unused (#if 0'd, so no way to use them!) and require
a memory-hogging hash_map. Goodbye!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

PG: convert hash_maps to maps, remove unused.

waiting_for_[missing|degraded]_object don't need to be
hash_maps, and we don't use stat_object_temp_rd at all.
Swap to map and remove to reduce per-PG memory consumption!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

debug.h: cleanup includes

Shouldn't need to include DoutStreambuf.h; that's all implementation.
Don't include Mutex.h, since we don't use it.
*Do* include config.h, since we need it.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common: Move hex dump functions into hex.h

Move hex dump functions into hex.h. Remove unecessary includes from
debug.cc

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Makefile: version.cc should depend on ceph_ver

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

debug.h: move Ceph version stuff into version.h

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Revert "Makefile.am: remove unused libs from linking with librbd"

librados doesn't export ceph::buffer_total_alloc

This reverts commit 9bbd6c32a59ce0a2e4cc21a498e0b04bcd4781ed.

testlibrbdpp: fix off by one error in read test

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Makefile.am: remove unused libs from linking with librbd tests and rbd

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Makefile.am: remove unused libs from linking with librbd

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge remote branch 'origin/max_commit_size'

pybind/rados: write_full: remove silly extra param

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind/rados: implement Pool.write_full

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librbd: hold image context lock minimally

Holding the image context lock during snapshot removal prevented the
client from responding to a notify, causing a deadlock. This could be
triggered by removing a snapshot while concurrently adding more to the
same image.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

pybind/rados: implement Pool::change_auid

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind/rados: add rados.version

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind/rados: Add Rados.pool_exists

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind/rados: Snap.name should be a py string

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind/rados: add snapshots

Add snapshot lookup, iteration, creation, destruction interface.

Add test.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add Pool::list_objects

Add a Pool::list_objects method. Add a test for this to pybind-test.py

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add pybind-test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados-python bindings: Fix pool deletion a bit

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mkcephfs: fix premature tmp directory deletion

Previously, the temp directory would be deleted after the first daemon
on a host was started leaving the second one to fail.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

.gitignore: ignore testsnaps

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

.gitignore: ignore debian packaging outputs

monmaptool: fix command-line output

Don't check errno if it isn't set.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

dout: don't print version when forcing fg logging.

dout: don't print version when forcing fg logging.

Fix tests that were assuming us to spew errors about /var/log, which we
no longer do.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

dout initialization: remove unecessary flush()

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

dout: properly output ceph version on opening dout

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

monmaptool: set_foreground_logging

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

simplemessenger: Fix num_threads bug printout.

Also add documentation to get_num_threads since its contract
changed significantly.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

msgr: complain if there are > 1 threads, not 1

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

init-ceph: use do_cmd for pid_file dir creation

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix population on unconnected_watchers on obc load

Fixes: #807
Signed-off-by: Sage Weil <sage@newdream.net>

common: thread: get number of threads from /proc

The kernel knows how many threads we have; just ask it. One less atomic
variable to carry around.

We will eventually have to avoid doing this check for non-daemon code,
but that's a separate issue.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Remove ExportControl (we have better auth now)

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

init-ceph: status: use daemon_is_running

daemon_is_running does some nice things like check /proc/$pid/cmdline.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

init-ceph: fix status for multi-node clusters

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/osd: Fix indentation on RadosModel.h and TestSnaps.cc

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

testsnaps: add snapshot test

Uses RadosModel.h to check the results of a randomized sequence of
writes, reads, snapshots, snapshot removals, and rollbacks for errors.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'dout2'

Journaler: add some checks for expire_pos.

I don't think these are necessary checks, but the expire_pos >= trim_pos
invariant got broken somehow by johnl, and these checks won't hurt!

Signed-off-by: Greg Farnum <gregf@hq.newdream.net>

Journaler: call set_layout after init_headers.

set_layout modifies last_committed, but then init_headers
uses operator= and overwrites those changes. In this case
it doesn't matter as they're both writing the same changes,
but make the ordering explicit for the future.

Signed-off-by: Greg Farnum <gregf@hq.newdream.net>

OSD: ignore osd_max_write_size if it's set to 0.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

common/lockdep.cc: don't use dout unlocked

Lockdep should use the regular dout() interfaces, rather than going
around them. In particular, we shouldn't output to dout without taking
dout_lock.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

assert: allow assertions inside calls to dout()

We should handle the situation where we assert() while already holding
the dout() lock. At the same time, we want to get the dout lock if we
can, because it makes the logs look nicer. pthread_mutex_trylock solves
the dilemma.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

dout: Convert _dout_lock to plain pthread_mutex_t

Convert _dout_lock to plain pthread_mutex_t. This way, we don't have to
depend on the order of global constructor initialization. It should also
be slightly more efficient. The dout_lock was never subject to lockdep
anyway, so that's not an issue.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

assert.cc: some cleanup

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common/debug.h: use std::string rather than string

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common: Remove common/tls.cc

Using ELF TLS via the __thread keyword is much faster than using
pthread_getspecific and pthread_setspecific. It's also much nicer
looking syntactically. Finally, the __thread keyword is going to be
standardized in C++0x. So there's no reason to have an infrastructure
dependent on pthread_getspecific.

There were no users so this shouldn't affect anything negatively.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

dout: use DoutLocker rather than Mutex::Locker

Use DoutLocker rather than Mutex::Locker, in preparation for making the
dout_lock a plain old pthread_mutex_t.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

os/FileStore: use derr/dendl for dout locking

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

os/FileStore: use ceph_abort rather than abort

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

lockdep: balance dout and dendl, fix whitespace

Make lockdep use dendl the same way as the other code. This is in
preparation for making lockdep use normal dout() rather than an unlocked
version.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Journaler: fix bad assert.

We can call reread_head during normal replay under
certain circumstances. So add the REREAD_HEAD state
as allowed.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

testlibrbd: fix printf args

Stupid me!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

common: create dout_emergency interface and use it

Create the dout_emergency interface, which is safe to call from a signal
handler or from inside dout itself.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cfuse: use safe_read and check return value

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

testlibrbd: check return values

Stupid printf!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

debian: add python, python-dev build-deps

Might be overkill?  The error I see from pbuilder is

checking for a Python interpreter with version >= 2.4... none
error: configure: in `/tmp/buildd/ceph-0.24.3-676-gcde53e9':
error: configure: Failed to find Python 2.4 or newer

...but I'm guessing python-dev is needed too?

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: clean up Pipe::queue_received locking

Ensure we maintain the invariant that a pipe has a non-empty queue IFF
the pipe is queued.

Prompted by #798.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

MDCache: switch CDir::_commit so that it can limit max write size.

This should fix #777.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

MDCache: add max_dir_commit_size.

Configured by setting mds_dir_max_commit_size in conf, or else
by looking at osd_max_write_size. This should lead to sane
max commits even if the user doesn't specify anything.

This will be used in the next commit or to by CDir.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

MDS: Don't always _commit_full just because we have a complete dir.

Instead, commit if a certain percentage of the dentries are dirty.
Configurable via mds_dir_commit_ratio!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote branch 'origin/librbd'

objecter: set linger op target pg when a linger is resent

send_linger always creates a new Op, but op_submit does not fill in
the target pg if an existing session is passed in, so when a linger
was resent, it had the wrong pg set.

This caused a crash in cosd with debugging turned on when running
testlibrbd twice. This occurred because the object context for the
linger in the wrong pg had no object name set.

Disable lockdep for ExportControl, ConfFile locks

Currently, we haven't read the configuration at the time we initialize
these locks. So we can't know whether lockdep has been enabled, or what
verbosity it is supposed to have. So just disable it on these locks.

Potentially ExportControl's initialization could be moved to after
g_conf.lockdep and g_conf.debug_lockdep have been read from the
configuration, if lockdep is needed for this component.

ConfFile probably doesn't need a lock at all, but that's another story.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

MDCache: switch CDir::_commit so that it can limit max write size.

This should fix #777.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

MDCache: add max_dir_commit_size.

Configured by setting mds_dir_max_commit_size in conf, or else
by looking at osd_max_write_size. This should lead to sane
max commits even if the user doesn't specify anything.

This will be used in the next commit or to by CDir.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

MDS: Don't always _commit_full just because we have a complete dir.

Instead, commit if a certain percentage of the dentries are dirty.
Configurable via mds_dir_commit_ratio!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

.gitignore: py-compile

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'pybind'

librados: include headers in dist tarball

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph.spec.in: add librbd

Make git ignore librbd tests.

ceph: add --concise flag

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'stable'

Conflicts:
configure.ac

monclient: fix weirdness when we explicitly start with non-master mon

Fix

$ ./ceph -m 10.0.1.252:6790 health
2011-02-11 12:30:38.132728 mon <- [health]
2011-02-11 12:30:38.133243 mon <- [health]
2011-02-11 12:30:38.133697 mon0 -> 'Access denied' (-13)

when the IP we specify is, e.g., for mon.b.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

config: Remove g_conf.num_osd, num_mds, num_mon

These configuration options don't serve any use and were confusing.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Make git ignore qa/workunits/direct_io_test.

MonitorStore: check for ENOENT, not EEXIST

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common: bufferlist::read_file: use safe_read

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

auth/Crypto.cc: use safe_read_exact

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

os/FileStore: remove unused variable

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

v0.24.3

make:add messages/MOSDRepScrub.h to NOINST_HEADERS

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cephtool: get_indata should use safe_read_exact

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mon/MonitorStore.cc: use safe_read

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados.cc: use fwrite to output data to stdout

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common/common_init: use safe_read

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

os/FileStore: fix error handling

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

s3.sh: simple bug fix

rados: add create op

common/DoutStreambuf: use safe_io.h

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

tools/common: use fwrite to write stdout

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: write_meta/read_meta: fix error handling

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

msgr: better error checking for write_pid_file

Use safe_write. Check error code of close.

Fix what looks like a "lack of braces around multi-statement block"
error where we always complain that we're unable to write the pid file,
even if we succeed.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

os/FileStore.cc: use safe_write

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common/buffer.cc: kill deadcode

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>