do_autogen.sh: don't clear CFLAGS, CXXFLAGS

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

testceph: add more tests

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

libceph: fix some bugs

Fix some bugs in libceph and testceph.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

libceph: avoid DIR*, combine connect and mount

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

hadoop: whitespace fixes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Client: unmount should not be able to fail

If unmount fails, what is the API users supposed to do?
The client needs to be able to clean up after itself.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

libceph API change

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mkcephfs: Fail if there are no mons defined.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mkcephfs: Fail if a mon has no address set.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mkcephfs: Remove temp dir, not just its contents.

Also fixes the case where no temp files were created in the dir.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mon: consisder osd recovery state in health check

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: include unfound count in pg recovery summary

Fixes: #1003
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: fix up osd health report

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: fix up pg health report

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: generalize health check a bit

Any PaxosService can return a status (or not).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'master' of ssh://ceph.newdream.net/home/sage/ceph.newdream.net/git/ceph

Makefile.am: add -fno-strict-aliasing

Until we audit every place we have a typecast to make sure it's safe,
this should be off.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Makefile.am: Wpointer-arith, Wstrict-null-sentinel

Add Wpointer-arith, Wstrict-null-sentinel, Winit-self

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

clitests: fix osdmap unit test

Broken by removal of casdata in 8cd3f1cd2546a61bb5402fa374de7f37147dc3a0.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: add some new warnings to CXXFLAGS

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: don't daemonize when doing journal reset.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rbd: rename kernel add/remove to map/unmap

Unmap takes a device now, and uses the major number
to find the right sequence number in sysfs.

Also updated the man page.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge remote branch 'origin/next'

rbd: add commands for manipulating the kernel sysfs interface

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

mount.ceph: use common implementations of secret handling and address resolving

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

addr_parsing: use strtok_r to be safe

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

addr_parsing: rename mount_resolve_dest to resolve_addrs and don't modify its argument

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

addr_parsing: move implementation out of header

Also include missing headers and make it linkable with C,
since it's duplicated from mount.ceph.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: use rados' init_with_config because we already called common_init

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

common, mount.ceph: move functions for working with secrets into secret.h

Use sprintf instead of safe_cat, since we're just writing a string once.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge remote branch 'origin/next'

testlibrbd: fix signed/unsigned comparisons

testlibrbd.c: In function 'write_test_data':
testlibrbd.c:191: warning: comparison between signed and unsigned integer expressions
testlibrbd.c: In function 'aio_read_test_data':
testlibrbd.c:207: warning: comparison between signed and unsigned integer expressions
testlibrbd.c: In function 'read_test_data':
testlibrbd.c:222: warning: comparison between signed and unsigned integer expressions

Signed-off-by: Sage Weil <sage@newdream.net>

osd: drop 'casdata' bucket by default

We don't use it for anything.  It just means we create a bunch of useless
PGs.

Signed-off-by: Sage Weil <sage@newdream.net>

cmds: make journal-check mode run in the foreground.

daemonize does a bunch of stuff we don't want to do.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: massage handling of oneshot-replay.

We accidentally broke it in our last round of changes to support
standby-replay; patch it up again.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

OSDMon: make the defaults in reweight-by-utilization work.

an oload <= 100 fails out; set the new default to 120.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

testlibrbd: fix signed/unsigned comparisons

testlibrbd.c: In function 'write_test_data':
testlibrbd.c:191: warning: comparison between signed and unsigned integer expressions
testlibrbd.c: In function 'aio_read_test_data':
testlibrbd.c:207: warning: comparison between signed and unsigned integer expressions
testlibrbd.c: In function 'read_test_data':
testlibrbd.c:222: warning: comparison between signed and unsigned integer expressions

Signed-off-by: Sage Weil <sage@newdream.net>

osd: drop 'casdata' bucket by default

We don't use it for anything.  It just means we create a bunch of useless
PGs.

Signed-off-by: Sage Weil <sage@newdream.net>

auth: const cleanup

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

obysnc: use python2.5 compatible exception syntax

Apparently using "as" in "except" blocks is a new (and not
backwards-compatible) thing in Python 2.6.x. Who knew?

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

PGMonitor: unregister the config obs in ~PGMonitor

Using the new unregister call, unregister the config observer in
~PGMonitor.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: add remove_observer

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

PGMon: use the config observer framework

We now register observers on mon_osd_full_ratio and nearfull_ratio
to update the OSD full/nearfull lists.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ProfLogger: add NULL to KEYS in the config observer.

I believe that without the null pointer it will overrun into
uninitialized memory.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

config: add comment clarifying virtual destructor.

Just in case anybody else is like me and gets confused.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

PGMap: fill in full OSD sets on decode.

Previously these sets were only modified when applying an incremental,
which might have caused problems when restarting monitors. Whoops!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote branch 'origin/classdist'

conf: reopen proflogger on conf change

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

conf: add observer framework

Objects can now register as configuration observers interested in a
subset of the configuration keys. The observers will be told exactly
which keys have changed.

The first user is dout, which now no longer needs the infamous SIGHUP
hack to know when to reopen the config file.

librados: Remove rados_reopen_log, which was basically a means for the
library user to trigger the SIGHUP behavior.

Changes are accumulated and applied all at once by apply_changes. This
function is called as part of common_init, and after every call to
injectargs.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cauthtool: -C not -c in man page

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

dout: make dout a member of md_config_t

Since logging options are per-config, logically DoutStreambuf instances
should also be per-config. This also allows us to eliminate the
"if (uninitialized)" checks at the beginning of every call to dout.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

dout: remove DOUTSB_FLAG_STDOUT

This was used to implement log_to_stdout, but that doesn't exist any
more. Instead of we have log_to_stderr and the other stuff.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

PGMap: add redo_full_sets() function for when ratios change.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osdmon: Clean up debug output on map full flag.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

PGMap: add [near]full_ratio to the Incremental and encoding.

This way these ratios can be distributed by map updates.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

PGMap: always decode pg_remove.

pg_remove has been included for longer than we've had versions
in the struct, so this check for end is useless -- if pg_remove
wasn't encoded we're already broken by decoding the version at
the beginning.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: better debug output on replay completion

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mkcephfs: allow a prebuild osdmap to be specified

Otherwise we'll create one with osdmaptool --createsimple with the default
generic settins.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

cfuse, mds, dout: Do not create "rank" symlinks for log files.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Revert "Revert "autoconf: Complain if tcmalloc is not found.""

This reverts commit 05c281bfa9e9d69ea3d0197590950c8e6845a13a.

This should be okay now.

debian: Handle missing tcmalloc on Debian lenny.

lenny doesn't have a suitable libgoogle-perftools-dev, and
release.sh edits it out of build-deps. Detect that and tell
configure that not having tcmalloc is ok.

This should make 05c281bfa9e9d69ea3d0197590950c8e6845a13a
unnecessary.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: Build without tcmalloc on non-i386/amd64.

This is not strictly needed as of 05c281bfa9e9d69ea3d0197590950c8e6845a13a,
but that reverting is hopefully only temporary.

Without this, with 05c281 undone, non-mainstream architectures
would fail to build.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

remove old debian build helper scripts.

This is now handled by ceph-build.git

Signed-off-by: Sage Weil <sage@newdream.net>

mon: remove class distribution infrastructure

This is now the admin's job.  Removes a lot of code with limited testing
and coverage.

We rev the internal monitor protocol because the state machine ids changed.
This should not affect the on-disk format. Just stop and restart all the
monitors at once during the upgrade.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: move ClassHandler.{cc,h} to osd/

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: load classes from a fixed location off disk

Simplify the class loading code to just load out of a local directory.
Do not request classes from the monitor.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

common: add environment.cc/environment.h

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: remove MDSlaveUpdate from list on deletion

These are added to the LogSegment list on the slaves, but also need to be
removed from that list when we replay a COMMIT|ROLLBACK or when the op's
fate is determined during the resolve stage.

This fixes a crash like

./include/elist.h: In function 'elist<T>::item::~item() [with T =
MDSlaveUpdate*]', in thread '0x7fb2004d5700'
./include/elist.h: 39: FAILED assert(!is_on_list())
 ceph version 0.26 (commit:9981ff90968398da43c63106694d661f5e3d07d5)
 1: (MDSlaveUpdate::~MDSlaveUpdate()+0x59) [0x4d9fe9]
 2: (ESlaveUpdate::replay(MDS*)+0x422) [0x4d2772]
 3: (MDLog::_replay_thread()+0xb90) [0x67f850]
 4: (MDLog::ReplayThread::entry()+0xd) [0x4b89ed]
 5: (()+0x7971) [0x7fb20564a971]
 6: (clone()+0x6d) [0x7fb2042e692d]
 ceph version 0.26 (commit:9981ff90968398da43c63106694d661f5e3d07d5)
 1: (MDSlaveUpdate::~MDSlaveUpdate()+0x59) [0x4d9fe9]
 2: (ESlaveUpdate::replay(MDS*)+0x422) [0x4d2772]
 3: (MDLog::_replay_thread()+0xb90) [0x67f850]
 4: (MDLog::ReplayThread::entry()+0xd) [0x4b89ed]
 5: (()+0x7971) [0x7fb20564a971]

Fixes: #1019
Signed-off-by: Sage Weil <sage@newdream.net>

common_init: move version printout to common_init

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

interval_set: slightly better API for erase

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Change some global config options to env vars

Options that are inherently global, like malloc settings, and also
inherently debugging or profiling settings should be environment
variables.

tcmalloc_profiler_run, profiler_allocation_interval,
profiler_highwater_interval, and buffer_track_alloc fall into this
category.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge commit '8038c491ba90a8cbcd569e84d4cafc8bbdff81d5' into next

Merge remote branch 'origin/stable' into next

journaler: fix do_flush adjustment

Signed-off-by: Sage Weil <sage@newdream.net>

osd: assert on query on deleting pg

This shouldn't happen; assert if it does.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: do not create pgs on query

We do not need to create the pg on query.  If we are a new replica we can
create it when we get the PGLog activate message.

Signed-off-by: Sage Weil <sage@newdream.net>

journaler: truncate/zero ahead of write position

Remove/zero objects N periods ahead of the journal write position.  This
ensures that when we reprobe the journal length, we will always detect the
end position as the correct write_pos, even when there is weird data
"ahead" of us that we may bump up against.

Signed-off-by: Sage Weil <sage@newdream.net>

filer: remove whole objects on zero()

Make filer::zero() remove any whole objects.  This is required by the
Journaler, given the way it probes the journal length.

Signed-off-by: Sage Weil <sage@newdream.net>

interval_set: implement erase(iterator)

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'next'

journaler: check return code on _finish_write_head.

If we get an error code and assume we successfully wrote the head,
there are going to be all kinds of issues on replay!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: make ZERO on non-existent object a no-op

Fixes bug where oi.size gets out of sync with the object size because we
actually write zeros.  (This explains #933.)

Signed-off-by: Sage Weil <sage@newdream.net>

dout: remove references to g_conf

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

vstart tests: specify ceph.conf

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: return appropriate error for unresolvable email in acl

config: reorder config_option struct members

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

clitests: fix radosgw_admin test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: remove unused OPT_NONE type

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

clitests: eliminate use of old-style section name

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

MDS: move slave rename xlock handling before finish_export_inode.

finish_export_inode changes states! That's not good for our checks,
so just handle unpinning and stuff before we finish_export_inode.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

improve debug printing

mds: Unify migration-handling code in _commit_slave_rename.

We need to handle locks and pins on exported inodes but we
were using a separate if block with its own (non-matching!) check
for no good reason.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: _commit_slave_rename needs to drop auth_pins for exported xlocks.

Otherwise these pins are never dropped from the inode since we
don't go through our normal xlock teardown code. Now we do!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

MDS: Make _rename_apply inode import auth_pinning more intelligent.

We don't want auth_pins on the locallocks (they're never auth_pinned)
and we only want new auth_pins that are for locks on the inode that we
imported -- not for each xlock that the mdr has everywhere (like,
say, on the srcdn)!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: If we're a slave, clean up xlocks when we export an inode.

Because we can do an inode import during a rename that skips the usual
channels, we were getting into an odd state with the xlocks (which we
did as a slave for an inode that we exported away). Clean up the
record of these xlocks for inodes before we get into the request
cleanup (at which point we are labeled as no-longer-auth, and the
standard cleanup routines will break).

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: properly drop imported xlocks.

Because we can do an inode import during a rename that skips the usual
channels, we were getting into an odd state with the xlocks (which
were formerly remote and are now local). Clean up the record of
those remote xlocks.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

MDS: Server takes auth_pins for xlocks on imported inodes.

Should fix #934.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

objecter: resub ops on full->nonfull transition

This was broken a while ago during the last refactor.  Whoops!  Clean it
up to be smarter (and work at all).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: show "full" or "nearfull" in osdmap summary line

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'origin/stable'

Conflicts:
src/osdc/Journaler.cc