cosd: replace cerr with derr

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: FileJournal: use derr

Use derr to announce errors in FileJournal.

Handle EINTR where necessary (still haven't fixed
read/write/pread/pwrite uses).

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: re-introduce derr

Re-introduce derr as a special log level (level -1) which will show up
in all logs, and on stderr. These messages are the only messages which
will show up on stderr.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: send notify message only to unexpired watchers

osd: timed out watcher is added to unconnected map

logging: close file when reloading global config

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: fix watch timer, locking

tools: don't start msgr thread before daemonize

Calling messenger->add_dispatcher_head() has the side-effect of starting
the messenger thread. So we must not do it before calling
messenger->start(), which sometimes calls daemonize.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

Merge branch 'objecter' into unstable

objecter: check for pg mapping changes in each incremental; refactor misc resubmission code

We need to detect when a pg mapping changes but the primary stays the same.
That means we can't just look at the final osdmap and see what is says; we
have to look at each intervening map and check each request to see if
something switched and the osd has thrown our request out.

Also refactor and clean up the linger vs normal op stuff some more.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: add reopen_session helper

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: cleanup: rename op maps

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: track pending requests by osd, not pg

This is a big cleanup.  Also
 - switch to keeping per-osd Connection *'s
 - make requests time out independently (not very efficiently yet)

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix inode ancestor attr encoding

Signed-off-by: Sage Weil <sage@newdream.net>

msgr: mark down by Connection*

Signed-off-by: Sage Weil <sage@newdream.net>

automake: ignore rmdir errors during uninstall

We don't want to fail "make distcheck" for a silly reason.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: send pending notification for reconnected watcher

automake: add osd/Watch.h to noinst_HEADERS

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: use Mutex::Locker

Use Mutex::Locker to make logging exception-safe. That is, if you are
doing "dout() << foo() << dendl;" and foo throws an exception, the dout
mutex will not be left in a locked state.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logger: Fix DoutStreambuf::create_rank_symlink

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

cephtool: rename tools files

Rename tools files to be more consistent. For example, the main()
function for ./ceph should be in ceph.cc.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

Merge branch 'sync2' into unstable

librados: drop watch_lock

Use the existing lock to do protect all of this.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: drop linger_info_mutex

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: simplify linger register

Drop single-use helper; make unregister_linger part of the public
interface.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: fix up linger ack/commit to trigger first time only

We only want the user-provided ack/commit callbacks to trigger the first
time we register the lingering op.  Same goes for the eversion_t *objver.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: clean up linger interface

Put LingerOp on heap.  Use xlist to attach to PGs.  Add in/out bufferlists.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'origin/unstable' into sync2

Conflicts:
src/auth/Crypto.cc
src/osd/ReplicatedPG.cc
src/osd/ReplicatedPG.h
src/osd/osd_types.h

logging: Fix use-before-access in debug.cc

Signed-off-by: Vangelis Koukis <vkoukis@cslab.ece.ntua.gr>
Signed-off-by: Constantinos Venetsanopoulos <cven@cslab.ece.ntua.gr>
Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: return ENOSPC for non-mds if full flag is set in osdmap

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'origin/syslog' into unstable

Conflicts:
src/mon/Paxos.cc
src/osd/PG.cc

Merge branch 'gceph' into unstable

gceph: Add gceph to rpm, deb

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

gceph: run shutdown functions at exit

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

gceph: fix compile

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

gceph: add -h argument

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

ceph tool: Create gceph

Put the gui into a separate binary.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

librados, objecter: fix unwatch operation

rpm: update changelog

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

rpm: fix ceph.spec to work with gcephtool

Don't try to package gui_resources unless we are building the GUI.
Get GUI dependencies correct.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

objecter: resend_linger copies ops

Fix overflow in FileJournal::_open_file()

[ The following text is in the "iso-8859-7" character set. ]
    [ Your display is set for the "iso-8859-1" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Running the unstable branch, mkcephfs fails when trying to create
a 3GB journal file on the OSDs.

Relevant messages from the osd logfile:

2010-12-09 19:03:54.419737 7fdde4d51720 journal _open_file: unable to extend journal to 18446744072560312320 bytes
2010-12-09 19:03:54.419789 7fdde4d51720 filestore(/osd) mkjournal error creating journal on /osd/journal

The problem is that the calculation of the journal size in bytes
overflows, in FileJournal::_open_file().

Signed-off-by: Vangelis Koukis <vkoukis@cslab.ece.ntua.gr>
Signed-off-by: Sage Weil <sage@newdream.net>

ReplicatedPG.cc: Fixes a bug in snap_trimmer where a pointer to a stack
Cond is left in the mode.waiting_cond list.

Signed-off-by: Samuel Just <samuelj@hq.newdream.net>

ReplicatedPG: snap_trimmer now acquires a read lock on the osd map
before calling share_pg_info.

Signed-off-by: Samuel Just <samuelj@hq.newdream.net>

rpm: don't try to package radosacl

radosacl is just a test binary, so unless we build with --with-debug, we
won't get it.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

rpm: add pkgconfig to BuildRequires

You can't build without pkgconfig.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

rpm: set files-attr for radosgw

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

filejournal: reset last_commited_seq if we find journal to be invalid

If we read an event that's later than our expected entry, we set read_pos
to -1 and discard the journal.  If that happens we also need to reset
last_committed_seq to avoid a crash like

2010-12-08 17:04:39.246950 7f269d138910 journal commit_finish thru 16904
2010-12-08 17:04:39.246961 7f269d138910 journal committed_thru 16904 < last_committed_seq 37778589
os/FileJournal.cc: In function 'virtual void FileJournal::committed_thru(uint64_t)':
os/FileJournal.cc:854: FAILED assert(seq >= last_committed_seq)
 ceph version 0.24~rc (commit:fe10300317383ec29948d7dbe3cb31b3aa277e3c)
 1: (FileJournal::committed_thru(unsigned long)+0xad) [0x588e7d]
 2: (JournalingObjectStore::commit_finish()+0x8c) [0x57f2ec]
 3: (FileStore::sync_entry()+0xcff) [0x5764cf]
 4: (FileStore::SyncThread::entry()+0xd) [0x506d9d]
 5: (Thread::_entry_func(void*)+0xa) [0x4790ba]
 6: /lib/libpthread.so.0 [0x7f26a2f8373a]
 7: (clone()+0x6d) [0x7f26a1c2569d]

Fixes #631

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: create a new op for resending lingering requests

logger: tweak cmon log output a bit

Make the output of cmon on stderr a little bit less verbose.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: DoutStreambuf: handle daemonizing better

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

objecter: a few lingering fixes

mon: use helper for clock drift check; log relative instead of absolute time

Signed-off-by: Sage Weil <sage@newdream.net>

logging: debug.h: use DoutStreambuf

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: Remove _dout_check_log

_dout_check_log is unneeded, since every invocation of dout makes the
same check.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: debug.h: minor cleanup

Don't put std::ostream into the global namespace. Copyright update.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: eliminate dbeginl

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: Implement rank symlinks

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: Support isym_path

Support instance symlinks, which are activated when we are using
g_conf.log_per_instance.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: rename_output_file -> handle_pid_change

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: _calculate_opath: use g_conf.log_dir

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: DoutStreambuf: better debug output

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: create_symlink:sometimes use rel symlinks

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: implement get_dirname, move get_basename

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: fix normalize_relative

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

mds: sync->mix replica state is sync->mix(2)

When auth first moves to sync->mix,
 - auth sends AC_MIX to replicas
 - replicas go to sync->mix
 - replicas finish gather, send AC_SYNCACK, move to sync->mix(2)
 - auth gets all acks, sends AC_MIX again
 - replica moves to MIX

So any new replica should just get sync->mix(2), so that it is not confused
by the second AC_MIX.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: no not choose lock state on replicas

The lock state has already been set during rejoin.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: small rejoin cleanup

Signed-off-by: Sage Weil <sage@newdream.net>

mds: rev mds cluster internal protocol

The lock encoding changed with the dirty bit on scatterlocks.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix replay of already-journaled requests

Check for already-completed tids for both retried and replayed requests.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: open undef dirfrags during rejoin

Any invented dirfrags have a version of 0.  This will cause problems later
if we pre_dirty() anything in that dir because the dir version won't be
in sync (it'll be way too small).  Also, we can do that at any point,
e.g. when flushing dirty caps, and aren't allowed to delay, so we need to
load those dirfrags now.

In theory we could read only the fnode and not all the dentries, but we
may as well.  We should be more careful about memory that this patch is,
though.

Fixes #15.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: add missing try_clear_more() to scatterlock

Signed-off-by: Sage Weil <sage@newdream.net>

mds: explicitly pass scatterlock dirty flag to auth on gather

This ensures that if the replica is thinks it is flushing something the
auth will always do a scatter_writebehind.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: send LOCKFLUSHED to trigger finish_flush on replicas

Since f741766a we have triggered start_flush and finish_flush on replicas.
The problem is that the finish_flush didn't always happen for the mix->lock
case: we sould start_flush when we sent the AC_LOCKACK, but could only
finish_flush if/when we got another SYNC or MIX.  If the primary stayed in
the LOCK state, we would keep our flushing flag.  That in turn causes
problems later when we try to eval_gather() (esp if we are auth at that
point?).

Fix this by sending an explicit AC_LOCKFLUSHED message to replicas after
we do a scatter_writebehind.  The replica will only set flushing if it
flushed dirty data, which forces scatter_writebehind, so we will always
get the LOCKFLUSHED to match.  Replicas that didn't flush will also get
it, but oh well.  We'd need to keep track which ones sent dirty data to
do that properly, though.

TODO: still need to verify that this is correct for rejoin.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: clear EXPORTINGCAPS on export_reverse

We need to reverse the effects of encode_export_inode_caps(), which is just
the pin and state bit.

The original problem can be reproduced with
 - ceph tell mds 0 injectargs '--mds-kill-import-at 5'
 - restart mds
 - recovery completes successfully
 - wait for the subtree to be reexported
 - fail with bad EXPORTINGCAPS get in encode_export_inode_caps

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix LOOKUPHASH to avoid creating bogus replica CDir

We can't create the CDir if we are non-auth.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: introduce rejoin_invent_dirfrag() helper

Signed-off-by: Sage Weil <sage@newdream.net>

logging: Add symlink helper functions

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logger: fix EINTR handling

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: rename_output_file: fix bug

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: DoutStreambuf: Implement log-to-file

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: Add log_to_file option

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: DoutStreambuf improvements

Write to stdout_fileno directly rather than using a buffer, which we
would then have to flush. Fix a bug in the buffering of priorities.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

automake: in scripts, use sysconfdir as-is

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

automake: in deb pkg, use --syconfdir=/etc

When building the debian packages, use --sysconfdir=/etc.

Also, don't fudge sysconfdir in the init-ceph script.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: add DoutStreambuf::set_prio

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: DoutStreambuf must handle stdout + stderr

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: Add log_to_syslog option

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

mkcephfs: require -k; update man page

Force users to specify keyring location; update man page accordingly.

Signed-off-by: Sage Weil <sage@newdream.net>

Rename SyslogStreambuf -> DoutStreambuf

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: debug.h: move some debug functions

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: optimize with likely/unlikely macros

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: Replace derr with dout

derr was really just an alias for STDERR. Unfortunately, after we call
daemonize, STDERR is connected to /dev/null. So just replace calls to
derr with dout so that our important messages don't get lost.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: add g_conf.clog_to_syslog

Add a new configuration option that allows you to send central log
messages to syslog.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: LogEntry: don't pass enums by reference

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: MLog.h: const cleanup

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: LogClient: refactor handle_log_ack

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging:Move LogEntry.h into common with LogClient

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: better syntax for LogClient

Rather than having to write logclient.log(LOG_ERROR, ss), coders can now
write clog.error() << "str". Auto-flushing, if enabled, is still
handled automatically.

Rename instances of LogClient to clog (central log) for consistency and
brevity.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

configure: detect crypto++ library