logging: _calculate_opath: use g_conf.log_dir

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: DoutStreambuf: better debug output

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: create_symlink:sometimes use rel symlinks

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: implement get_dirname, move get_basename

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: fix normalize_relative

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: Add symlink helper functions

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logger: fix EINTR handling

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: rename_output_file: fix bug

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: DoutStreambuf: Implement log-to-file

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: Add log_to_file option

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: DoutStreambuf improvements

Write to stdout_fileno directly rather than using a buffer, which we
would then have to flush. Fix a bug in the buffering of priorities.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: add DoutStreambuf::set_prio

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: DoutStreambuf must handle stdout + stderr

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: Add log_to_syslog option

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

Rename SyslogStreambuf -> DoutStreambuf

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: debug.h: move some debug functions

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: optimize with likely/unlikely macros

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: Replace derr with dout

derr was really just an alias for STDERR. Unfortunately, after we call
daemonize, STDERR is connected to /dev/null. So just replace calls to
derr with dout so that our important messages don't get lost.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: add g_conf.clog_to_syslog

Add a new configuration option that allows you to send central log
messages to syslog.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: LogEntry: don't pass enums by reference

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: MLog.h: const cleanup

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: LogClient: refactor handle_log_ack

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging:Move LogEntry.h into common with LogClient

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logging: better syntax for LogClient

Rather than having to write logclient.log(LOG_ERROR, ss), coders can now
write clog.error() << "str". Auto-flushing, if enabled, is still
handled automatically.

Rename instances of LogClient to clog (central log) for consistency and
brevity.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

Create SyslogStreambuf

SyslogStreambuf is a kind of stream buffer that allows you to output
characters from an ostream to syslog. Most standard IO streams can make
use of this Streambuf.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

v0.25~rc

Merge remote branch 'origin/lost' into unstable

Conflicts:
src/osd/osd_types.h

osd: refactor object_info_t constructor a bit

Create a copy constructor for object_info_t, since we often want to copy
an object_info_t and would rather not try to remember all the fields.
Drop the lost parameter from one of the other constructors, because it's
not used that much.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: share_pg_log: update peer_missing

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: mark_obj_as_lost: fix oloc init, eversion

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: mark_all_unfound_as_lost: bugfix, refactor

mark_all_unfound_as_lost: just delete items from the rmissing set as we
find them, rather than using a multi-pass system.

Update info.last_update as we go so that log printouts will look correct
(the log printout function checks info.last_update)

Don't remove from missing or missing_loc in mark_obj_as_lost.
PG::missing_loc should never have the soid, and PG::missing we handle
elsewhere.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: mark_obj_as_lost: don't assume we have obj

In PG::mark_obj_as_lost, we have to mark a missing object as lost. We
should not assume that we have an old version of the missing object in
the ObjectStore. If the object doesn't exist in the object store, we
have to create it so that recovery can function correctly.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: create lost2 test

This one verifies:
1. Client asks for an unfound object and gets put to sleep
2. Object gets declared lost
3. Client wakes up

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: mark_all_unfound_as_lost: set lost attr

In mark_all_unfound_as_lost, we need to set the lost bit in the objects'
object_info_t.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

radostool: fix memleak in error path

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: mark_all_unfound_as_lost: wake waiters

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

test_lost: add lost1 test

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: ReplicatedPG::do_op: error on read-from-lost

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: don't mark objs as lost unless we're active

We don't have enough information to mark objects as lost until we
activate the PG. might_have_unfound isn't even built until PG::activate.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

mds: fix resolve for surviving observers

Make all survivors participate in resolve stage, so that survivors can
properly determine the outcome of migrations to the failed node that did
not complete.

The sequence (before):
 - A starts to export /foo to B
 - C has ambiguous auth (A,B) in it's subtree map
 - B journals import_start
 - B fails
...
 - B restarts
 - B sends resolves to everyone
   - does not claim /foo
 - A sends resolve _only_ to B
   - does claim /foo
 - B knows it's import did not complete
 - C doesn't know anything.  Also, the maybe_resolve_finish stuff was
   totally broken because the recovery_set wasn't initialized

See new (commented out) assert in Migrator.cc to reproduce the above.

Signed-off-by: Sage Weil <sage@newdream.net>

test: dump_osd_store: sort dump output

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: active replicas process logs from primaries

In _process_pg_info, if the primary sends us a PG::Log, a replica should
merge that log into its own.

mark_all_unfound_as_lost / share_pg_log: don't send the whole PG::Log.
Just send the new entries that were just added when marking the objects
as lost.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: object_info_t: add lost field

We can now permanently mark objects as lost by setting the lost bit in
their object_info_t. Rev the object_info_t struct.

get_object_context: re-arrange this so that we're always setting the
lost bit. Also avoid some unecessary steps.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

Add ./ceph dump pg debug degraded_pgs_exist

./ceph dump pg debug degraded_pgs_exist returns TRUE if some pgs are
degraded; false otherwise.

tests: move start_recovery into test_common.sh.
Create recovery1 test.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

(re)add mechanism for marking objects as lost

In activate_map, we now mark objects that we know are unfindable as
lost. This relies on the might_have_unfound set introduced earlier.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: fix object_info_t() initialization of oloc

Signed-off-by: Sage Weil <sage@newdream.net>

mds: add debug output to make completions easier to track

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix misuses of OLOC_BLANK

Commit 6e2b594b fixed a bunch of bad get_object_context() calls, but even
with the parameter fixed some were still broken.  Pass in a valid oloc in
those cases.  The only places where OLOC_BLANK _is_ still uses is when we
know we have the object locally and will load a valid value off disk.

Signed-off-by: Sage Weil <sage@newdream.net>

Revert "mds: resolve cleanup"

This reverts commit cd53719f3ce712a060e4ac80cab934c597531a5e.

We need this on surviving nodes too to resolve ambiguous migrations to/from recoverying
nodes.

Merge branch 'testing' into unstable

Conflicts:
src/os/FileJournal.cc

osd: make recovery_oids debug list per-pg

Otherwise we hit bad asserts if an object of the same name in different
pools is getting recovered simultaneously.

Signed-off-by: Sage Weil <sage@newdream.net>

client: Set the DirResult buffer to NULL when deleting it.

This should fix a crash exposed by our bonnie workunit. Previously
the client would keep trying to read out of the (deleted) buffer!

ceph.spec.in: include gui files

Signed-off-by: Sage Weil <sage@newdream.net>

debian: many many cleanups

Signed-off-by: Laszlo Boszormenyi <gcs@debian.hu>

filejournal: fix throttle vs FULL behavior

We don't want to add to the throttler if we aren't going to queue the
write, or else we'll never take it off again.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'osd_journaling' into unstable

filestore: make sure blocked op_start's wake up in order

If they wake up out of order (which, theoretically, they could before) we
can screw up journal submitting order in writebehind mode, or apply order
in parallel and writeahead journal mode.

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: assert op_submit_finish is called in order

Verify/assert that we aren't screwing up the submission pipeline ordering.
Namely, we want to make sure that if op_apply_start() blocks, we wake up
in the proper order and don't screw up the journaling.

Signed-off-by: Sage Weil <sage@newdream.net>

filejournal: rework journal FULL behavior and fix throttling

Keep distinct states for FULL, WAIT, and NOTFULL.

The old code was more or less correct at one point, but assumed the seq
changed on each commit, not each operation; in it's prior state it was
totally broken.

Also fix throttling (we were leaking items in the throttler that were
submitted while the journal was full).

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: refactor op_queue/journal locking

- Combine journal_lock and lock.
- Move throttling outside of the lock (this fixes potential deadlock in
  parallel journal mode)
- Make interface nomenclature a bit more helpful

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: do not throttle op_queue in queue_op()

In parallel mode, queue_op is called while holding the journal lock, so it
is not okay to throttle there.  Instead, throttle in the caller.

The throttling still needs improvement, but this at least fixes the locking
problem.

Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: add bloom_filter.hpp to noinst_HEADERs

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

Makefile: Fix VPATH builds

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: osd_types.h: const cleanup

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: don't try to load a PG in a nonexistent pool

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

filestore: simplify apply_transactions

Always use queue_transactions, even in no-journal case.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: PG::trim: fix inverted conditional in assert

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

common: prevent infinite recursion on SIGSEGV

Install SIGSEGV / SIGABORT handlers with sigaction using SA_RESETHAND.
This will ensure that if the signal handler itself encounters another
fault, the default signal handler (usually dump core) will be what is
used. Also, flush the log before dumping core.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: Create pg_split test

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

logger: Fix a crash when the MDS shuts down cleanly.

We weren't holding the lock on the logger_timer before calling shutdown.

Timer: add some asserts to catch certain errors.

Makefile: Add --as-needed to LDFLAGS

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

vstart.sh: don't specify journaling mode

Let the autodetection kick in, or let the dev specify via -o '...'.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: PG::trim: add assert

Assert that we're not trimming the PG log past last_complete.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: _process_pg_info: add assert for replicas

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: dump_missing: also dump missing_loc

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: discover_all_missing fix

Don't request information from an OSD unless it is up and part of the
might_have_unfound set. Add more logging.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

gui: some cleanup

Rather than vectors of pointers, use vectors of NodeInfo structures.
This avoids the problem of freeing the NodeInfo structures.

GuiMonitor::gen_node_info_from_icons: initialize status.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

gui: more reindenting

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

gui: reindent a bunch of code

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

mdcache: in trim_non_auth, only print out path if it has a parent dentry.

This should only occur with the root inode, but caused a segfault for
anybody running more than one MDS who restarted.

Signed-off-by: Greg Farnum <gregf@hq.newdream.net>

mds: Reply checking_lock while reading filelock

Use checking_lock to repalce lock_state in extra buffer list to let client can get correct file lock reply.

client: remove inode from flush_caps list when auth_cap changes

Avoid confusing other code (e.g. kick_flushing_caps) by staying on the mds
flushign_caps list when we don't even have an auth_cap with them anymore.
We'll need to re-flush to a new MDS later.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix set_state_rejoin auth_pin check

We carry an auth pin IFF !stable AND auth.

Signed-off-by: Sage Weil <sage@newdream.net>

init-ceph: tolerate failure in cleanallogs

Otherwise /var/log/ceph/stat makes rm -f error out and we fail.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix recover_replicas() unfound check

missing_loc.count(soid) == 0 only means unfound if it's not missing on the
primary.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: recover_primary() until primary has all found objects

The logic in that if was effectively reversed.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: only discover_all_missing if unfound

Signed-off-by: Sage Weil <sage@newdream.net>

osd: add get_num_unfound() helper

Signed-off-by: Sage Weil <sage@newdream.net>

osd: only search_for_missing if there are unfound objects

Signed-off-by: Sage Weil <sage@newdream.net>

osd: removing unused variable, fix warning

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix is_all_uptodate()

This should only return true when recovery is done, i.e., no more missing
objects.  Nothing to do with unfound.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix PG::is_all_uptodate

In PG::is_all_uptodate, don't try to look for peer_missing[osd->whoami].
The primary keeps that in PG::missing!

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

osd: PG::read_log: don't be clever with lost xattr

Formerly, we had a special case in read_log for dealing with objects
whose objects were present on the disk, but not their attributes. This
conflicts with our plans to mark objects as lost by putting a bit in the
object attributes, since without those attributes, we'll never know if
the objects were formerly marked as lost.

This should almost never happen, and if it does, we just handle the
objects as missing in the normal way.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

Rename  peer_summary_requested to peer_backlog_req

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

Build might_have_unfound set at activation

The might_have_unfound set is used by the primary OSD during recovery.
This set tracks the OSDs which might have unfound objects that the
primary OSD needs. As we receive Missing from each OSD in
might_have_unfound, we will remove the OSD from the set.

When might_have_unfound is empty, we will mark objects as LOST if the
latest version of the object resided on an OSD marked as lost.

Signed-off-by: Colin McCabe <colinm@hq.newdream.net>

monmaptool: Return a non-zero error code and print a useful error
message if unable to read the monmap file.

Signed-off-by: Samuel Just <samuelj@hq.newdream.net>

mds: allow for old fs's with stray instead of stray0

New fs's get stray0, but we want to still behave with old ones.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'testing' into unstable

Conflicts:
configure.ac

v0.23.1