Merge branch 'fix32'

On i386, pthread ids are 8 hexdigits. Adjust cli tests.

msg/SimpleMessenger.c: fix signed/unsigned compare

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

os/FileStore: fix signed/unsigned comparison warn

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

os/FileJournal: compare like types in MAX()

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

tools/gui.cc: fix signed/unsigned comparison

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

make:add messages/MOSDRepScrub.h to NOINST_HEADERS

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'stable'

Conflicts:
src/osd/OSD.cc

Merge remote branch 'origin/rep_scrub_wq' into stable

osd: discard scrub reply if pg changed

build_scrub_map will bail out if the pg changed.  Discard the result in
that case since the primary will ignore it anyway.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: avoid map_lock for scrub_map reply

Using osd->osdmap->epoch without map_lock is dangerous.  We can avoid it
entirely by replying on the same connection as the request.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: never rewrite log after {advance,activate}_map

pg->dirty_log is never true, so this is dead code.  And nothing in either
of those two methods updates the pg log.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: always write backlog after creation

dirty_log is never set to true, so we would set the log.backlog flag but
not write it to disk.  If we restarted the OSD, we would think we had the
backlog in the log but in reality we would not.  clean_up_local() could
then erase almost every object in the PG.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix no missing inferance

Add missing continue in last_update==last_complete (no missing) case.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

common_init: don't force profiling_logger = true

common_init: don't force profiling_logger = true for daemons. Users can
always set "profiling logger = true" when they want to activate it.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common: Rename Logger to ProfilingLogger

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common: combine set_fg_logging into common_init

Combine set_fg_logging into common_init. This eliminates the problem
where we briefly try to apply the logging settings from the config file
even in a program that needs to set foreground logging.

TestTimers, TestSignalHandlers, TestDoutStreambuf: don't initialize ceph
authentication. These applications don't need it.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

gitignore: ignore massif.out.*

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

config: kill more unused options

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

cfuse: remove high level fuse interface

Untested, unused, and inferior to the lowlevel implementation.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

PG: remove sub_op_scrub

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: switch _request_scrub_map to send MOSDRepScrub

Also switches sub_op_scrub_reply to sub_op_scrub_map to handle the
OSD_OP_SCRUB_MAP response.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

OSD: Adds handler for MOSDRepScrub

handle_rep_scrub enqueues the message in rep_scrub_wq.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: added replica_scrub

Adds handler in PG for MOSDRepScrub messages.  replica_scrub will
replace sub_op_scrub.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

OSD: Add rep_scrub_wq

Previously, replica scrubs would be handled in sub_op_scrub in the op
queue.  Replica scrubs will now be processed by rep_scrub_wq using the
disk tp.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rados: Adds CEPH_OSD_OP_SCRUB_MAP sub op

Previously, maps were requested with a sub_op and sent with a
sub_op_reply.  As maps will now be requested using a different message,
replicas will transmit scrub maps requested via MOSDRepScrub messages by
sending a sub_op of type CEPH_OSD_OP_SCRUB_MAP.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

MOSDRepScrub: Adds a message for initiating a replica scrub

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

common: call closelog before openlog

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

vstart.sh: -k doesn't take an argument

-k doesn't take an argument, so don't call shift a second time. (It's
already called once at the end of the loop.)

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: call openlog to set ident

Set the 'ident' field in our syslog output.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

dout: force reopen after conf read, fg logging set

After the configuration is read, or a program sets foreground logging,
force those changes to be applied to the dout() configuration.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mkcephfs: fix bug in multi-node setups

mkcephfs needs to create $TDIR on the other nodes in a multi-node setup
before trying to scp files into that directory.

Also, do mkdir -p on the directory where the configuration is going to
go.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

safe_io: move to C

Fixes unistd vs XOPEN_SOURCE define weirdness.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph: use new safe_read wrapper

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

add safe_{read/write} wrappers

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

build_debian_image: include bind9-host

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

build_debian_image: fixes

- /dev/pts
- copy console helper scripts
- run from any directory

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filejournal: fix return code check

pwrite() returns number of bytes, or -1.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: test for proper i_complete handling

This reproduces some buggery with rename vs I_COMPLETE on the kernel
client.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

build_debian_image: include libcrypto++ (need for runtime)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

kernel console debug output setup scripts

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

script: add script to build uml images

Set up a tap interface in your bridge, and then

./linux ubda=image mem=512M eth0=tuntap,tap0 ip=<ip> netmask=<netmask> gw=<gw>

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

remove old script

remove ancient web content

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: ignore mds boot messages with zeroed port

On 0.24.2 I saw a zeroed port in the cmds log and in the mdsmap.  Ignore
anything from a cmds with a zeroed port to prevent the insanity from
spreading.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: more carefully gaurd local cache truncate

This fixes an assert when len=0 in file_to_extents when we get some weird
metadata from the MDS.

Fixes: #778
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: make remove_pid_file fail silently if nothing to remove

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

remove ancient mds_local_osd option

This used to add a few osds to the end of the osdmap, not included in the
crush map, with the assumption that they would be colocated with the MDSs
and used as the primary MDS journal copy.  Not a useful hack anymore; you
can do this all explicitly by setting up your own crush map.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

fakemessenger: remove fakemessenger and related dead code

FakeMessenger has hopelessly drifted from the current abstract interface;
all of this code is years old.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

clock: remove tare function

This was used way back when to set a clock's relative "zero" time to
simplify stat collection when running performance benchmarks.  It has no
use in a real system.  If we need relative time for something, we can
subtract off the zero time in the output.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

config: remove (most) unused options

Left in a few that may be used later:

client_unmount_timeout
osd_shed_reads*
osd_min_raid_width
osd_recovery_forget_lost_objects

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: set lru midpoint

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

logger: don't calculate variance by default

Less memory, less CPU.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: check return values

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: check return values for write_pid_file

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: SimpleMessenger::start: fix unchecked return

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

msgr: remove deadcode, warnings

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

msgr: remove_pid_file: fix unchecked returns

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

os: FileJournal::do_write: fix error handling

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'wannoying2'

Conflicts:
src/os/FileJournal.cc

filestore: handle read_op_seq errors

Fixes: #775
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: Return error code when a write is too large.

This should prevent us from crashing the OSD by trying
to pass in a write that is larger than its journal can handle.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

config: add new osd_max_write_size option.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

assert.h: Add ceph_abort

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

os: _check_disk_write_cache: fix error handling

Fix error handling; use popen instead of a temporary file.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cmds: check return code of chdir

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

unittest_encoding: build without -fstrict-aliasing

unittest_encoding gets "this breaks strict-aliasing" warnings when
compiled under Ubuntu. So don't crank up optimization on the unit test.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

FileStore::mount: fix error handling

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

os: fix zeroing first chunk of FileJournal

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

os: FileStore::mkjournal: fix error handling

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

os: FileJournal::create: fix error handling

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cmon: check return code of chdir

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

tools/common: EINTR/unchecked return fix

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados: fix pedantic gcc warning on writing stdout

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

filejournal: fix unchecked returns, modularize

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

signal: fix redefine warnings

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ReplicatedPG: snap_trimmer fix leaked lock

Previous patch 7a02070b741d3482ff6b28827c1eb274a2134486 leaks the pg
lock.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG:snap_trimmer should return if !clean or !active or !primary

The PG may become !clean or !active while in the osd snap_trim_wq.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

cosd: check chdir return code

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: Ager: remove Ebofs leftovers

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add do_autogen.sh

It's handy when making builds with various different warning levels.
Also saves you from having to remember --prefix, etc. all the time.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mkcephfs: clean up temp files, honor $TEMPDIR

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

filer: fix probe

ENOENT is allowed and expected at the end of a journal.  It is treated the
same as a zero size object.

Fixes 800507e891170b99cae10a9bf7c5166fd3512927.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client/fuse_ll.c: const cleanup

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Revert "client/fuse_ll: fully initialize callback struct"

Older build environments may not have all the fuse function ptrs that
are present in newer ones. It's best to just let them all silently be
set to 0.

This reverts commit 516ac99bb8c16614757f3b61f4bf266e4531861e.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Revert "client/fuse.cc: explicity zero fuse function ptrs"

Older build environments may not have all the fuse function ptrs that
are present in newer ones. It's best to just let them all silently be
set to 0.

This reverts commit 38b50013e27e79dd43c5f514c8c24bd8b1e69e5f.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'master' of ssh://ceph.newdream.net/git/ceph

filestore: fix wipe_subvol the rest of the way

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: fix (part of) wipe_subvol

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: add nonce parsing to entity_addr_t::parse()

Reported-by: John Leach <john@brightbox.co.uk>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

os: FileStore::wipe_subvol: fix removing old dir

Older versions of btrfs which don't have BTRFS_IOC_SNAP_DESTROY need to
manually remove snapshots.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cosd.cc: better error message if osd->init fails

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

auth/ExportControl: fix allow_def leak

allow_def would be leaked if the first branch is not taken.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'stable'

Conflicts:
src/common/signal.cc
src/os/FileStore.cc

mds: make SETDIRLAYOUT start with ancestor policy

When we set a new policy on a directory, make the starting point the
current policy, as specified by an ancestor.  Remove any reference to
g_default_file_layout.

This aligns the code with the inode creation paths (the other time we
look at the layout policy).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wannoying'

MonMap: add,remove,contains can take const references

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

SimpleMessenger: learned_addr can take a const ref

peer_addr_for_me can be passed as a const ref.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

mds: handle_client_setdirlayout: init fl_pg_pool

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: print out errs in do_import, check EINTR

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>