obsync: implement RadosStore

Implement RadosStore, a storage backend which accesses librados
directly, without going through RGW (Rados GateWay).

This version is still very preliminary because ACLs aren't supported.
We need ACLs even to do things like properly create buckets.
Instead, this version has ACL_HACK, which is just for testing purposes.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

radostool: fix getxattr / setxattr return code

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rbd: make showmapped output a bit prettier

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rbd: showmapped

Show mapped rbd devices.

Fixes: #1024
Signed-off-by: Sage Weil <sage@newdream.net>

pybind-rados: fix Ioctx::close

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test-obsync.py: add tests with --no-preserve-acls

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

objectstore: drop decode support for <= v0.19 encoded transactions

This only affects online upgrade or journal replay of v0.19 generated
transactions.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: fix pg stat summary

Had the pg state counts in there twice.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: undump-journal

Signed-off-by: Sage Weil <sage@newdream.net>

buffer: break out read_fd method

Read N bytes from the provided fd into the bufferlist.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: check if bucket is empty before removing it

obsync: another fix for --no-preserve-acls

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

obsync: fix bug in --no-preserve-acls

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: Don't twiddle lock states in the middle of an import.

This should have gone in a028c8954ca240ec9a12682678aaee02eb507ae3.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge branch 'stable'

Conflicts:
src/mds/MDLog.cc
src/osdc/Journaler.cc
src/osdc/Journaler.h

rgw: send content length on put operation

mds: only move the journaler expire_pos forward

We were seeing weird trim errors because expire_pos was getting moved
backwards after a standby-replay -> replay transition.  Make sure the two
places that update the expire_pos only move it forward--never backward.

Fixes: #1023
Signed-off-by: Sage Weil <sage@newdream.net>

mds: always trim standby segments after rereading the head

When we re-read the head we may get an expire_pos that has moved forward in
time.  That is the appropriate time to trim segments during standby-replay.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: only write head once after expiring logsegments

Signed-off-by: Sage Weil <sage@newdream.net>

mds: small journaler cleanups

Signed-off-by: Sage Weil <sage@newdream.net>

journaler: separate out trimmed_pos setter

Signed-off-by: Sage Weil <sage@newdream.net>

mds: wait for blacklisting osdmap on standby-replay -> replay final pass

Signed-off-by: Sage Weil <sage@newdream.net>

librados python binding: always use 64-bit offsets

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind-rados: fix read API

Read should have the arguments in the same order as pread.
Add a test for reading from a nonzero offset.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

hypertable: update bindings to use new libceph API

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

hadoop: re-add ceph_set_default_preferred_pg

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

libceph: re-add set_default_preferred_pg

Hadoop needs it.

Signed-off-by: Sage Weil <sage@newdream.net>

hadoop: get hadoop bindings to build again

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Client.cc: fix mode/flags confusion

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

libceph: Remove *_precise

In Linux, struct stat includes nanosecond resolution already. No need to
create a separate stat structure that duplicates the fields of struct stat.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

client, libceph: clean up layout methods

Signed-off-by: Sage Weil <sage@newdream.net>

CephFSInterface: don't include Timer.h

We don't use it.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

hadoop: convert to new libceph interface

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

do_autogen: add -H for --with-hadoop

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

do_autogen.sh: don't clear CFLAGS, CXXFLAGS

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

testceph: add more tests

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

libceph: fix some bugs

Fix some bugs in libceph and testceph.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

libceph: avoid DIR*, combine connect and mount

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

hadoop: whitespace fixes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Client: unmount should not be able to fail

If unmount fails, what is the API users supposed to do?
The client needs to be able to clean up after itself.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

libceph API change

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mkcephfs: Fail if there are no mons defined.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mkcephfs: Fail if a mon has no address set.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mkcephfs: Remove temp dir, not just its contents.

Also fixes the case where no temp files were created in the dir.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

journaler: fix flush completion when nothing to flush

Signed-off-by: Sage Weil <sage@newdream.net>

journaler: default to readonly; fix asserts

Previously were were never in a readonly state, which made all the existing
asserts meaningless.

Signed-off-by: Sage Weil <sage@newdream.net>

vstart.sh: fix -s

Signed-off-by: Sage Weil <sage@newdream.net>

mon: consisder osd recovery state in health check

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: include unfound count in pg recovery summary

Fixes: #1003
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: fix up osd health report

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: fix up pg health report

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: generalize health check a bit

Any PaxosService can return a status (or not).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'master' of ssh://ceph.newdream.net/home/sage/ceph.newdream.net/git/ceph

Makefile.am: add -fno-strict-aliasing

Until we audit every place we have a typecast to make sure it's safe,
this should be off.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Makefile.am: Wpointer-arith, Wstrict-null-sentinel

Add Wpointer-arith, Wstrict-null-sentinel, Winit-self

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

v0.27

Signed-off-by: Sage Weil <sage@newdream.net>

clitests: fix osdmap unit test

Broken by removal of casdata in 8cd3f1cd2546a61bb5402fa374de7f37147dc3a0.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: add some new warnings to CXXFLAGS

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: don't daemonize when doing journal reset.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rbd: rename kernel add/remove to map/unmap

Unmap takes a device now, and uses the major number
to find the right sequence number in sysfs.

Also updated the man page.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge remote branch 'origin/next'

rbd: add commands for manipulating the kernel sysfs interface

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

mount.ceph: use common implementations of secret handling and address resolving

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

addr_parsing: use strtok_r to be safe

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

addr_parsing: rename mount_resolve_dest to resolve_addrs and don't modify its argument

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

addr_parsing: move implementation out of header

Also include missing headers and make it linkable with C,
since it's duplicated from mount.ceph.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: use rados' init_with_config because we already called common_init

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

common, mount.ceph: move functions for working with secrets into secret.h

Use sprintf instead of safe_cat, since we're just writing a string once.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge remote branch 'origin/next'

testlibrbd: fix signed/unsigned comparisons

testlibrbd.c: In function 'write_test_data':
testlibrbd.c:191: warning: comparison between signed and unsigned integer expressions
testlibrbd.c: In function 'aio_read_test_data':
testlibrbd.c:207: warning: comparison between signed and unsigned integer expressions
testlibrbd.c: In function 'read_test_data':
testlibrbd.c:222: warning: comparison between signed and unsigned integer expressions

Signed-off-by: Sage Weil <sage@newdream.net>

osd: drop 'casdata' bucket by default

We don't use it for anything.  It just means we create a bunch of useless
PGs.

Signed-off-by: Sage Weil <sage@newdream.net>

cmds: make journal-check mode run in the foreground.

daemonize does a bunch of stuff we don't want to do.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: massage handling of oneshot-replay.

We accidentally broke it in our last round of changes to support
standby-replay; patch it up again.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

OSDMon: make the defaults in reweight-by-utilization work.

an oload <= 100 fails out; set the new default to 120.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

testlibrbd: fix signed/unsigned comparisons

testlibrbd.c: In function 'write_test_data':
testlibrbd.c:191: warning: comparison between signed and unsigned integer expressions
testlibrbd.c: In function 'aio_read_test_data':
testlibrbd.c:207: warning: comparison between signed and unsigned integer expressions
testlibrbd.c: In function 'read_test_data':
testlibrbd.c:222: warning: comparison between signed and unsigned integer expressions

Signed-off-by: Sage Weil <sage@newdream.net>

osd: drop 'casdata' bucket by default

We don't use it for anything.  It just means we create a bunch of useless
PGs.

Signed-off-by: Sage Weil <sage@newdream.net>

auth: const cleanup

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

obysnc: use python2.5 compatible exception syntax

Apparently using "as" in "except" blocks is a new (and not
backwards-compatible) thing in Python 2.6.x. Who knew?

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

PGMonitor: unregister the config obs in ~PGMonitor

Using the new unregister call, unregister the config observer in
~PGMonitor.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: add remove_observer

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

PGMon: use the config observer framework

We now register observers on mon_osd_full_ratio and nearfull_ratio
to update the OSD full/nearfull lists.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ProfLogger: add NULL to KEYS in the config observer.

I believe that without the null pointer it will overrun into
uninitialized memory.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

config: add comment clarifying virtual destructor.

Just in case anybody else is like me and gets confused.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

PGMap: fill in full OSD sets on decode.

Previously these sets were only modified when applying an incremental,
which might have caused problems when restarting monitors. Whoops!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote branch 'origin/classdist'

conf: reopen proflogger on conf change

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

conf: add observer framework

Objects can now register as configuration observers interested in a
subset of the configuration keys. The observers will be told exactly
which keys have changed.

The first user is dout, which now no longer needs the infamous SIGHUP
hack to know when to reopen the config file.

librados: Remove rados_reopen_log, which was basically a means for the
library user to trigger the SIGHUP behavior.

Changes are accumulated and applied all at once by apply_changes. This
function is called as part of common_init, and after every call to
injectargs.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cauthtool: -C not -c in man page

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

dout: make dout a member of md_config_t

Since logging options are per-config, logically DoutStreambuf instances
should also be per-config. This also allows us to eliminate the
"if (uninitialized)" checks at the beginning of every call to dout.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

dout: remove DOUTSB_FLAG_STDOUT

This was used to implement log_to_stdout, but that doesn't exist any
more. Instead of we have log_to_stderr and the other stuff.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

PGMap: add redo_full_sets() function for when ratios change.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osdmon: Clean up debug output on map full flag.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

PGMap: add [near]full_ratio to the Incremental and encoding.

This way these ratios can be distributed by map updates.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

PGMap: always decode pg_remove.

pg_remove has been included for longer than we've had versions
in the struct, so this check for end is useless -- if pg_remove
wasn't encoded we're already broken by decoding the version at
the beginning.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: better debug output on replay completion

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mkcephfs: allow a prebuild osdmap to be specified

Otherwise we'll create one with osdmaptool --createsimple with the default
generic settins.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

cfuse, mds, dout: Do not create "rank" symlinks for log files.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Revert "Revert "autoconf: Complain if tcmalloc is not found.""

This reverts commit 05c281bfa9e9d69ea3d0197590950c8e6845a13a.

This should be okay now.

debian: Handle missing tcmalloc on Debian lenny.

lenny doesn't have a suitable libgoogle-perftools-dev, and
release.sh edits it out of build-deps. Detect that and tell
configure that not having tcmalloc is ok.

This should make 05c281bfa9e9d69ea3d0197590950c8e6845a13a
unnecessary.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: Build without tcmalloc on non-i386/amd64.

This is not strictly needed as of 05c281bfa9e9d69ea3d0197590950c8e6845a13a,
but that reverting is hopefully only temporary.

Without this, with 05c281 undone, non-mainstream architectures
would fail to build.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>