journaler: fix requested_pos badness in _issue_read

We want to advance requested_pos *only* as far as we actually want to read,
i.e., the previously-probed end of the journal.

This, among other things, will screw us up later when we reprobe and try to
read more because requested_pos is already past read_pos.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'mds_ino'

Merge remote branch 'origin/rgw'

boto_tool: often use get_bucket rather than lookup

get_bucket throws an exception when the bucket doesn't exist, whereas
lookup just returns None. Sometimes we want the former behavior.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: fix find_ino_dir completion

Pass the path back in so that we can retraverse the path on retry.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: find_ino_dir debugging

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix dir fetch on lookup_hash fix

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix discover_path

If we have the base dirfrag, do not request it.  Otherwise we can get a
reply that contains only that (partial progress), and we will then fail
to wake up our dentry waiter.

This was broken with the rewrite in b58b8d098e4c.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: don't query self from find_ino_peers

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: lookup_ino command

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: fix lookup_hash request hash value

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph_argparse: fix silly usage message

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: typecheck initialization macros

When initializing the config_options array, complain if the size of the
option field we're trying to initialize doesn't match the size of our
type. This will prevent careless type annotations from overwriting
neighboring option fields.

Also create a header called "static assert" which implements a
compile-time assert.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: introduce OPT_U64

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

client: fix lookuphash to use rjenkins

This is the default.  Not ideal, but avoids explicitly specifying the hash
function.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: add LOOKUPINO op

This just looks up a single ino (instead of the ino + dirino/hash
LOOKUPHASH business).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: rename chained _lookup_hash_? methods

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: also try to look up ino directly

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

makefile: include MMDSFindIno[Reply].h

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

journaler: fix comments

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'mds_ino'

mds: try find_ino_dir if find_ino_peers fails for lookuphash

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: find_ino_dir

Search for a dir ino by checking the ondisk directory object path xattr.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

common: fix cmdline of indashes_to_underscores

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: Properly support truncating on S3 PUT

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: use helper to generate dir inode object names

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: allow explicit finisher context for path_traverse

Previously we could only path_traverse and retry a request or message.
This just allows an explicit context to be used as well.  It's the caller's
job to clean it up if we return <= 0.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix possible null dereference

Fix null deref on forward and req (!mdr) caller.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: add mds failure/recovery support for find_ino_peers

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: try find_ino_peers for lookuphash requests

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: find_ino_peer

Search for an ino on peer MDSs.  Still need to handle mds failure/recovery
handling.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: lookuphash

Implement low-level lookuphash on client, for MDS testing.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

vstart.sh: don't put quotes in generated ceph.conf

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ceph_argparse: convert dashes to underscores

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

client: Use NULL instead of 0, for clarity.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

man: Make </VirtualHost> be on its own line.

Apparently, now there's a vertical paragraph space before it,
but that's better than being tacked at the end of the previous
line.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rgw: fix buffer overflow

Remove unecessary code that was causing a buffer overflow.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: recover from partial bucket creates

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: use tmap key (excl) create when creating a bucket

This way we fail with EEXIST if the bucket already exists in the user
object.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

config: remove kill_after

It's unused.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

testlibrbd: fix warnings

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge branch 'next'

testlibrbd: clean up and add tests for snapshots

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: fix snapshot handling

To ensure consistency, always set the snap context when the header is
updated. If snapid is set, we update librados' snapid when refreshing
the header as well. Also use CEPH_NOSNAP instead of 0 as the default
snapid to prevent confusion. These changes fix snapshot creation
and removal, and prevent writing to a snapshot.

Rollback is fixed by using selfmanaged_snapshot_rollback.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados, librbd: add some debugging messages

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados: add selfmanaged_snap_rollback

This was removed in 2cb86f713df38ebee6aa10a81157f99264a59a70, but is
required for selfmanaged snaps because their snapids aren't in the
pool's snap list, which is how regular rollback finds them.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados: return -EROFS when trying to write to a snapshot

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge branch 'next'

Conflicts:
src/cmon.cc
src/osdc/Journaler.cc

mkcephfs: minor fix

line 400: && $type == "osd" is not always valid.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

mkcephfs: copy to daemon nodes for each daemon

The tmp directory is removed after each daemon.  Previously, this would
break if two daemons were on the same node.  Now, the files will be
copied for each daemon.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

journaler: don't block when we adjust back write_pos

is_readable() may need to adjust the write_pos backward, but will return
false.  If we are at the end we still need to wake up any waiters so they
know about it.

Signed-off-by: Sage Weil <sage@newdream.net>

config: parse_argv fix

md_config_t::parse_argv: fold md_config_t::parse_argv_part2 into
parse_argv. Fix brokenness introduced by the std::string switchover.

OPTION macro: move single-character options out of the OPTION macro and
into config.cc

Fix ceph_argparse_witharg / ceph_argparse_flag uses to include a
trailing (char*)NULL, to ensure that we terminate with a pointer rather
than a 32-bit int.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados: print error message on exit

Signed-off-by: Sage Weil <sage@newdream.net>

rados: add 'tmap {set,create} obj key value'

Signed-off-by: Sage Weil <sage@newdream.net>

osd: add tmap key (exclusive) create

Return EEXIST if a key already exists.

Signed-off-by: Sage Weil <sage@newdream.net>

config: fix use of uint64_t* as uint32_t*

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: tighten up uint32_t use

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mkcephfs: copy to daemon nodes for each daemon

The tmp directory is removed after each daemon.  Previously, this would
break if two daemons were on the same node.  Now, the files will be
copied for each daemon.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

common_init: fix keyring loading

Can't pass a NULL char* to std::string constructor.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

common: Add strict_strtof, strict_strtod, unittest

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

OSD: ReplicatedPG: commits should be high priority, like acks.

This was consistently breaking stuff for some people, as the acks were
high priority but the commits weren't. They should match.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

config: use std::string in md_config_t

Use std::string to represent md_config_t strings. This makes memory
management a lot easier and should fix some leaks. "No value" is now
represented by an empty string, whereas before some places were using
empty strings and some were using NULL.

config.cc: Fix a minor decode bug.

In pid_file.cc, copy the pid_file using snprintf, since strncpy
does not always NULL-terminate.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/bufferlist.cc: Added append tests

Tests appending many randomly sized chunks many of which are size zero.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ConfUtils: move parsing into config.cc

Move parsing into config.cc, since there was already parsing code there.
Move metavariable escaping out of ConfUtils; having this in ConfUtils
makes it impossible to de-globalize g_conf.

Create a nicer API for pulling stuff out of the configuration file.
Since the value we pull is determined by the config structure in effect
at the time, it should be an instance method of md_config_t.

Remove some deadcode. Add some comments.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ConfUtils: fix up read API a bit

Remove def_val. Return error codes for not found, parse error, etc.
Fix API users.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ConfUtils: decode to std::string

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mount.ceph: Use LDADD not LDFLAGS to bring in -lkeyutils.

Without this, flab (gcc 4.2.4) failed with:

libtool: link: gcc -Wall -D__CEPH__ -D_FILE_OFFSET_BITS=64 -D_REENTRANT -D_THREAD_SAFE -rdynamic -g -O2 -o mount.ceph mount.ceph.o armor.o  -lkeyutils /usr/lib/libcrypto++.so
/usr/lib/libcrypto++.so: undefined reference to `pthread_key_create'
/usr/lib/libcrypto++.so: undefined reference to `pthread_getspecific'
/usr/lib/libcrypto++.so: undefined reference to `pthread_key_delete'
/usr/lib/libcrypto++.so: undefined reference to `pthread_setspecific'
collect2: ld returned 1 exit status

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

cosd: fix bind to specific address

Do not default to public_addr for cluster, heartbeat messengers.

Signed-off-by: Sage Weil <sage@newdream.net>

mkcephfs: minor fix

line 400: && $type == "osd" is not always valid.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Add an RPM BuildRequires on keyutils-libs-devel.

See 9aab8c6b051469904d7ecd5a9a6dd7aaf5cc3931.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

config: temporary fix for monclient conf issue

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cmon: fix typo

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'stable' into next

Conflicts:
configure.ac
debian/changelog
src/msg/SimpleMessenger.cc

mount.ceph: fix missing includes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add a Debian Build-Dep on libkeyutils-dev.

This is needed by bee85518e2885cc93fe8ca634292ad4846515456.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mount.ceph: Use kernel key management API when possible.

Backwards compatible with older kenrnels, for now.

Fixes: #852
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

common: Make armor.h safe to use from C.

mount.ceph needs to base64-decode the secrets, so we can get rid of
the kernel-side base64 decode, but it doesn't need all of common lib.
And it is written in C.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mount.ceph: Modprobe ceph before trying the mount.

This will be needed for the next few commits, where we try to load the
keys into the kernel; without ceph.ko loaded, the key type will not be
recognized.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mount.ceph: Silently pass through options "name" and "secret".

Fixes: #914
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mount.ceph: Fix strncmp length comparing "secretfile".

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rgw: add bucket to user list prior to creating bucket

This avoids hitting the osd immediately following a bucket creation, which
will avoid any map update latency for most requests.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: create bucket object and xattr in single operation

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librados: add create to ObjectOperation

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objecter: add create to ObjectOperation, known strings

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

SyntheticClient: const cleanup

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

FileStore: convert dev, jdev to std::string

Convert dev, jdev to std::string in preparation for adding std::string
to g_conf.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cmon: add --inject-monmap option

This lets you manually inject a monmap into a down monitor.  This is useful
in cases where you need to change the monmap but aren't able to get a
quorum with the old map.

Signed-off-by: Sage Weil <sage@newdream.net>

mkcephfs: don't clobber osdmap when importing crushmap

Signed-off-by: Sage Weil <sage@newdream.net>

librados: compound ObjectOperation

Wrap the internal ObjectOperation and expose via librados.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

config: refactor conf_set_conf_val a bit

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: be stricter in set_val

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: whitespace, code ordering cleanup

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common: rename old argument parsing macros

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common: more argument parsing into ceph_argparse

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

vstart.sh: Filter out IPv6 and localhost IP addresses.

On e.g. Ubuntu 10.10, hostname --ip-address outputs something
like "::1 10.1.2.3 127.0.1.1", and this makes the generated
config be invalid. Get rid of the entries we can't use.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

common: separate file for strict_strtol

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

objecter: ObjectOperation fixes

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

journaler: remove obsolete journaler_cache thing

This is ancient code used for testing impact of journal IO, no longer
useful.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: remove extra read_acls

msgr: move pidfile handling, signal handing out

Move signal handling and pidfile writing out of the messenger and into
signal.cc and pidfile.cc, respectively.

(Try to) remove our pid file when we crash due to a fatal signal.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>