dumpjnl: call msgr->register_entity before start

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

simple_spin: use file-scope global not function

function-scoped globals are protected by a mutex, and taking a mutex
inside a spin lock implementation kind of defeats the point...

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

simple_spin: fix linker error

For some stupid reason the linker is pulling this in twice, resulting in

/bin/sh ../libtool --tag=CXX   --mode=link g++  -Wall -D__CEPH__ -D_FILE_OFFSET_BITS=64 -D_REENTRANT -D_THREAD_SAFE -rdynamic -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -fno-strict-aliasing -Wnon-virtual-dtor -Wno-invalid-offsetof -Wstrict-null-sentinel  -g -Wextra -Wno-missing-field-initializers -Wno-missing-declarations -Wno-unused-parameter -Wl,--as-needed -latomic_ops  -o radosgw_admin radosgw_admin-rgw_fs.o radosgw_admin-rgw_rados.o radosgw_admin-rgw_user.o radosgw_admin-rgw_access.o radosgw_admin-rgw_op.o radosgw_admin-rgw_rest.o radosgw_admin-rgw_rest_os.o radosgw_admin-rgw_rest_s3.o radosgw_admin-rgw_common.o radosgw_admin-rgw_cache.o radosgw_admin-rgw_os.o radosgw_admin-rgw_os_auth.o radosgw_admin-rgw_formats.o radosgw_admin-rgw_log.o rgw_escape.o radosgw_admin-rgw_admin.o librgw.a librados.a -lfcgi -lcurl -lexpat -lpthread -lm -lcrypto++
libcommon.a(libcommon_a-simple_spin.o):(.data+0x8): multiple definition of `foo'
test_trans.o:/home/sage/ceph/src/./common/debug.h:51: first defined here

...which points to some larger linking issue.  However, this fixes it for
now.

Signed-off-by: Sage Weil <sage@newdream.net>

fix the MonClient problems for --with-debug programs.
Still doesn't compile, though.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Refactor MonClient, KeyRing

MonClient should contain a KeyRing and a RotatingKeyRing. All the
MonClient users, except possibly csyn, don't want to manage those
objects themselves.

Don't chdir until after we have opened the KeyRing. If the KeyRing is at
a relative path, a chdir may make it inaccessible. Separate the chdir
function from the daemonize function.

Refactor the cmds argument parsing a little bit. Separate the special
actions from the normal operations of the daemon.

This should allow librados and libceph to support CephX finally! yay!

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

remove g_keyring

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

dout:remove stream from dout_emerg_streams earlier

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

DoutStreambuf: de-globalize dout lock

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

DoutStreambuf: de-globalize emergency logging

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add simple_spin

Add simple spinlock implementation that is safe to use from anywhere.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Revert "cfuse.cc: use safe_write"

This reverts commit e8ac5aa2a4c4e3ce84ed553dbebfb1cccf5679a9.

This commit is just erroneous. It adds checks on a pipe write
for the result and an abort if the write failed. But that's broken
in the desired case where we succeed, block on ceph_fuse_ll_main(),
and the parent process is long-gone by the time we get to this code!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rados_bencher: re-add written objects constraint to read benchmark.

Somehow, in the last major change, the constraints that kept the
bencher from trying to read non-existent objects got removed. Put
a check back in the main bench loop to fix that.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: remove now-erroneous comment

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge branch 'next'

Conflicts:
src/mds/Server.cc

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: Clean up _rename_prepare journaling

This has been broken for a while in terms of journaling
things the MDS isn't auth for. This patch should fix that, and
adds a few asserts to that effect.

Also adds a new not_journaling flag to _rename_prepare
for those cases which call the function and then discard
the bufferlist results.
Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: _rename_prepaer should only journal dest if auth for it

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

uclient: reset flushing_caps on (mds) cap import.

Previously, we could get stuck thinking that we'd flushed caps
(that went to the original MDS, waited on freeze for export,
and then were dropped) without ever telling the auth MDS that we
wanted to do so. This caused hung shutdowns:
1) during shutdown we drop all our caps
2) we get stuck and notice that we have a flushing cap
3) we send cap flush
4) MDS ignores it (I think because actual data already got updated?
and now we don't have the proper caps either)

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

uclient: don't use racy check for uncommitted data.

Previously we used a check for if there were CEPH_CAP_FILE_BUFFER refs,
but that was racy if we had other threads (they could hold caps for
sync writes or something). Instead, see if we have any in-flight
writes or uncommitted objects.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

uclient: call the right function pointer on truncate

fixes 67533e14439e9b to do what it meant to.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: fail out of path_traverse if we have a null dentry.

Previously if we had a null dentry which we were not auth for,
we would go into a loop of discover lookups on that dentry.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rados_sync: add test for temp file deletion, fix

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_sync: in export, download, then rename

Download files to a 'temporary' name and then rename them when they are
complete. If the download gets aborted halfway through, this prevents
rados sync from getting confused next time when it sees the downloaded
file. This is similar to how rsync operates.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Remove unneeded libcrush1 files

mds: journal parents of srci when srcdn is remote

If srcdn is a remote dentry, we will be journaling the src inode to update
the mtime, but we need to ensure the parent dirs are in the metablob.

Fixes: #1132
Signed-off-by: Sage Weil <sage@newdream.net>

journaler: also initialize safe_pos

on reread_head.  Keep consistent across the two methods.

Signed-off-by: Sage Weil <sage@newdream.net>

journaler: fix trim crash after standby-replay -> active

The reread_head method needs to initialize trimming_pos (like read_head
does) or else we get confused later.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: cleanup rename_prepare a bit

Use *srci tmp.

Signed-off-by: Sage Weil <sage@newdream.net>

vstart.sh: turn down debug ms

Signed-off-by: Sage Weil <sage@newdream.net>

mds: pin inode while one renamed_files list

Make sure the inode is pinned while it is on the LogSegment::renamed_files
list.  Avoids a crash when an inode on that list is wrongly trimmed.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: make sure osd paxos is writeable before doing timeouts

The osd paxos machine has to be writeable before we can update it.

Fixes: #1130
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

boto_tool: add get_bucket_acl

Signed-off-by: "Colin McCabe" <colin.mcccabe@dreamhost.com>

Merge branch 'stable' into next

Merge branch 'stable'

dout: use recursive mutex for dout

Using a recursive mutex for dout is desirable because it allows us to
survive situations like this:

> foo() { dout << "foo" << dendl; }
> dout(0) << foo() << dendl;

These cases should be fixed because they produce garbled output.
However, with this change, at least they will not cause deadlock.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

lockdep: fix shadowed global, add printout

Fix a bug that was keeping lockdep from starting. Add a printout that
lets the user know that lockdep is enabled.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Allow embedded '\0' in bufferlists when copying to std::string.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rados_sync: don't hash paths with periods

A period is not such a bad character.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test_rados_tool.sh: test hashed paths

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados export: better name mangling rules, fix test

Introduce a versioning scheme for name mangling, so that we can change
it in the future if we want to.

For names that require no mangling, don't do anything (don't add any
hash at the end, etc.) Make the hash 16 bytes long rather than 8 bytes.

Fix the unit test to take into account the new rules.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_sync: prefix user extended attributes

Start user extended attributes with USER_XATTR_PREFIX.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: implement aio_flush

Implement a per-ioctx flush that blocks until all previously submitted
aio operations on the ioctx are safe.  Each aio gets a sequence number and
is put on a linked list attached to the ioctx.  The flush operation waits
for it to drain to the watermark set when flush is first called.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crushtool: error out if uniform weights vary

Fixes: #1075
Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix ScrubFinalizeWQ::_clear condition

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

debian: depend on libboost-dev >= 1.34

for statechart.  Partially fixes #1124.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: don't leak Connection reference

Signed-off-by: Sage Weil <sage@newdream.net>

osd: ignore old/stale heartbeat messages

If we get heartbeat messages from old epochs from peers that are not
current, drop them and mark the connection down.  Even if they are peers
we _should_ have (because we haven't gotten a notify yet to learn about
a pg we should have but don't yet) we have a newer map epoch and will learn
about them shortly, reopening the connection.

Fixes: #1107
Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix map sharing due to heartbeats

- share the map with the cluster addr
- use the new {note,get}_peer_epoch helpers to do it sanely
- don't share if we're booting; see 818fa33a661

Signed-off-by: Sage Weil <sage@newdream.net>

osd: protect recovery_wq ops with the recovery lock

We were calling recovery_item.remove_myself() without holding the
recoveryWQ::lock. Naughty naughty!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

crushtool: add -v verbose for --test mode

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Add content to obsync package

v0.28.2

hadoop: track Hadoop API changes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

SimpleMessenger: allow multiple calls to shutdown

Fixes a case where radostool crashed on an error shutdown.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

common/Thread.h: const cleanup

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'wip-obsync'

mkcephfs: pass config to osdmaptool

This lets OSDMap::create_simple() see g_conf.osd_pool_default_size when
creating the initial data, metadata, and rbd pools.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

drop useless cm.txt

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: take default pool size from config

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crushtool: update help

Signed-off-by: Sage Weil <sage@newdream.net>

obysnc: rgw target: validate all users

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'wip-obsync'

mon: remove pg_temp mappings when we delete pools

Signed-off-by: Sage Weil <sage@newdream.net>

test-obsync: test sync directly from s3->rgw

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

crushtool: fix --add-item weight being zero when parent bucket(s) created

Signed-off-by: Sage Weil <sage@newdream.net>

obsync: fix bucket creation through rgw target

The rgw: target can now create buckets. Add a test.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'stable'

test-obsync: test big objects, user-defined xattr

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mkcephfs: set rdir for local mon setup

Fixes: #1113
Reported-by: Bernard Grymonpon <bernard@openminds.be>
Signed-off-by: Sage Weil <sage@newdream.net>

init-ceph: ssh

Another bell/whistle.

Signed-off-by: Sage Weil <sage@newdream.net>

obysnc: fix content-type on RGWStore

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test-obsync: compare_directory now compares xattrs

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ceph-pybind-test: test embedded NULLs in data

Test embedded nulls in rados data. Fix a bug in rados.Object.__str__

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

obsync: more fixes for RgwStore

* Fix content-type handling

* add vvprint and use it in Object::equals.

* support RgwStore::prefix

* more tests

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind/rados: correctly return data with NULLs

Correctly handle returning data with embedded NULLs in it.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind/rados.py: throw NoData on ENODATA

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: fix canceled lock attempt

If client tries to lock a file, has to wait, and then cancels the attempt,
the client will send an unlock request to unwind its state.

 - the unlock now removes the waiting lock attempt from the wait list
 - when the lock request retries and finds it is no longer on the wait
   list it will fail.

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/rados.py: rados.Object.key should be string

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

obysnc: RgwStore: make sure destination users exist

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

obsync: fix DST_OWNER

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: return EACCES if acl xattr doesn't exist

obsync: Add boto_retries, remove rgw_store.prefix

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados python bindings: handle xattrs with NULL

Handle extended attributes that contain NULL bytes correctly, rather
than treating everything as zero-terminated C strings.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

PG: fix race in _activate_committed

Previously, _activate_committed would access the osdmap epoch racing
with handle_osd_map's osdmap update.  This would allow a message to be
sent from a replica to the primary tagged with the same epoch as
last_warm_restart, though the event actually occured before
last_warm_restart.  Thus the primary would fail to ignore the event and
transition to crashed.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

RgwStore: fix some ACL issues

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test-obysnc.py: support librgw testing

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Rename RadosStore to RgwStore

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test-obsync: refactor a little bit

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Proper ACL support for rados targets

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: do not shift to EXCL or MIX while rdlocked

There was an old change in file_eval() that was allowing us to switch from
SYNC to MIX or EXCL while there were rdlocks, which either caused lots of
lock thrashing or could (I think) hang things up completely.  This was
from ea10a672, an ancient fix for something related that appears to have
taken out the rdlocked check by accident.

In my tests (one writer, one stat-er), this took things from long stalls
(up to 20 seconds) to very responsive stats.  Yay!

Fixes: #791
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crushtool: clean up add-item a bit; don't add item to same bucket twice

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crushtool: fix remove-item

Scan all buckets instead of doing a tree traverse.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

radosgw_admin: update clitest

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mkcephfs.in: print out usage if no actions given

If the user didn't specify any actions, print out a usage message rather
than silently exiting.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: Fix RGWAccess::init_storage_provider

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mkcephfs: error out on bad usage

Signed-off-by: Sage Weil <sage@newdream.net>

make: fix build for rgw

rgw_admin: clean warning

Merge commit 'origin/master' into rgw-multiuser

rgw_admin: add key create

rgw_admin: subuser and key removal