cfuse recognizing same xattr subset as kernel client

Hopefully this patch is formatted/signed off correctly.

Signed-off-by: Brian Chrisman <brchrisman@gmail.com>

TestSnaps.cc: allow client id to be overridden

CEPH_CLIENT_ID environment variable if set now overrides the rados
client id for testsnaps

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: utility, not daemon

- we don't want to open a log file
- we don't want to print a stability warning
- we don't want to fork (not that we do, but anyway!)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

CDir: check_rstats will now print out dir stats whenever there's a bug.

Previously it only printed out dir stats at high debug levels, which
meant you could get output of the dentries without seeing what the
totals were supposed to be!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

obsync: preserve ACLs

Preserve ACLs by default. Also add --no-preserve-acls option to disable
this behavior.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

obsync: escape obj names between S3 and local FS

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: futz with stat again.

We really need to fill in the result, even if it's that the object
doesn't exist!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

test/osd: Add latency stat collection to TestSnaps.cc/RadosModel.h

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: fix error in snap_trimmer

newsnaps continued to be used after it was swapped with coi.snaps.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd: more futzing with stat

We can get here when the object doesn't exist if the client specifies
may read and may write (in this case, Filer::probe).  Look at the exists
bit in the object context.  If the object is supposed to exist, we should never get ENOENT from stat.

Signed-off-by: Sage Weil <sage@newdream.net>

libceph: Make header usable from plain old C.

Fixes: #986
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

radosgw: Remove PHP setting from VirtualHost documentation

Signed-off-by: Wido den Hollander <wido@widodh.nl>
Signed-off-by: Sage Weil <sage@newdream.net>

test-obsync.py: add test for --follow-symlinks

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test-obsync: some fixes, be more verbose

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

boto_tool.py: fix rmbucket

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

obsync: add --follow-symlinks

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ReplicatedPG: tolerate ENOENT when statting.

This got broken in commit f842bcf893e42a4cefda07d0bbf86303b17ddcd7
so that on any MDS restart the OSD would die. Bad OSD! Bad!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

clitest: Disabled problematic rados test, for now.

Failed on gitbuilder due to odd libtool interaction.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

clitest: Add more simple basic functionality tests.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

librados-config: Distribute the librados-config.8 manpage.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

librados: Stop bundling the now-unused headers.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move atomic.h include from buffer.h to buffer.cc

Fix up other places that failed to include atomic.h even
if they used atomic_t directly.

Fixes: #979
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer, atomic: Explicitly include assert.h where asserts are used.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move buffer::raw* out of the header.

This paves the way for getting librados.hpp clean of atomic_t
leaking to client code.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move buffer accounting into functions, don't expose atomic_t.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

pybind: Distribute the test script, too. Fixes "make distcheck".

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Revert "mds: rip out rename linkmerge support"

This reverts commit d467fbfd1714ef04faf41e604d6a78e33ef7fa10.

The linkmerge stuff _is_ needed for reintegrating multi-linked stray
inodes back into the hierarchy.  E.g.,

 touch a
 ln a b
 rm a

The MDS will want to renamed the stray inode to the b dentry.

Conflicts:

src/mds/Server.cc

Signed-off-by: Sage Weil <sage@newdream.net>

config: complain if --name gives an invalid type

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ConfFile: add support for backslashes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind: install ceph-pybind-test on debug builds

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

conf: small code cleanups

md_config_t: store a ConfFile by value rather than by pointer.

ConfFile::parse_file: negate error when returning errors.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: fix signed/unsigned comp

warning: osd/ReplicatedPG.cc:1208: comparison between signed and unsigned integer expressions

Signed-off-by: Sage Weil <sage@newdream.net>

ConfUtils: new parser

A new parser for ConfUtils that outputs a list of syntax errors when
files are wrong.

It also checks that configuration files are valid utf-8 and don't contain
embedded nulls.

Also added a few more unit tests.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: simplify stat

We don't actually need to stat the object (again) since we have the size
(and mtime) in the object_info_t.  Keep the existing stat in there, tho,
and assert the answer matches for the time being, since I saw behavior
that suggested they were out of sync.  See #933.

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: remove unnecessary includes

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

config: expand metavariables when needed

md_config_t::get_val_from_config_file now has an option that causes it
to expand metavariables. This is necessary because many callers want
metavariables to be expanded in the results they get back.

md_config_t::set_val should expand metavariables.

Remove some unecessary typecasts from
md_config_t::get_val_from_config_file()

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: add ability to complain about parse errs

Change the ConfUtils interface so that we have a way to return
information about parse errors.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

confutils: test unicode parsing

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: fix metavariable substitution

common_init: Do metavariable expansion as a separate step after the
configuration, argv, and env have been read. This ensures that we get
the correct values for the metavariables.

Change librados to follow suit.

config: Enforce the "no metavariables in default values" restriction.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: remove some unecessary g_conf references

Class methods of md_config_t should refer to the enclosing config.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cclass: add bindir option

Ceph binaries don't live in /usr/local on autotest machines.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

client: close snaprealm on last cap removal during reconnect

The invariant is is_any_caps() IFF in->snaprealm, where is_any_caps() is
!caps.empty || exporting_mds >= 0.  If we clear exporting_mds and caps can
be non-empty, we need to drop the snaprealm reference.

Fixes crash like

client/Client.cc: In function 'void Client::add_update_cap(Inode*, int, uint64_t, unsigned int, unsigned int, unsigned int, inodeno_t, int)', in thread '0x7f213df10700'
client/Client.cc: 2244: FAILED assert(in->snaprealm == 0)
client/Client.cc: In function 'void Client::add_update_cap(Inode*, int, uint64_t, unsigned int, unsigned int, unsigned int, inodeno_t, int)', in thread '0x7f213df10700'
client/Client.cc: 2244: FAILED assert(in->snaprealm == 0)
 ceph version 0.25-570-g8ea9380 (commit:8ea938000624b5d67237b52cceee83b04d310d3c)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x53) [0x7bd566]
 2: (Client::add_update_cap(Inode*, int, unsigned long, unsigned int, unsigned int, unsigned int, inodeno_t, int)+0xe6) [0x64af82]
 3: (Client::add_update_inode(InodeStat*, utime_t, int)+0xca2) [0x63e1ec]
 4: (Client::insert_trace(MetaRequest*, utime_t, int)+0x5cf) [0x63f4cb]
 5: (Client::handle_client_reply(MClientReply*)+0x915) [0x6448c7]
 6: (Client::ms_dispatch(Message*)+0x167) [0x644d5b]
 7: (Messenger::ms_deliver_dispatch(Message*)+0x63) [0x62e911]
 8: (SimpleMessenger::dispatch_entry()+0x6fd) [0x61ace1]
 9: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x610b90]
 10: (Thread::_entry_func(void*)+0x23) [0x62d7f3]
 11: (()+0x68ba) [0x7f21418e98ba]
 12: (clone()+0x6d) [0x7f21407e502d]
 ceph version 0.25-570-g8ea9380 (commit:8ea938000624b5d67237b52cceee83b04d310d3c)

triggered by an MDS reconnect on a 3-node cluster.  (Workload in this case
was just csyn --syn makedirs 7 7 7.)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix trim_non_auth assert for base inodes

No parent dentry for base dirfrags (e.g., root).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

MDS: change messenger name for replay mdses

This will cause read operations from standby mdses to be distinguishable
from those from the normal by changing the node name in the messenger.
Previously, the replay node would have the same name as the node it's
following.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

confutils: more parsing tests

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

journaler: fix partial tail entry correction

If we encounter a partial tail entry, we drop it by moving the write_pos
(end of journal) back to read_pos.  We also need to reset the read
state (read_buf, requested/received_pos) so that subsequent replay attempts
won't be horribly confused.

Signed-off-by: Sage Weil <sage@newdream.net>

config: Add confutils unit test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

journaler: fix signedness

Signed-off-by: Sage Weil <sage@newdream.net>

journaler: fix warning

warning: osdc/Journaler.cc:682: comparison between signed and unsigned integer expressions

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix bounds on import

The add_ambiguous_import() call was clobbering the bounds field for
EImportStart::replay(), screwing up the subtree auth adjustment.  Make the
argument const.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: rejoin dirfrags explicitly

We need to rejoin dirfrag replicas explicitly.  We were doing only dentries,
but that won't necessarily include every dirfrag.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: set dir_auth between EImport{Start,Finish}

This ensures that try_trim_non_auth_subtree() doesn't throw out a subtree
we're in the midst of importing (during journal replay).

Signed-off-by: Sage Weil <sage@newdream.net>

mds: use try_trim_non_auth_subtree helper

This helper captures the logic of keeping subtrees when necessary but
dropping them when possible, and cleaning up as appropriate.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: close exported dirfrag

We have to keep export bounds open for auth subtrees.  After we export a
subtree, though, there are two opportunities to drop empty dirfrags from
our cache:

 - The children of the exported subtree may now be trimmable, if they are
   also non-auth and empty.
 - The exported subtree may be trimmable if it is empty and the parent is
   also non-auth.  This may be true for ancestors further up the hierarchy
   as well.

This helps ensure that when we get to rejoin, the only non-auth subtrees we
have are there because they are non-empty or because they are bounds on our
own subtrees.

Signed-off-by: Sage Weil <sage@newdream.net>

MDSMonitor: MDS do not standby replay until other is active

Previously, an mds could go into standby replay before the mds which it
is replaying has finished creating.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

journaler: fix requested_pos badness in _issue_read

We want to advance requested_pos *only* as far as we actually want to read,
i.e., the previously-probed end of the journal.

This, among other things, will screw us up later when we reprobe and try to
read more because requested_pos is already past read_pos.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'mds_ino'

Merge remote branch 'origin/rgw'

boto_tool: often use get_bucket rather than lookup

get_bucket throws an exception when the bucket doesn't exist, whereas
lookup just returns None. Sometimes we want the former behavior.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: fix find_ino_dir completion

Pass the path back in so that we can retraverse the path on retry.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: find_ino_dir debugging

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix dir fetch on lookup_hash fix

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix discover_path

If we have the base dirfrag, do not request it.  Otherwise we can get a
reply that contains only that (partial progress), and we will then fail
to wake up our dentry waiter.

This was broken with the rewrite in b58b8d098e4c.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: don't query self from find_ino_peers

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: lookup_ino command

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: fix lookup_hash request hash value

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph_argparse: fix silly usage message

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: typecheck initialization macros

When initializing the config_options array, complain if the size of the
option field we're trying to initialize doesn't match the size of our
type. This will prevent careless type annotations from overwriting
neighboring option fields.

Also create a header called "static assert" which implements a
compile-time assert.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: introduce OPT_U64

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

client: fix lookuphash to use rjenkins

This is the default.  Not ideal, but avoids explicitly specifying the hash
function.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: add LOOKUPINO op

This just looks up a single ino (instead of the ino + dirino/hash
LOOKUPHASH business).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: rename chained _lookup_hash_? methods

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: also try to look up ino directly

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

makefile: include MMDSFindIno[Reply].h

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

journaler: fix comments

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'mds_ino'

mds: try find_ino_dir if find_ino_peers fails for lookuphash

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: find_ino_dir

Search for a dir ino by checking the ondisk directory object path xattr.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

common: fix cmdline of indashes_to_underscores

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: Properly support truncating on S3 PUT

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: use helper to generate dir inode object names

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: allow explicit finisher context for path_traverse

Previously we could only path_traverse and retry a request or message.
This just allows an explicit context to be used as well.  It's the caller's
job to clean it up if we return <= 0.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix possible null dereference

Fix null deref on forward and req (!mdr) caller.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: add mds failure/recovery support for find_ino_peers

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: try find_ino_peers for lookuphash requests

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: find_ino_peer

Search for an ino on peer MDSs.  Still need to handle mds failure/recovery
handling.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: lookuphash

Implement low-level lookuphash on client, for MDS testing.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

vstart.sh: don't put quotes in generated ceph.conf

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ceph_argparse: convert dashes to underscores

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

client: Use NULL instead of 0, for clarity.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

man: Make </VirtualHost> be on its own line.

Apparently, now there's a vertical paragraph space before it,
but that's better than being tacked at the end of the previous
line.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rgw: fix buffer overflow

Remove unecessary code that was causing a buffer overflow.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: recover from partial bucket creates

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: use tmap key (excl) create when creating a bucket

This way we fail with EEXIST if the bucket already exists in the user
object.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>