mon: don't take over for a standby-replay

If a standby-replay is laggy we shouldn't "take over" for them (they're
not part of the cluster yet).  They should be removed like a regular
standby.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: improve scatterlog debug msg

mds: clear flush state on rejoin ack

If we sent scatterlock state during rejoin, the auth will send us an inode
base.  Clear scatterlock flush state if that happens.

Fixes: #637
Signed-off-by: Sage Weil <sage@newdream.net>

mds: send any dirty scatterlock state on rejoin

Not just inodes for auth dirfrags, but for any inode with dirty scatterlock
state.  Include the root inode.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: reset dirty->flushing on rejoin scatterflush

Reset dirty/flushing state during rejoin.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: drop debugging cruft

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: make _create_system_file dirty dentries properly

Properly dirty the new dentries so they get written to the directory
objects later on.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix create_mydir_hierarchy to save dir

Mark the dentries dirty so they get saved to disk (they're not journaled!).
This fixes rstat problems on startup, where populate_mydir was recreating
the entries and munging rstats accordingly.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

radosacl: read Ceph configuration file

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cephtool: don't print anything on SIGTERM, SIGINT

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: don't crash on bad export_dir command

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix file_eval on subtree roots

We were going from mix->sync when we shouldn't.  Fix mis-coded filelock
and look at _lock_ state, not inode's!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'origin/dead-code'

osd: fix ref leak on pg when queueing

op_wq.queue() takes a ref; don't take one here too.  We were leaking like
crazy.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: take reference for RepModify

We put in the completions on done; take the matching reference.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix race in op enqueuing

This is never really exposed except when op_threads == 0.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: simplify osd health output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: only log scrub errors (not ok's)

Otherwise we fill up the error log with "foo is ok!" messages.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

TestSnaps: Fix compile error

Introduced in 0e371de79e4047b92317dee353f6d75c2aba4260.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

cfuse: pass through SIGTERM, SIGINT to FUSE

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ceph_fuse_ll_main: restructure error handling

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librbdtest: check for error on out of bounds I/O

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: check for out of bounds I/O in all forms of read/write

This was found by qemu-io tests, which tried to read and write past
the end of an rbd image. The test hung waiting for a completion that
was never scheduled, since it did not check the return value of
rbd_aio_write.

Reported-by: Stefan Hajnoczi <stefanha@gmail.com>
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

cfuse recognizing same xattr subset as kernel client

Hopefully this patch is formatted/signed off correctly.

Signed-off-by: Brian Chrisman <brchrisman@gmail.com>

TestSnaps.cc: allow client id to be overridden

CEPH_CLIENT_ID environment variable if set now overrides the rados
client id for testsnaps

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: utility, not daemon

- we don't want to open a log file
- we don't want to print a stability warning
- we don't want to fork (not that we do, but anyway!)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

CDir: check_rstats will now print out dir stats whenever there's a bug.

Previously it only printed out dir stats at high debug levels, which
meant you could get output of the dentries without seeing what the
totals were supposed to be!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

obsync: preserve ACLs

Preserve ACLs by default. Also add --no-preserve-acls option to disable
this behavior.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

obsync: escape obj names between S3 and local FS

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: futz with stat again.

We really need to fill in the result, even if it's that the object
doesn't exist!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

test/osd: Add latency stat collection to TestSnaps.cc/RadosModel.h

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: fix error in snap_trimmer

newsnaps continued to be used after it was swapped with coi.snaps.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

crush, clitest: Port old test_crushtool.sh to clitest.

Also test --help, while we're at it.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

osd: more futzing with stat

We can get here when the object doesn't exist if the client specifies
may read and may write (in this case, Filer::probe).  Look at the exists
bit in the object context.  If the object is supposed to exist, we should never get ENOENT from stat.

Signed-off-by: Sage Weil <sage@newdream.net>

libceph: Make header usable from plain old C.

Fixes: #986
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

crush: Remove dead code.

The Makefile never integrated with the surrounding automake
setup, and is unused.

The test helper has failed to compile for quite a long time.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

crush: Remove dead code.

Looks like these never went anywhere.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

radosgw: Remove PHP setting from VirtualHost documentation

Signed-off-by: Wido den Hollander <wido@widodh.nl>
Signed-off-by: Sage Weil <sage@newdream.net>

osd: Remove dead code.

RAID4PG has been "temporarily ripped out" for 3.5 years.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

msg, os: Remove dead code.

Looks like these never went anywhere.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

test-obsync.py: add test for --follow-symlinks

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test-obsync: some fixes, be more verbose

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

boto_tool.py: fix rmbucket

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

obsync: add --follow-symlinks

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ReplicatedPG: tolerate ENOENT when statting.

This got broken in commit f842bcf893e42a4cefda07d0bbf86303b17ddcd7
so that on any MDS restart the OSD would die. Bad OSD! Bad!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

qa: Remove fsx workunit.

Covered better by autotest, these days.

common, msg: Remove dead code.

Looks like these never went anywhere.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

crush.old: Remove dead code.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ebofs: Remove dead code.

Nothing has used the EboFS object storage in years.

Fix usage string of dupstore.

FileJournal and Journal are now part of OSD proper, not
EboFS.

The old scripts in src/jobs are probably broken, but
not touching them further right now.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

osbdb: Remove dead code.

Nothing has used the BerkeleyDB-backed object storage in years.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

clitest: Disabled problematic rados test, for now.

Failed on gitbuilder due to odd libtool interaction.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

clitest: Add more simple basic functionality tests.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

librados-config: Distribute the librados-config.8 manpage.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

librados: Stop bundling the now-unused headers.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move atomic.h include from buffer.h to buffer.cc

Fix up other places that failed to include atomic.h even
if they used atomic_t directly.

Fixes: #979
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer, atomic: Explicitly include assert.h where asserts are used.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move buffer::raw* out of the header.

This paves the way for getting librados.hpp clean of atomic_t
leaking to client code.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move buffer accounting into functions, don't expose atomic_t.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

pybind: Distribute the test script, too. Fixes "make distcheck".

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Revert "mds: rip out rename linkmerge support"

This reverts commit d467fbfd1714ef04faf41e604d6a78e33ef7fa10.

The linkmerge stuff _is_ needed for reintegrating multi-linked stray
inodes back into the hierarchy.  E.g.,

 touch a
 ln a b
 rm a

The MDS will want to renamed the stray inode to the b dentry.

Conflicts:

src/mds/Server.cc

Signed-off-by: Sage Weil <sage@newdream.net>

config: complain if --name gives an invalid type

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ConfFile: add support for backslashes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind: install ceph-pybind-test on debug builds

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

conf: small code cleanups

md_config_t: store a ConfFile by value rather than by pointer.

ConfFile::parse_file: negate error when returning errors.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: fix signed/unsigned comp

warning: osd/ReplicatedPG.cc:1208: comparison between signed and unsigned integer expressions

Signed-off-by: Sage Weil <sage@newdream.net>

ConfUtils: new parser

A new parser for ConfUtils that outputs a list of syntax errors when
files are wrong.

It also checks that configuration files are valid utf-8 and don't contain
embedded nulls.

Also added a few more unit tests.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: simplify stat

We don't actually need to stat the object (again) since we have the size
(and mtime) in the object_info_t.  Keep the existing stat in there, tho,
and assert the answer matches for the time being, since I saw behavior
that suggested they were out of sync.  See #933.

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: remove unnecessary includes

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

config: expand metavariables when needed

md_config_t::get_val_from_config_file now has an option that causes it
to expand metavariables. This is necessary because many callers want
metavariables to be expanded in the results they get back.

md_config_t::set_val should expand metavariables.

Remove some unecessary typecasts from
md_config_t::get_val_from_config_file()

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: add ability to complain about parse errs

Change the ConfUtils interface so that we have a way to return
information about parse errors.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

confutils: test unicode parsing

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: fix metavariable substitution

common_init: Do metavariable expansion as a separate step after the
configuration, argv, and env have been read. This ensures that we get
the correct values for the metavariables.

Change librados to follow suit.

config: Enforce the "no metavariables in default values" restriction.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: remove some unecessary g_conf references

Class methods of md_config_t should refer to the enclosing config.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cclass: add bindir option

Ceph binaries don't live in /usr/local on autotest machines.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

client: close snaprealm on last cap removal during reconnect

The invariant is is_any_caps() IFF in->snaprealm, where is_any_caps() is
!caps.empty || exporting_mds >= 0.  If we clear exporting_mds and caps can
be non-empty, we need to drop the snaprealm reference.

Fixes crash like

client/Client.cc: In function 'void Client::add_update_cap(Inode*, int, uint64_t, unsigned int, unsigned int, unsigned int, inodeno_t, int)', in thread '0x7f213df10700'
client/Client.cc: 2244: FAILED assert(in->snaprealm == 0)
client/Client.cc: In function 'void Client::add_update_cap(Inode*, int, uint64_t, unsigned int, unsigned int, unsigned int, inodeno_t, int)', in thread '0x7f213df10700'
client/Client.cc: 2244: FAILED assert(in->snaprealm == 0)
 ceph version 0.25-570-g8ea9380 (commit:8ea938000624b5d67237b52cceee83b04d310d3c)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x53) [0x7bd566]
 2: (Client::add_update_cap(Inode*, int, unsigned long, unsigned int, unsigned int, unsigned int, inodeno_t, int)+0xe6) [0x64af82]
 3: (Client::add_update_inode(InodeStat*, utime_t, int)+0xca2) [0x63e1ec]
 4: (Client::insert_trace(MetaRequest*, utime_t, int)+0x5cf) [0x63f4cb]
 5: (Client::handle_client_reply(MClientReply*)+0x915) [0x6448c7]
 6: (Client::ms_dispatch(Message*)+0x167) [0x644d5b]
 7: (Messenger::ms_deliver_dispatch(Message*)+0x63) [0x62e911]
 8: (SimpleMessenger::dispatch_entry()+0x6fd) [0x61ace1]
 9: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x610b90]
 10: (Thread::_entry_func(void*)+0x23) [0x62d7f3]
 11: (()+0x68ba) [0x7f21418e98ba]
 12: (clone()+0x6d) [0x7f21407e502d]
 ceph version 0.25-570-g8ea9380 (commit:8ea938000624b5d67237b52cceee83b04d310d3c)

triggered by an MDS reconnect on a 3-node cluster.  (Workload in this case
was just csyn --syn makedirs 7 7 7.)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix trim_non_auth assert for base inodes

No parent dentry for base dirfrags (e.g., root).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

MDS: change messenger name for replay mdses

This will cause read operations from standby mdses to be distinguishable
from those from the normal by changing the node name in the messenger.
Previously, the replay node would have the same name as the node it's
following.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

confutils: more parsing tests

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

journaler: fix partial tail entry correction

If we encounter a partial tail entry, we drop it by moving the write_pos
(end of journal) back to read_pos.  We also need to reset the read
state (read_buf, requested/received_pos) so that subsequent replay attempts
won't be horribly confused.

Signed-off-by: Sage Weil <sage@newdream.net>

config: Add confutils unit test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

journaler: fix signedness

Signed-off-by: Sage Weil <sage@newdream.net>

journaler: fix warning

warning: osdc/Journaler.cc:682: comparison between signed and unsigned integer expressions

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix bounds on import

The add_ambiguous_import() call was clobbering the bounds field for
EImportStart::replay(), screwing up the subtree auth adjustment.  Make the
argument const.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: rejoin dirfrags explicitly

We need to rejoin dirfrag replicas explicitly.  We were doing only dentries,
but that won't necessarily include every dirfrag.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: set dir_auth between EImport{Start,Finish}

This ensures that try_trim_non_auth_subtree() doesn't throw out a subtree
we're in the midst of importing (during journal replay).

Signed-off-by: Sage Weil <sage@newdream.net>

mds: use try_trim_non_auth_subtree helper

This helper captures the logic of keeping subtrees when necessary but
dropping them when possible, and cleaning up as appropriate.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: close exported dirfrag

We have to keep export bounds open for auth subtrees.  After we export a
subtree, though, there are two opportunities to drop empty dirfrags from
our cache:

 - The children of the exported subtree may now be trimmable, if they are
   also non-auth and empty.
 - The exported subtree may be trimmable if it is empty and the parent is
   also non-auth.  This may be true for ancestors further up the hierarchy
   as well.

This helps ensure that when we get to rejoin, the only non-auth subtrees we
have are there because they are non-empty or because they are bounds on our
own subtrees.

Signed-off-by: Sage Weil <sage@newdream.net>

MDSMonitor: MDS do not standby replay until other is active

Previously, an mds could go into standby replay before the mds which it
is replaying has finished creating.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

journaler: fix requested_pos badness in _issue_read

We want to advance requested_pos *only* as far as we actually want to read,
i.e., the previously-probed end of the journal.

This, among other things, will screw us up later when we reprobe and try to
read more because requested_pos is already past read_pos.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'mds_ino'

Merge remote branch 'origin/rgw'

boto_tool: often use get_bucket rather than lookup

get_bucket throws an exception when the bucket doesn't exist, whereas
lookup just returns None. Sometimes we want the former behavior.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: fix find_ino_dir completion

Pass the path back in so that we can retraverse the path on retry.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: find_ino_dir debugging

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>