mds: fix _freeze_dir assert for refragment case

The is_freezeable_dir() is true at freeze time but not forever after over
the lifetime of the freeze.  We split later on and _freeze_dir on the new
fragments, so this assertion isn't necessarily true then.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix choose_lock_state() on xlocked object

This crops up on inodes during clientreplay when we reconnect the cap
on the newly created (and still xlocked) object.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: Use syncfs when available

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: create bucket with empty name should return a valid error

rgw: recreation of bucket returns success

unless it was owned by a different user, at which case it
returns -EEXIST.

mds: update rstats on stray dir when you rename over existing inode.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

sample.ceph.conf: add log file and pid file

These really should be included in a sample...

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

vstart.sh: use "log file" instead of "log dir"

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: listing non existent bucket returns NoSuchBucket

Merge remote branch 'origin/mon_mds'

osd: move MAX_CEPH_OBJECT_NAME_LEN into object.h

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: check obj name length to avoid ENAMETOOLONG

Since the object store is ultimately based on ext3, ext4, or btrfs, and
object names ultimately get translated into file names, we need to
impose a corresponding limit on the length of ceph object names.

Otherwise, the "writeback" thread in the FileStore gets ENAMETOOLONG,
and the transaction does not succeed, even though we journalled it.

Perhaps we will extend or eliminate MAX_CEPH_OBJECT_NAME_LEN at some
point by using prehashing or some other technique. Until then, we need
to be sure to check for this.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librbd: don't write to stdout

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Add test_mutate

Add test_mutate, in an effort to track down an objecter bug.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mdsmap: initialize standby_for_rank

This is initialized in MDSMonitor anyway; do so where for completeness.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: simplify mds follow checks

Instead of assigning followers in the last_beacon laggy check loop, do it
at the end, the same way we let standby nodes take over.

This also fixes a bug where a non-standby node (say, up:replay) that used
to be up:standby-replay and has standby_for_rank set gets reset back to
up:standby-replay.

Fixes: #1001
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: simplify mds laggy check

We should never have a laggy standby, so technically this doesn't change
any behavior, but it makes the flow less confusing.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: don't take over for a standby-replay

If a standby-replay is laggy we shouldn't "take over" for them (they're
not part of the cluster yet).  They should be removed like a regular
standby.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rados-tool: use init_with_config interface

Programs that use both librados and common_init should use
init_with_config.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: improve scatterlog debug msg

mds: clear flush state on rejoin ack

If we sent scatterlock state during rejoin, the auth will send us an inode
base.  Clear scatterlock flush state if that happens.

Fixes: #637
Signed-off-by: Sage Weil <sage@newdream.net>

mds: send any dirty scatterlock state on rejoin

Not just inodes for auth dirfrags, but for any inode with dirty scatterlock
state.  Include the root inode.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: reset dirty->flushing on rejoin scatterflush

Reset dirty/flushing state during rejoin.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: drop debugging cruft

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: make _create_system_file dirty dentries properly

Properly dirty the new dentries so they get written to the directory
objects later on.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix create_mydir_hierarchy to save dir

Mark the dentries dirty so they get saved to disk (they're not journaled!).
This fixes rstat problems on startup, where populate_mydir was recreating
the entries and munging rstats accordingly.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

radosacl: read Ceph configuration file

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cephtool: don't print anything on SIGTERM, SIGINT

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: don't crash on bad export_dir command

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix file_eval on subtree roots

We were going from mix->sync when we shouldn't.  Fix mis-coded filelock
and look at _lock_ state, not inode's!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'origin/dead-code'

osd: fix ref leak on pg when queueing

op_wq.queue() takes a ref; don't take one here too.  We were leaking like
crazy.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: take reference for RepModify

We put in the completions on done; take the matching reference.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix race in op enqueuing

This is never really exposed except when op_threads == 0.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: simplify osd health output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: only log scrub errors (not ok's)

Otherwise we fill up the error log with "foo is ok!" messages.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

TestSnaps: Fix compile error

Introduced in 0e371de79e4047b92317dee353f6d75c2aba4260.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

cfuse: pass through SIGTERM, SIGINT to FUSE

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ceph_fuse_ll_main: restructure error handling

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librbdtest: check for error on out of bounds I/O

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: check for out of bounds I/O in all forms of read/write

This was found by qemu-io tests, which tried to read and write past
the end of an rbd image. The test hung waiting for a completion that
was never scheduled, since it did not check the return value of
rbd_aio_write.

Reported-by: Stefan Hajnoczi <stefanha@gmail.com>
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

cfuse recognizing same xattr subset as kernel client

Hopefully this patch is formatted/signed off correctly.

Signed-off-by: Brian Chrisman <brchrisman@gmail.com>

TestSnaps.cc: allow client id to be overridden

CEPH_CLIENT_ID environment variable if set now overrides the rados
client id for testsnaps

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: utility, not daemon

- we don't want to open a log file
- we don't want to print a stability warning
- we don't want to fork (not that we do, but anyway!)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

CDir: check_rstats will now print out dir stats whenever there's a bug.

Previously it only printed out dir stats at high debug levels, which
meant you could get output of the dentries without seeing what the
totals were supposed to be!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

obsync: preserve ACLs

Preserve ACLs by default. Also add --no-preserve-acls option to disable
this behavior.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

obsync: escape obj names between S3 and local FS

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: futz with stat again.

We really need to fill in the result, even if it's that the object
doesn't exist!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

test/osd: Add latency stat collection to TestSnaps.cc/RadosModel.h

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: fix error in snap_trimmer

newsnaps continued to be used after it was swapped with coi.snaps.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

crush, clitest: Port old test_crushtool.sh to clitest.

Also test --help, while we're at it.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

osd: more futzing with stat

We can get here when the object doesn't exist if the client specifies
may read and may write (in this case, Filer::probe).  Look at the exists
bit in the object context.  If the object is supposed to exist, we should never get ENOENT from stat.

Signed-off-by: Sage Weil <sage@newdream.net>

libceph: Make header usable from plain old C.

Fixes: #986
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

crush: Remove dead code.

The Makefile never integrated with the surrounding automake
setup, and is unused.

The test helper has failed to compile for quite a long time.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

crush: Remove dead code.

Looks like these never went anywhere.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

radosgw: Remove PHP setting from VirtualHost documentation

Signed-off-by: Wido den Hollander <wido@widodh.nl>
Signed-off-by: Sage Weil <sage@newdream.net>

osd: Remove dead code.

RAID4PG has been "temporarily ripped out" for 3.5 years.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

msg, os: Remove dead code.

Looks like these never went anywhere.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

test-obsync.py: add test for --follow-symlinks

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test-obsync: some fixes, be more verbose

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

boto_tool.py: fix rmbucket

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

obsync: add --follow-symlinks

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ReplicatedPG: tolerate ENOENT when statting.

This got broken in commit f842bcf893e42a4cefda07d0bbf86303b17ddcd7
so that on any MDS restart the OSD would die. Bad OSD! Bad!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

qa: Remove fsx workunit.

Covered better by autotest, these days.

common, msg: Remove dead code.

Looks like these never went anywhere.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

crush.old: Remove dead code.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ebofs: Remove dead code.

Nothing has used the EboFS object storage in years.

Fix usage string of dupstore.

FileJournal and Journal are now part of OSD proper, not
EboFS.

The old scripts in src/jobs are probably broken, but
not touching them further right now.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

osbdb: Remove dead code.

Nothing has used the BerkeleyDB-backed object storage in years.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

clitest: Disabled problematic rados test, for now.

Failed on gitbuilder due to odd libtool interaction.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

clitest: Add more simple basic functionality tests.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

librados-config: Distribute the librados-config.8 manpage.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

librados: Stop bundling the now-unused headers.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move atomic.h include from buffer.h to buffer.cc

Fix up other places that failed to include atomic.h even
if they used atomic_t directly.

Fixes: #979
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer, atomic: Explicitly include assert.h where asserts are used.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move buffer::raw* out of the header.

This paves the way for getting librados.hpp clean of atomic_t
leaking to client code.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move functions touching buffer::raw internals into buffer.cc

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

buffer: Move buffer accounting into functions, don't expose atomic_t.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

pybind: Distribute the test script, too. Fixes "make distcheck".

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Revert "mds: rip out rename linkmerge support"

This reverts commit d467fbfd1714ef04faf41e604d6a78e33ef7fa10.

The linkmerge stuff _is_ needed for reintegrating multi-linked stray
inodes back into the hierarchy.  E.g.,

 touch a
 ln a b
 rm a

The MDS will want to renamed the stray inode to the b dentry.

Conflicts:

src/mds/Server.cc

Signed-off-by: Sage Weil <sage@newdream.net>

config: complain if --name gives an invalid type

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ConfFile: add support for backslashes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

pybind: install ceph-pybind-test on debug builds

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

conf: small code cleanups

md_config_t: store a ConfFile by value rather than by pointer.

ConfFile::parse_file: negate error when returning errors.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: fix signed/unsigned comp

warning: osd/ReplicatedPG.cc:1208: comparison between signed and unsigned integer expressions

Signed-off-by: Sage Weil <sage@newdream.net>

ConfUtils: new parser

A new parser for ConfUtils that outputs a list of syntax errors when
files are wrong.

It also checks that configuration files are valid utf-8 and don't contain
embedded nulls.

Also added a few more unit tests.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: simplify stat

We don't actually need to stat the object (again) since we have the size
(and mtime) in the object_info_t.  Keep the existing stat in there, tho,
and assert the answer matches for the time being, since I saw behavior
that suggested they were out of sync.  See #933.

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: remove unnecessary includes

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

config: expand metavariables when needed

md_config_t::get_val_from_config_file now has an option that causes it
to expand metavariables. This is necessary because many callers want
metavariables to be expanded in the results they get back.

md_config_t::set_val should expand metavariables.

Remove some unecessary typecasts from
md_config_t::get_val_from_config_file()

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: add ability to complain about parse errs

Change the ConfUtils interface so that we have a way to return
information about parse errors.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

confutils: test unicode parsing

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: fix metavariable substitution

common_init: Do metavariable expansion as a separate step after the
configuration, argv, and env have been read. This ensures that we get
the correct values for the metavariables.

Change librados to follow suit.

config: Enforce the "no metavariables in default values" restriction.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: remove some unecessary g_conf references

Class methods of md_config_t should refer to the enclosing config.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cclass: add bindir option

Ceph binaries don't live in /usr/local on autotest machines.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

client: close snaprealm on last cap removal during reconnect

The invariant is is_any_caps() IFF in->snaprealm, where is_any_caps() is
!caps.empty || exporting_mds >= 0.  If we clear exporting_mds and caps can
be non-empty, we need to drop the snaprealm reference.

Fixes crash like

client/Client.cc: In function 'void Client::add_update_cap(Inode*, int, uint64_t, unsigned int, unsigned int, unsigned int, inodeno_t, int)', in thread '0x7f213df10700'
client/Client.cc: 2244: FAILED assert(in->snaprealm == 0)
client/Client.cc: In function 'void Client::add_update_cap(Inode*, int, uint64_t, unsigned int, unsigned int, unsigned int, inodeno_t, int)', in thread '0x7f213df10700'
client/Client.cc: 2244: FAILED assert(in->snaprealm == 0)
 ceph version 0.25-570-g8ea9380 (commit:8ea938000624b5d67237b52cceee83b04d310d3c)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x53) [0x7bd566]
 2: (Client::add_update_cap(Inode*, int, unsigned long, unsigned int, unsigned int, unsigned int, inodeno_t, int)+0xe6) [0x64af82]
 3: (Client::add_update_inode(InodeStat*, utime_t, int)+0xca2) [0x63e1ec]
 4: (Client::insert_trace(MetaRequest*, utime_t, int)+0x5cf) [0x63f4cb]
 5: (Client::handle_client_reply(MClientReply*)+0x915) [0x6448c7]
 6: (Client::ms_dispatch(Message*)+0x167) [0x644d5b]
 7: (Messenger::ms_deliver_dispatch(Message*)+0x63) [0x62e911]
 8: (SimpleMessenger::dispatch_entry()+0x6fd) [0x61ace1]
 9: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x610b90]
 10: (Thread::_entry_func(void*)+0x23) [0x62d7f3]
 11: (()+0x68ba) [0x7f21418e98ba]
 12: (clone()+0x6d) [0x7f21407e502d]
 ceph version 0.25-570-g8ea9380 (commit:8ea938000624b5d67237b52cceee83b04d310d3c)

triggered by an MDS reconnect on a 3-node cluster.  (Workload in this case
was just csyn --syn makedirs 7 7 7.)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix trim_non_auth assert for base inodes

No parent dentry for base dirfrags (e.g., root).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>