doc: Say "radosgw" not "rgw".

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Shrinking of MDSes is not supported yet.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile.am: fix test_librbd

osd/OSD.cc: don't crash on incorrect injectargs

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

PG: fix typo in PgPriorSet constructor

down.insert(o) rather than down.insert(0)

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

tools: fix compile

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Get rid of silly unused parameter

Apparently we were passing a string to the function so it could do:
rs = rs;

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

monclient: reopen session on monmap change

If our cur_mon is removed from the monmap, reopen the session.  Do not
call _pick_new_mon() directly or we won't reset state, won't
reauthenticate, etc.

Signed-off-by: Sage Weil <sage@newdream.net>

monclient: use cur_con throughout

We choose a monitor and open a connection in exactly two places.  Use
cur_con and cur_con->get_peer_addr() everywhere else.

Simplify and clean up handle_monmap a bit.

Also add an assert to catch #1534.

Signed-off-by: Sage Weil <sage@newdream.net>

msgr: send_keepalive to a Connection

This avoids the lookup, if we already have a reference.  Mirrors the
send_message() Connection variant.

Signed-off-by: Sage Weil <sage@newdream.net>

test_librbd: remove unsigned/signed comparison

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

librbd: Update .gitignore after moving to gtest.

Commits e2ec946858b26788595b39e9da9dcc6ce9c78f73
and 5cb7b3729bfe139cbf15e9e8508a6661483964fe
changed the names of binaries created, ignore the
new name or "make distcheck" will whine.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

get_*_compat_set: get ctor param ordering right

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

PG: assemble backlog directly rather than queueing on corrupt log

The queue responsible for calling generate_backlog hasn't actually been
started at that stage of startup.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

qa: add test_librbd workunit

Signed-off-by: Sage Weil <sage@newdream.net>

Remove global ctors/dtors for CompatSet

Instead of having global CompatSet objects, just have functions that can
return appropriate CompatSet objects. This avoids global constructor
and destructor ordering issues.

Fixes bug #1512

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: add conf_parse_env()

With optional env var name, defaults to CEPH_ARGS.

Parse it for librados and librbd api gtests.

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: move c++ tests to gtest

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: convert C tests to gtest

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: rev LIBRBD_VER_EXTRA for rbd_flush() addition

This allows qemu driver to conditionally call it if it exists.

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: implement rbd buffered write window

Normal disks have a write cache and acknowledge writes before they reach
the platter.  Among other things, this masks write latency.  A flush
operation is needed when the user really cares that the writes are stable.

Implement a librbd write window that allows a window including the most
recent N bytes of writes to be immediately acked.  An flush operation
blocks while they are pushed out to disk.

This differs from the typical disk in that writes are always immediately
sent to the backend store, while disks will buffer small writes for a time
(and, in fact, can be made to hold small writes in the cache indefinitely
under certain workloads).

Thus, 'rbd_writeback_window' may be a bit of a misnomer...

Currently this applies only to aio writes, not sync writes.  That could
most easily be fixed by reimplementing write in terms of aio_write.

Signed-off-by: Sage Weil <sage@newdream.net>

client: fix odd crash on rename

If the old_dentry is in the same dir, and it is the last dentry, we need
to keep the dir open.

This is hard to hit because the rename itself will typically instantiate
a null dentry on the target, and it's hard to construct a working where
a racing process makes us drop it.  Fortunately this was triggered
reliably by the snaptest-git-ceph.sh workunit.

Fixes: #1519
Signed-off-by: Sage Weil <sage@newdream.net>

rbd: show progress

Show progress for any rbd _with_progress operations, and for import/export.

Resolves: #1495, #1496
Signed-off-by: Sage Weil <sage@newdream.net>

mon: stray PGInfo is not an error

It is normal to get these immediately after we delete a pool.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix leak of osd_lock on pg lookup vs pool deletion race

Fixes: #1520
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

store_test: fix warning

test/store_test.cc:318: warning: suggest a space before ';' or explicit braces around empty body in 'for' statement

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

PG: generate backlog when confronted with corrupt log

Currently we throw out the log and start up anyway.  With this change, we
would throw out the log, generate a fresh backlog, and then start up.
That may not be the best possible thing, but it's better than what we
currently do.  Indirectly fixes #1502.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

man: Generate manpages from doc/man.

Keeping the generated files in version control lets us
support builds from scratch without requiring the full
documentation toolchain to be installed.

The files were just copied over from build-doc/output/man,
after a ./admin/build-doc call. When redoing this, also
take care to remove any roff output if a file was removed
from doc/man, and update Makefile.am.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Convert manpages to doc/man/.

Includes various content cleanups: layout of synopsis, layout
of comments in examples, typos.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Hide TODO note in rbd.8

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Make rbd.8 "pool-name", like the old man page.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Remove duplicate paragraph from rbd.8

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ReplicatedPG: Fix src hobject_t construction in clone_range call

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

workunit: use sudo when first reading /sys... too

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

workunit: and delete root-owned file

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

workunit: you need sudo to look at /sys/kernel/debug

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge branch 'wip-leaks'

rgw: rgw_admin, init gen_secret, gen_key

debian: no /var/run!

Signed-off-by: Sage Weil <sage@newdream.net>

PG.h: Initialize invalid_hash in Entry()

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

debian: create var/run/ceph

Signed-off-by: Sage Weil <sage@newdream.net>

debian: radosgw: add dirs to package

Signed-off-by: Sage Weil <sage@newdream.net>

ceph: avoid noise when there is no output

Signed-off-by: Sage Weil <sage@newdream.net>

OSD: reset filestore_update_collections to false after do_convertfs

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

object.h: initalize hobject_t correctly

key previously was erroneously initialized to soid.oid.name in one
constructor.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

workunits: make file_layout.sh more robust

We don't have a great way to guarantee mdsmap updates, but they
should happen on their own and we can loop. Closes #1518.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rpm: Include ceph.spec in release tarball.

This lets users say "rpmbuild -ta ceph-XXX.tgz".

Closes: #1521
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rpm: Create the /etc/ceph directory.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rpm: Add newly-added files to %files.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rpm: Build without tcmalloc, by default.

You can still say "rpmbuild -ba --with tcmalloc ceph.spec"
if you have it.

Add BuildRequires on google-perftools when building with tcmalloc.
Ensure it's the right architecture.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ceph: -o - to stdout, not stderr

Don't guess fd numbers.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'master' into wip-leaks

Merge branch 'master' of ceph.newdream.net:/git/ceph

mon: Clean up MonSession/Connection leak on MForwards

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: Clean up PG leak

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: clean up leak on old/unneeded peering messages

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: Sessions get weak pointers to Connections

We were previously setting up a reference loop. But the only way
to get Sessions is via the Connection, so let's just give Sessions
the pointer, and give Connections a counted ref.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

filestore: free fiemap used for testing fs capabilities

No more memory leak!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

WorkQueue: don't deliberately leak memory

We can't do that if we're trying to be Valgrind-clean, so just
make the lock name part of the class.
As best I can tell, that ordered initialization is safe because
data members are initialized in the order they are declared. See eg
http://xenon.arcticus.com/c-morsels-initializer-list-execution-order

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ceph: clean up command output

Open + truncate output file _once_, and concatenate output of each command
to that fd.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: some debugging

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: fix leaked Dir on rename

Need to close old_dentry->dir if the dir is empty.  This was introduced by
ed7ebf35b124a236b88a478c33db82f015a8d4a4.

Fixes: #1509
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: clean up Inode ref counts slightly

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

OSD: info.pgid gets removed from creating_pgs before _create_lock_new_pg

Fix bug introduced in cd2716c4151f5c0996bb4d0cf08961f8da281dac.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rbd: --size is megabytes, not bytes

Signed-off-by: Sage Weil <sage@newdream.net>

osd: take ondisk_read_lock on src_oids

We need to take the ondisk read lock on src oids for multiobject operations
(like clonerange) to ensure that written data has hit disk before we
clone it elsewhere.

Order of acquisition doesn't actually matter here, since the ondisk locks
are all leaves in the lock dependency hierarchy.

Signed-off-by: Sage Weil <sage@newdream.net>

object.h: make hobject key private, prevent objects where key == oid

There should be no difference between an object with a key identical to
its name and an object with the same name but no key.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: make loop exit case explicit, removes warning

README: update a bit

Signed-off-by: Sage Weil <sage@newdream.net>

IndexManager: delete index during RemoveOnDelete

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

OSD: use creating_pgs[pgid].history in get_or_create_pg for new pg

If info.pgid is in creating_pgs, we should use the history from
creating_pgs.  The history passed in will be an empty history from a
creation probe in that case.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

client: add inode on IMPORT

If we get an IMPORT and don't have the inode, add it.  This fixes a race
like:

 mds0 -> client .. mknod reply (or similar)
 mds0 -> mds1 .. migrate cap
 mds1 -> client .. send IMPORT
 client <- mds1 .. rx IMPORT, but don't have inode
 client <- mds0 .. rx mknod reply, add it.

With this fix, we add the inode and set up the cap on IMPORT, and when we
get the mknod reply we update the inode immutable fields that aren't
present in the cap message (rdev, symlink target).

Fixes: #1513
Signed-off-by: Sage Weil <sage@newdream.net>

client: factor out flush_cap_releases() helper

Signed-off-by: Sage Weil <sage@newdream.net>

mds: bracket LOCK|AUTH -> PREXLOCK transition with start/finish_locking

Unlike other lock transitions, we are moving to an _unstable_ state and
then taking our (x)lock.  That means that if we don't actually finish for
some reason (lock order changes, request is canceled, whatever) we leave
things in an unstable state--in this case, PREXLOCK, where nothing else
will touch it.

Call cancel_lock in drop_locks (or in acquire_locks when the order changes)
to clean up after an aborted lock attempt.

Original but reproduced (though not easily) by multimds collection task
fsstress_thrash_subtrees.yaml.

Fixes: #1425
Signed-off-by: Sage Weil <sage@newdream.net>

mds: _finish_xlock helper

Factor out _finish_xlock helper to move from XLOCKDONE (or, later,
PREXLOCK) to a non-xlock state.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: mutation: add start/finish locking hooks

Keep track of which lock we are currently working on locking in the
Mutation.  Use helpers to make sure we start/finish locking in a sane
way.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: move Mutation + children out of MDCache.h

Uninline while we're at it.

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: {remove,resize,snap_rollback}_with_progress

C and C++ APIs.  Mirrors the copy_with_progress addition.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: log when we get an error

...before we assert(!err).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph.spec: spec file should be creating /var/run/ceph

Signed-off-by: Brian Chrisman <brchrisman@gmail.com>

osd: nss build of OSD.cc needs CRYPTO_CXXFLAGS since

nss build fails like this:

if /bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I. -I.   \
-D__KERNEL_STRICT_NAMES -Wall -D__CEPH__ -D_FILE_OFFSET_BITS=64 -D_REENTRANT \
-D_THREAD_SAFE -D__STDC_FORMAT_MACROS -D_GNU_SOURCE -rdynamic   -Winit-self \
-Wpointer-arith -fno-strict-aliasing -Wnon-virtual-dtor -Wno-invalid-offsetof \
-Wstrict-null-sentinel   -g -MT OSD.lo -MD -MP -MF ".deps/OSD.Tpo" -c -o OSD.lo `test \
                -f 'osd/OSD.cc' || echo './'`osd/OSD.cc; \
        then mv -f ".deps/OSD.Tpo" ".deps/OSD.Plo"; else rm -f ".deps/OSD.Tpo"; exit \
1; fi  g++ -DHAVE_CONFIG_H -I. -I. -I. -D__KERNEL_STRICT_NAMES -Wall -D__CEPH__ \
-D_FILE_OFFSET_BITS=64 -D_REENTRANT -D_THREAD_SAFE -D__STDC_FORMAT_MACROS \
-D_GNU_SOURCE -rdynamic -Winit-self -Wpointer-arith -fno-strict-aliasing \
-Wnon-virtual-dtor -Wno-invalid-offsetof -Wstrict-null-sentinel -g -MT OSD.lo -MD -MP \
-MF .deps/OSD.Tpo -c osd/OSD.cc  -fPIC -DPIC -o .libs/OSD.o In file included from \
./os/LFNIndex.h:27,  from ./os/HashIndex.h:20,
                 from ./os/FileStore.h:26,
                 from osd/OSD.cc:25:
./common/ceph_crypto.h:44:22: error: pk11pub.h: No such file or directory

Commit d59f34ab made FileStore.h include HashIndex.h, which ultimately
results in OSD.cc including ceph_crypto.h.

Signed-off-by: Jim Schutt <jaschut@sandia.gov>

rgw: fix arg parsing

rbd: fix command option parsing, honor --

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cephtool: add --concise option again

This was accidentally removed in the arg parsing conversion.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: prepare for supporting other meta headers

osdmaptool: fix typo in argument

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

doc: Note that mkcephfs can't do upgrades etc.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: mkcephfs ssh's directly as root@

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

mkcephfs: Config $user is irrelevant when we want to run as root.

It used to just not sudo if $user was empty, on non-ssh do_root_cmd's.
This bug was exposed by commit 47b09e2f0463f6a5ea8cc8e127b012a0cf1ec4af.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Merge branch 'wip-argparse'

Conflicts:
src/rgw/rgw_admin.cc

ceph tool: convert to new-style arg parsing

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ceph_argparse: remove unused macros, functions

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cfuse: use new-style argument parsing stuff

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cmon: use new argument parsing stuff

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cosd.cc: use new argument parsing functions

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

crushtool: use new argument parsing stuff

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_admin: use new argument parsing stuff

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rbd.cc: use new argument parsing stuff

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add ceph_argparse_withlonglong

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

monmaptool, osdmaptool: fix double-dash handling

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>