cls_rgw: store more info for each entry, stats categories

rgw: account object removal

rgw: account clone_range dest object

rgw: class init_index

objclass: add stat operation

cls_rgw: list marker uses lower_bound

rgw: implement rgw_rados helpers for class calls

rgw: fix epoch check (greg)

rgw: reuse pools

rgw: fix bucket removal (old scheme)

rgw: fix object removal

rgw: fix bucket listing (still using old scheme)

rgw: rgw_admin -- add bucket marker to bucket stats

pool sharing, initial work

rgw: define rgw class api

rgw: Makefile changes, move around definitions

rgw: make cls_rgw compile

rgw: class for bucket directory

doesn't compile yet

conf: allow ; as a list separator

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-1477'

Make g_conf constant for all conf variable types

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

const-ify integer config values

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: define config vars once in config_opts.h

Rather than defining configuration variables twice, define them once in
config_opts.h. This makes it easier to work with them.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

init-ceph.in: change init script status output

Per Kyle Bader's suggestion: add $name: to the status output.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileStore: use config observers

Use config observers to avoid thread-safety and performance problems in
FileStore.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

cfuse.cc: use apply_changes API

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

testmsgr.cc: use set_val / apply_changes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd/OSD.cc: use set_val / apply_changes API

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

doc: Typo.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Fix double dash handling

Several functions examine argv in order to set options. Only the last
argument parsing pass should remove the '--' from the argument vector.
If it is removed earlier than that, entries may be parsed as options,
when that was not the user's intent.

This changes fixes the common argument parsing loops so that they do not
remove the double dash. It also rearranges some programs so that the
user's argument parsing loop comes last, rather than coming before the
common argument parsing loops.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds/MDS.cc: don't crash on bad injectargs

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

doc: Instructions how to build RPMs.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Say "radosgw" not "rgw".

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Shrinking of MDSes is not supported yet.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile.am: fix test_librbd

osd/OSD.cc: don't crash on incorrect injectargs

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

PG: fix typo in PgPriorSet constructor

down.insert(o) rather than down.insert(0)

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

tools: fix compile

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Get rid of silly unused parameter

Apparently we were passing a string to the function so it could do:
rs = rs;

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

monclient: reopen session on monmap change

If our cur_mon is removed from the monmap, reopen the session.  Do not
call _pick_new_mon() directly or we won't reset state, won't
reauthenticate, etc.

Signed-off-by: Sage Weil <sage@newdream.net>

monclient: use cur_con throughout

We choose a monitor and open a connection in exactly two places.  Use
cur_con and cur_con->get_peer_addr() everywhere else.

Simplify and clean up handle_monmap a bit.

Also add an assert to catch #1534.

Signed-off-by: Sage Weil <sage@newdream.net>

msgr: send_keepalive to a Connection

This avoids the lookup, if we already have a reference.  Mirrors the
send_message() Connection variant.

Signed-off-by: Sage Weil <sage@newdream.net>

test_librbd: remove unsigned/signed comparison

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

librbd: Update .gitignore after moving to gtest.

Commits e2ec946858b26788595b39e9da9dcc6ce9c78f73
and 5cb7b3729bfe139cbf15e9e8508a6661483964fe
changed the names of binaries created, ignore the
new name or "make distcheck" will whine.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

get_*_compat_set: get ctor param ordering right

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

PG: assemble backlog directly rather than queueing on corrupt log

The queue responsible for calling generate_backlog hasn't actually been
started at that stage of startup.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

qa: add test_librbd workunit

Signed-off-by: Sage Weil <sage@newdream.net>

Remove global ctors/dtors for CompatSet

Instead of having global CompatSet objects, just have functions that can
return appropriate CompatSet objects. This avoids global constructor
and destructor ordering issues.

Fixes bug #1512

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados: add conf_parse_env()

With optional env var name, defaults to CEPH_ARGS.

Parse it for librados and librbd api gtests.

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: move c++ tests to gtest

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: convert C tests to gtest

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: rev LIBRBD_VER_EXTRA for rbd_flush() addition

This allows qemu driver to conditionally call it if it exists.

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: implement rbd buffered write window

Normal disks have a write cache and acknowledge writes before they reach
the platter.  Among other things, this masks write latency.  A flush
operation is needed when the user really cares that the writes are stable.

Implement a librbd write window that allows a window including the most
recent N bytes of writes to be immediately acked.  An flush operation
blocks while they are pushed out to disk.

This differs from the typical disk in that writes are always immediately
sent to the backend store, while disks will buffer small writes for a time
(and, in fact, can be made to hold small writes in the cache indefinitely
under certain workloads).

Thus, 'rbd_writeback_window' may be a bit of a misnomer...

Currently this applies only to aio writes, not sync writes.  That could
most easily be fixed by reimplementing write in terms of aio_write.

Signed-off-by: Sage Weil <sage@newdream.net>

client: fix odd crash on rename

If the old_dentry is in the same dir, and it is the last dentry, we need
to keep the dir open.

This is hard to hit because the rename itself will typically instantiate
a null dentry on the target, and it's hard to construct a working where
a racing process makes us drop it.  Fortunately this was triggered
reliably by the snaptest-git-ceph.sh workunit.

Fixes: #1519
Signed-off-by: Sage Weil <sage@newdream.net>

rbd: show progress

Show progress for any rbd _with_progress operations, and for import/export.

Resolves: #1495, #1496
Signed-off-by: Sage Weil <sage@newdream.net>

mon: stray PGInfo is not an error

It is normal to get these immediately after we delete a pool.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix leak of osd_lock on pg lookup vs pool deletion race

Fixes: #1520
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

store_test: fix warning

test/store_test.cc:318: warning: suggest a space before ';' or explicit braces around empty body in 'for' statement

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

PG: generate backlog when confronted with corrupt log

Currently we throw out the log and start up anyway.  With this change, we
would throw out the log, generate a fresh backlog, and then start up.
That may not be the best possible thing, but it's better than what we
currently do.  Indirectly fixes #1502.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

man: Generate manpages from doc/man.

Keeping the generated files in version control lets us
support builds from scratch without requiring the full
documentation toolchain to be installed.

The files were just copied over from build-doc/output/man,
after a ./admin/build-doc call. When redoing this, also
take care to remove any roff output if a file was removed
from doc/man, and update Makefile.am.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Convert manpages to doc/man/.

Includes various content cleanups: layout of synopsis, layout
of comments in examples, typos.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Hide TODO note in rbd.8

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Make rbd.8 "pool-name", like the old man page.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Remove duplicate paragraph from rbd.8

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ReplicatedPG: Fix src hobject_t construction in clone_range call

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

workunit: use sudo when first reading /sys... too

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

workunit: and delete root-owned file

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

workunit: you need sudo to look at /sys/kernel/debug

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge branch 'wip-leaks'

rgw: rgw_admin, init gen_secret, gen_key

debian: no /var/run!

Signed-off-by: Sage Weil <sage@newdream.net>

PG.h: Initialize invalid_hash in Entry()

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

debian: create var/run/ceph

Signed-off-by: Sage Weil <sage@newdream.net>

debian: radosgw: add dirs to package

Signed-off-by: Sage Weil <sage@newdream.net>

ceph: avoid noise when there is no output

Signed-off-by: Sage Weil <sage@newdream.net>

OSD: reset filestore_update_collections to false after do_convertfs

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

object.h: initalize hobject_t correctly

key previously was erroneously initialized to soid.oid.name in one
constructor.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

workunits: make file_layout.sh more robust

We don't have a great way to guarantee mdsmap updates, but they
should happen on their own and we can loop. Closes #1518.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rpm: Include ceph.spec in release tarball.

This lets users say "rpmbuild -ta ceph-XXX.tgz".

Closes: #1521
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rpm: Create the /etc/ceph directory.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rpm: Add newly-added files to %files.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rpm: Build without tcmalloc, by default.

You can still say "rpmbuild -ba --with tcmalloc ceph.spec"
if you have it.

Add BuildRequires on google-perftools when building with tcmalloc.
Ensure it's the right architecture.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ceph: -o - to stdout, not stderr

Don't guess fd numbers.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'master' into wip-leaks

Merge branch 'master' of ceph.newdream.net:/git/ceph

mon: Clean up MonSession/Connection leak on MForwards

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: Clean up PG leak

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: clean up leak on old/unneeded peering messages

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: Sessions get weak pointers to Connections

We were previously setting up a reference loop. But the only way
to get Sessions is via the Connection, so let's just give Sessions
the pointer, and give Connections a counted ref.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

filestore: free fiemap used for testing fs capabilities

No more memory leak!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

WorkQueue: don't deliberately leak memory

We can't do that if we're trying to be Valgrind-clean, so just
make the lock name part of the class.
As best I can tell, that ordered initialization is safe because
data members are initialized in the order they are declared. See eg
http://xenon.arcticus.com/c-morsels-initializer-list-execution-order

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ceph: clean up command output

Open + truncate output file _once_, and concatenate output of each command
to that fd.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: some debugging

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: fix leaked Dir on rename

Need to close old_dentry->dir if the dir is empty.  This was introduced by
ed7ebf35b124a236b88a478c33db82f015a8d4a4.

Fixes: #1509
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: clean up Inode ref counts slightly

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

OSD: info.pgid gets removed from creating_pgs before _create_lock_new_pg

Fix bug introduced in cd2716c4151f5c0996bb4d0cf08961f8da281dac.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rbd: --size is megabytes, not bytes

Signed-off-by: Sage Weil <sage@newdream.net>

osd: take ondisk_read_lock on src_oids

We need to take the ondisk read lock on src oids for multiobject operations
(like clonerange) to ensure that written data has hit disk before we
clone it elsewhere.

Order of acquisition doesn't actually matter here, since the ondisk locks
are all leaves in the lock dependency hierarchy.

Signed-off-by: Sage Weil <sage@newdream.net>

object.h: make hobject key private, prevent objects where key == oid

There should be no difference between an object with a key identical to
its name and an object with the same name but no key.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: make loop exit case explicit, removes warning