mds: revoke client leases on replica dn lock

If a replica dentry gets a LOCK message, revoke client leases.  Don't just
wait for them to expire!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix ambiguous check when journaling subtree map

We journal the EImportStart--and become ambiguous--when we set the state
to IMPORT_LOGGINGSTART; the subtree auth becomes ambiguous a few stages
before that.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: fix stalls on racing revocations can cap migrations

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmaptool: fix clitest

flags now unconditionally followed by space.

Signed-off-by: Sage Weil <sage@newdream.net>

formatter: escape quoted strings

Signed-off-by: Sage Weil <sage@newdream.net>

mon: rewrite pgmap json dump using Formatter

Signed-off-by: Sage Weil <sage@newdream.net>

formatter: dump float, simplify pending_string stuff

Signed-off-by: Sage Weil <sage@newdream.net>

osd: dump osdmap in json

Signed-off-by: Sage Weil <sage@newdream.net>

common: add Formatter class

This is based on the RGW class, but

 - uses a stringstream
 - has an additional dump_stream() method that gives you a usable ostream
 - handles object keys properly

We should merge these implementations.

Not sure if either a stringstream or the raw buffer in the rgw class is
ideal.  Probably output should accumulate on a bufferlist so we can avoid
ever reallocating for big dumps.

Signed-off-by: Sage Weil <sage@newdream.net>

admin_socket srv:set F_CLOEXEC on listening socket

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

admin_socket: fix uninitialized accept() param

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

admin_socket.cc: Use older sizeof syntax

Fix compilation for compilers that don't support the
sizeof(type::field_name) syntax.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: socket path includes full socket name

perfcounters: don't output trailing commas in JSON

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add line break after add_key() failure message.

Signed-off-by: Alexandre Oliva <oliva@lsd.ic.unicamp.br>
Signed-off-by: Sage Weil <sage@newdream.net>

mds: more CInode un-inlining

Signed-off-by: Sage Weil <sage@newdream.net>

mds: uninline a few things

Signed-off-by: Sage Weil <sage@newdream.net>

test/perf_counters: more renaming

* more renaming

* common/admin_socket_client.cc: use old-fashioned sizeof construct in
case some compilers can't handler newer syntax

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ceph_context: remove set_module_type

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'wip-proflogger'

Conflicts:
src/common/config.cc
src/common/config.h

Client.cc: rename client_logger -> client_counters

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Rename ProfLogger to Perfcounters: part 2

Part 2: class names, variable names

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Journaler: weaken a few asserts to let it pass errors back up.

Adjust MDLog to handle the possibility of errors appearing.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

C_Gather: fix comment.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Journaler: return errors from probing back to callers.

Adjust Dumper to handle the error codes. The other callers already do
so!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: handle EBLACKLISTED in MDCache::do_file_recover; assert on errors

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Filer: return error codes from probing up to the calling layer.

This is pretty limited; if you get multiple errors in one batch
of probes it'll only return the last one to get sent back (ENOENT is
excluded). But it's better than nothing.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Rename ProfLogger to PerfCounters

Part 1: file names

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

re-enable proflogger test

Re-enable the proflogger test. This time, it doesn't test the basic
socket functionality (that is what test/admin_socket.cc is for).

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: dump is_truncated when listing bucket

test/admin_socket: test message 0

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Move ProfLogger socket code into admin_socket

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: cinode: Add a little debugging and fix an output

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: Drop locks and auth pins when waiting for freezingtree

In most cases we don't end up in this branch because there's an escape
if you already have an auth_pin on the ref in question. But when
you've got snapshots going on, you can process the request, block on
a lock state change, commit a snapshot change to the inode, and then
try to process the request again. On this second attempt, though,
you end up with a different ref which you don't have a pin on.
Deadlock and breakage ensues!
To fix, drop all locks and auth pins when you wait -- either
you don't have any to drop anyway, or you've got them on a previous
version of the inode that isn't useful anymore.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

uclient: correctly initialize mseq in flush_snaps.

Previously we set mseq=0 unconditionally; this was a mistake that
creeped in via bitrot. Instead, set mseq from the auth cap.
This bug meant that any inodes which had gotten migrated between
MDSes would have their flush_snap messages dropped by the MDS.
Resolves #1324.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: mark rmdir with slaves as a master in journal

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix rmdir slave commit, rollback

We need to set the slave commit callback to journal a commit or rollback
as needed.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: simplify journaled subtree_map

We may have subtrees split locally due to migrations that are just getting
started or stopped.  Simplify the map we journal to disk.  Among other
things this makes the replay check simple (it can compare against the
current "live" map).

Signed-off-by: Sage Weil <sage@newdream.net>

mds: add mds_debug_subtrees option

Verify that replayed ESubtreeMap events match what we have.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: set LogEvent _start_off, drop unused _end_off

Signed-off-by: Sage Weil <sage@newdream.net>

mds: witness rmdir when subtrees are on other hosts

If there is an rmdir with an empty subtree on another mds, we need to witness/
journal that on the dirfrag's auth mds so that replay correctly updates the
subtree map.

This is simpler than the rename witnesses (and the link/unlink ones) because
we aren't actually journaling a modification to any actual metadata; it's just
the subtree map that is changing.

The projection of the subtree map update needs work still, but that is also the
case for renames.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-rgw-multithreaded'

rgw: setup as external server is configurable

rgw: clearer logging context

config.cc: fix comments

Fix comments that were reverted to an older version by commit
a51b983a8bbafc455bcd2acd993981ece00abcc7.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

PGMonitor.cc: add force_create_pg command to monitor

ceph pg force_create_pg <pgid> will now reset that pg's status to
creating to handle corner cases where no osd has that pg.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: fix concurrency issue

ProfLogger: add request codes

The client now starts by sending a 4-byte request code.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ProfLogger: use double quotes, add commas

Technically, JSON always uses double quotes, even though Javascript
often uses single quotes for strings. Also add a comma that had been
omitted.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'stable'

mds: clean up file flags to file mode translation

There was some seriously wrong and ancient cruft in there.  open(2)
specifies that one of O_RDONLY, O_WRONLY, and O_RDWR must always be
specified.  Drop all the crazy.

Linux VFS interprets O_WRONLY|O_RDWR as read+write, so we'll do the same.

Signed-off-by: Sage Weil <sage@newdream.net>

ProfLogger: enable by default for daemons

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Put each ProfLogger's data into its own object

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

client: check for invalid file mode

Signed-off-by: Sage Weil <sage@newdream.net>

mds: check for invalid file mode

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: fix i386 compilation

config: fix compilation in certain environments

mon: fix dump_json quoting

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix O_APPEND file mode calculation

Currently

  fd = open("aaa",O_RDWR|O_CREAT|O_APPEND,0666);

gives us fmode of 1 instead of 3.  We should only infer O_WRONLY if no
other mode is explicitly specified.

Reported-by: Fyodor Ustinov <ufm@ufm.su>
Signed-off-by: Sage Weil <sage@newdream.net>

ProfLogger: don't start threads until it's safe

Introduce a new configuration variable, internal_safe_to_start_threads.
This will be set by common_init_finish once it is safe to start threads.
That will trigger callbacks for any configuration observers listening
for this event.

This is used by ProfLogger to hold off on starting the UNIX domain
socket thread until it is safe to do so.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

vstart.sh: use out/proflog.$name

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

proflogger: Fix compiler warnings

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

proflogger: test multiple profloggers

Test multiple profloggers, and removing profloggers.

Add a clear function which removes all profloggers.

Make the order in which profloggers are output to JSON determininstic.

Const cleanup on get() methods.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: use thread pool for rgw work

test/profloger: test fl_avg variables

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/proflogger: test JSON formatting a bit

Test JSON formatting a bit. Use single quotes in our JSON because that
seems to make things easier.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

proflogger: Unlink our UNIX domain sockets on exit

When we're exiting normally, we ought to unlink our UNIX domain sockets.

Also fix a bug in ProfLogThread, and enable some more tests.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mon: pg dump_json

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: radosgw_admin log show dumps parsable json

test/proflogger.cc: read length of message first

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ProfLogger: write out length of message first

Write out the length of the message first, so that it's easier to write
clients. Also, serialize ProfLogger instances to memory rather than
directly to the socket, to avoid blocking while holding one of the
ProfLogger locks.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/proflogger: Add TeardownSetup and SimpleTest

common/ProfLogger.cc: check the sock_path length.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add test/proflogger.cc, fix ProfLogger::init()

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Rework ProfLogger

* Replace existing proflogger config options with "profiling_logger_uri".
This option controls profiling logger sinks.

* ProfLogger: replace file-writing code with code that sends the
information over a UNIX domain socket.

* handle_conf_change is now fully and correctly implemented. We never
read from the md_config_t structure except in this function, so there
are no races. We re-create the thread when the settings change (no need
for SIGHUP, etc.)

* Replace the single big lock with a lock per Proflogger.

* No need for favg any more; just use fset everywhere for floating-point
variables.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mon: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

tools: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

clien: rados: bench: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ProfLoggerBuilder: new API

There are now three different types of variables that can be created:
u64, float, float average.

The distinction between "inc" and "set" variables is gone, since there
will be no clearing of variables in the new Proflogger.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ReplicatedPG: _rollback_to

_delete_head modifies clone_overlap and stats incorrectly for
_rollback_to

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

testsnaps: use self managed snaps and other fixes

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Makefile: librados-config apparently depends on libcrypto++

In newer versions of gcc/ld, librados-config breaks without
the CRYPTO_LIBS.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

gitignore: ignore Eclipse files/folders

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Introduce ProfLoggerBuilder

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ProfLogger: don't return val from inc/set/etc.

Nobody was using those return values.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Remove lingering references to g_conf

These were harmless, but silly.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: fix range dump

mds: fix protocol versions

I screwed this up in a928f767b974cd36f4be6ea618c5cec366d28362.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: multithreaded handling

recalc_op_target must give an error if pool gone

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Makefile: remove dup testrados_list_parallel_SOURCES

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'origin/stable'

Merge remote branch 'origin/wip-1287'

rgw: multiple fixes, prepare main loop for multithreading

rgw: fix compilation for certain architectures

botol_tool: add --xattr argument for PUT

boto_tool can now set extended attributes when performing a PUT
operation.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

boto_tool: add --get-obj-xattr

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_op: mime-encode ugly xattrs in PUT

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_common: remove unused variable

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>