mds: Drop locks and auth pins when waiting for freezingtree

In most cases we don't end up in this branch because there's an escape
if you already have an auth_pin on the ref in question. But when
you've got snapshots going on, you can process the request, block on
a lock state change, commit a snapshot change to the inode, and then
try to process the request again. On this second attempt, though,
you end up with a different ref which you don't have a pin on.
Deadlock and breakage ensues!
To fix, drop all locks and auth pins when you wait -- either
you don't have any to drop anyway, or you've got them on a previous
version of the inode that isn't useful anymore.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

uclient: correctly initialize mseq in flush_snaps.

Previously we set mseq=0 unconditionally; this was a mistake that
creeped in via bitrot. Instead, set mseq from the auth cap.
This bug meant that any inodes which had gotten migrated between
MDSes would have their flush_snap messages dropped by the MDS.
Resolves #1324.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: mark rmdir with slaves as a master in journal

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix rmdir slave commit, rollback

We need to set the slave commit callback to journal a commit or rollback
as needed.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: simplify journaled subtree_map

We may have subtrees split locally due to migrations that are just getting
started or stopped.  Simplify the map we journal to disk.  Among other
things this makes the replay check simple (it can compare against the
current "live" map).

Signed-off-by: Sage Weil <sage@newdream.net>

mds: add mds_debug_subtrees option

Verify that replayed ESubtreeMap events match what we have.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: set LogEvent _start_off, drop unused _end_off

Signed-off-by: Sage Weil <sage@newdream.net>

mds: witness rmdir when subtrees are on other hosts

If there is an rmdir with an empty subtree on another mds, we need to witness/
journal that on the dirfrag's auth mds so that replay correctly updates the
subtree map.

This is simpler than the rename witnesses (and the link/unlink ones) because
we aren't actually journaling a modification to any actual metadata; it's just
the subtree map that is changing.

The projection of the subtree map update needs work still, but that is also the
case for renames.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-rgw-multithreaded'

rgw: setup as external server is configurable

rgw: clearer logging context

config.cc: fix comments

Fix comments that were reverted to an older version by commit
a51b983a8bbafc455bcd2acd993981ece00abcc7.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

PGMonitor.cc: add force_create_pg command to monitor

ceph pg force_create_pg <pgid> will now reset that pg's status to
creating to handle corner cases where no osd has that pg.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: fix concurrency issue

Merge branch 'stable'

mds: clean up file flags to file mode translation

There was some seriously wrong and ancient cruft in there.  open(2)
specifies that one of O_RDONLY, O_WRONLY, and O_RDWR must always be
specified.  Drop all the crazy.

Linux VFS interprets O_WRONLY|O_RDWR as read+write, so we'll do the same.

Signed-off-by: Sage Weil <sage@newdream.net>

client: check for invalid file mode

Signed-off-by: Sage Weil <sage@newdream.net>

mds: check for invalid file mode

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: fix i386 compilation

config: fix compilation in certain environments

mon: fix dump_json quoting

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix O_APPEND file mode calculation

Currently

  fd = open("aaa",O_RDWR|O_CREAT|O_APPEND,0666);

gives us fmode of 1 instead of 3.  We should only infer O_WRONLY if no
other mode is explicitly specified.

Reported-by: Fyodor Ustinov <ufm@ufm.su>
Signed-off-by: Sage Weil <sage@newdream.net>

rgw: use thread pool for rgw work

mon: pg dump_json

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: radosgw_admin log show dumps parsable json

osd: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mon: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

tools: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

clien: rados: bench: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ReplicatedPG: _rollback_to

_delete_head modifies clone_overlap and stats incorrectly for
_rollback_to

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

testsnaps: use self managed snaps and other fixes

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Makefile: librados-config apparently depends on libcrypto++

In newer versions of gcc/ld, librados-config breaks without
the CRYPTO_LIBS.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

gitignore: ignore Eclipse files/folders

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Remove lingering references to g_conf

These were harmless, but silly.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: fix range dump

mds: fix protocol versions

I screwed this up in a928f767b974cd36f4be6ea618c5cec366d28362.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: multithreaded handling

recalc_op_target must give an error if pool gone

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Makefile: remove dup testrados_list_parallel_SOURCES

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'origin/stable'

Merge remote branch 'origin/wip-1287'

rgw: multiple fixes, prepare main loop for multithreading

rgw: fix compilation for certain architectures

botol_tool: add --xattr argument for PUT

boto_tool can now set extended attributes when performing a PUT
operation.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

boto_tool: add --get-obj-xattr

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_op: mime-encode ugly xattrs in PUT

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_common: remove unused variable

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mime encoding: encode control chars

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add control character detection

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: fix log

rgw: cleanup, remove globals and static craft

bloom filter: update to latest version

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

filepath: remove asserts

These were included as a safety check and haven't been triggered, so
there's no point slowing anything down with them.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

honor CINIT_FLAG_NO_DEFAULT_CONFIG_FILE

Don't use CEPH_CONF_FILE_DEFAULT when CINIT_FLAG_NO_DEFAULT_CONFIG_FILE
is set.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

crushtool: don't parse any generic args

Notably, crushtool uses -c, which elsewhere means parse this config file.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'rgw-sync-cache'

Conflicts:
src/rgw/rgw_op.cc

rgw: configurable way to enable/disable cache

ReplicatedPG: project changes to clone_overlap

Previously, changes to clone_overlap were incorrect since make_writeable
is called after do_osd_ops.  Now, ctx->modified_ranges will hold the set
of offsets modified during the transaction to be applied to
clone_overlap during make_writeable.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'wip-codingstyle'

CodingStyle: whitespace

Signed-off-by: Sage Weil <sage@newdream.net>

CodingStyle: final decisions?

- _d suffix for naked struct/class types (not _t!)

- m_ prefix for class members

- prefer braces for single line ifs.

Signed-off-by: Sage Weil <sage@newdream.net>

ReplicatedPG: sub_op_push fix

We need to iterate over the intersection of data_subset and data_needed,
not the original data_needed.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge remote branch 'origin/wip-1257'

Don't build build tests unless requested

Build tests (that check if there are unresolved symbols in libraries)
can slow down the build a lot. We should only enable them when
developers need them.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados parallel tests: prepend "test"

Just to avoid any confusion, all these test apps should begin with the
word "test".

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: add test of list-while-modify

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: unwatch cache sync object before exiting

testrados: fix

Change testrados back to a simple single-threaded test. The
multi-threaded tests need to be properly designed to run multi-threaded.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

systest_runnable: print line when joining runnables

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Create rados_delete_pools_parallel test

* add rados_delete_pools_parallel test

* change RETURN_IF_NOT_VAL -> RETURN1_IF_NOT_VAL.
We want to return a non-zero error code when the value is something we
don't expect, even if that unexpected value is 0.

* st_rados_list_objects: add option to ignore list errors (for the
deletion test)

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ReplicatedPG: data_subset may not contain data_needed

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

encoding.h: need to pass by reference

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: fix snaps decode in recover_primary

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

test/osd: TestSnaps now writes large, random objects with holes

In order to test clone recovery pathways, TestSnaps now writes to random
subsets of an object between snapshots.  Objects may now also be very
large.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

mds: fix handle_simple_lock on LOCK replica

We were ignoring caps.  To avoid this kind of mistake, use eval_gather(),
like handle_file_lock() does.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: drop useless from field from MInodeFileCaps

While we're updating the protocol, clean this up too.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: print cur+nested auth_pins on inode

And clean up a bit.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: move to MIX state if writer wanted and no wanted loner

We can just look at the target loner here, which also takes any caps wanted
by other replicas on other MDSs into account.  Otherwise we need to
to duplicate the CInode::calc_ideal_loner() logic.

This assumes the loner field is accurate.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: print mds_caps_wanted on inode

along with the client caps info

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: improve print() for MInodeFileCaps

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: migrate loner_cap state

It is tedious to infer what the old loner_cap was pre-migration. Just send
it over the wire and set it explicitly. Usually when we eval() we would
have come to the same conclusion, but when we didn't, we got into
inconsistent/impossible states where the issue caps don't match the loner
state (Asx issued but no loner_cap set).  That meant the next issue was
a revocation with no lock state change, which led to yet more problems
down the line.

This is an internal MDS protocol change.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: clear [want_]loner_cap on cap export

These fields are meaningless on the replica.

Signed-off-by: Sage Weil <sage@newdream.net>

Move list objects logic to st_rados_list_objects

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: fix rados_ioctx_destroy uses

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: add test4

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: test adding objects + listing

Test one runnable adding objects while another runnable lists them.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add rerun.sh

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: make num_objects selectable

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

add CrossProcessSem::reinit, fix rados_list_parallel

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Now we can put runnable logs into separate files

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add rados_open_pools_parallel test

Also add rados_open_pools_parallel to .gitignore

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: clean up temp object if failed to PUT

Conflicts:

src/rgw/rgw_op.cc

rgw: clean up temp object if failed to PUT

rgw: remove include of curl/types.h

test/rgw_escape: add JSON + utf-8 test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rgw_escape.cc: add utf-8 test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

CodingStyle: Typos, whitespace.

rgw: fix escaping clobbering utf8

crushtool: Add call to global_init() to avoid dout segfault

Signed-off-by: Jim Schutt <jaschut@sandia.gov>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>