PGMonitor.cc: add force_create_pg command to monitor

ceph pg force_create_pg <pgid> will now reset that pg's status to
creating to handle corner cases where no osd has that pg.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'stable'

mds: clean up file flags to file mode translation

There was some seriously wrong and ancient cruft in there.  open(2)
specifies that one of O_RDONLY, O_WRONLY, and O_RDWR must always be
specified.  Drop all the crazy.

Linux VFS interprets O_WRONLY|O_RDWR as read+write, so we'll do the same.

Signed-off-by: Sage Weil <sage@newdream.net>

client: check for invalid file mode

Signed-off-by: Sage Weil <sage@newdream.net>

mds: check for invalid file mode

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: fix i386 compilation

config: fix compilation in certain environments

mon: fix dump_json quoting

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix O_APPEND file mode calculation

Currently

  fd = open("aaa",O_RDWR|O_CREAT|O_APPEND,0666);

gives us fmode of 1 instead of 3.  We should only infer O_WRONLY if no
other mode is explicitly specified.

Reported-by: Fyodor Ustinov <ufm@ufm.su>
Signed-off-by: Sage Weil <sage@newdream.net>

mon: pg dump_json

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: radosgw_admin log show dumps parsable json

osd: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mon: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

tools: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

clien: rados: bench: remove unused variables

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ReplicatedPG: _rollback_to

_delete_head modifies clone_overlap and stats incorrectly for
_rollback_to

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

testsnaps: use self managed snaps and other fixes

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Makefile: librados-config apparently depends on libcrypto++

In newer versions of gcc/ld, librados-config breaks without
the CRYPTO_LIBS.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

gitignore: ignore Eclipse files/folders

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Remove lingering references to g_conf

These were harmless, but silly.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: fix range dump

mds: fix protocol versions

I screwed this up in a928f767b974cd36f4be6ea618c5cec366d28362.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

recalc_op_target must give an error if pool gone

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Makefile: remove dup testrados_list_parallel_SOURCES

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'origin/stable'

Merge remote branch 'origin/wip-1287'

rgw: fix compilation for certain architectures

botol_tool: add --xattr argument for PUT

boto_tool can now set extended attributes when performing a PUT
operation.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

boto_tool: add --get-obj-xattr

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_op: mime-encode ugly xattrs in PUT

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_common: remove unused variable

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mime encoding: encode control chars

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add control character detection

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

bloom filter: update to latest version

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

filepath: remove asserts

These were included as a safety check and haven't been triggered, so
there's no point slowing anything down with them.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

honor CINIT_FLAG_NO_DEFAULT_CONFIG_FILE

Don't use CEPH_CONF_FILE_DEFAULT when CINIT_FLAG_NO_DEFAULT_CONFIG_FILE
is set.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

crushtool: don't parse any generic args

Notably, crushtool uses -c, which elsewhere means parse this config file.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'rgw-sync-cache'

Conflicts:
src/rgw/rgw_op.cc

rgw: configurable way to enable/disable cache

ReplicatedPG: project changes to clone_overlap

Previously, changes to clone_overlap were incorrect since make_writeable
is called after do_osd_ops.  Now, ctx->modified_ranges will hold the set
of offsets modified during the transaction to be applied to
clone_overlap during make_writeable.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'wip-codingstyle'

CodingStyle: whitespace

Signed-off-by: Sage Weil <sage@newdream.net>

CodingStyle: final decisions?

- _d suffix for naked struct/class types (not _t!)

- m_ prefix for class members

- prefer braces for single line ifs.

Signed-off-by: Sage Weil <sage@newdream.net>

ReplicatedPG: sub_op_push fix

We need to iterate over the intersection of data_subset and data_needed,
not the original data_needed.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge remote branch 'origin/wip-1257'

Don't build build tests unless requested

Build tests (that check if there are unresolved symbols in libraries)
can slow down the build a lot. We should only enable them when
developers need them.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados parallel tests: prepend "test"

Just to avoid any confusion, all these test apps should begin with the
word "test".

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: add test of list-while-modify

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: unwatch cache sync object before exiting

testrados: fix

Change testrados back to a simple single-threaded test. The
multi-threaded tests need to be properly designed to run multi-threaded.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

systest_runnable: print line when joining runnables

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Create rados_delete_pools_parallel test

* add rados_delete_pools_parallel test

* change RETURN_IF_NOT_VAL -> RETURN1_IF_NOT_VAL.
We want to return a non-zero error code when the value is something we
don't expect, even if that unexpected value is 0.

* st_rados_list_objects: add option to ignore list errors (for the
deletion test)

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

ReplicatedPG: data_subset may not contain data_needed

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

encoding.h: need to pass by reference

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: fix snaps decode in recover_primary

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

test/osd: TestSnaps now writes large, random objects with holes

In order to test clone recovery pathways, TestSnaps now writes to random
subsets of an object between snapshots.  Objects may now also be very
large.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

mds: fix handle_simple_lock on LOCK replica

We were ignoring caps.  To avoid this kind of mistake, use eval_gather(),
like handle_file_lock() does.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: drop useless from field from MInodeFileCaps

While we're updating the protocol, clean this up too.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: print cur+nested auth_pins on inode

And clean up a bit.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: move to MIX state if writer wanted and no wanted loner

We can just look at the target loner here, which also takes any caps wanted
by other replicas on other MDSs into account.  Otherwise we need to
to duplicate the CInode::calc_ideal_loner() logic.

This assumes the loner field is accurate.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: print mds_caps_wanted on inode

along with the client caps info

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: improve print() for MInodeFileCaps

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: migrate loner_cap state

It is tedious to infer what the old loner_cap was pre-migration. Just send
it over the wire and set it explicitly. Usually when we eval() we would
have come to the same conclusion, but when we didn't, we got into
inconsistent/impossible states where the issue caps don't match the loner
state (Asx issued but no loner_cap set).  That meant the next issue was
a revocation with no lock state change, which led to yet more problems
down the line.

This is an internal MDS protocol change.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: clear [want_]loner_cap on cap export

These fields are meaningless on the replica.

Signed-off-by: Sage Weil <sage@newdream.net>

Move list objects logic to st_rados_list_objects

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: fix rados_ioctx_destroy uses

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: add test4

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: test adding objects + listing

Test one runnable adding objects while another runnable lists them.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add rerun.sh

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados_list_parallel: make num_objects selectable

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

add CrossProcessSem::reinit, fix rados_list_parallel

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Now we can put runnable logs into separate files

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add rados_open_pools_parallel test

Also add rados_open_pools_parallel to .gitignore

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: clean up temp object if failed to PUT

Conflicts:

src/rgw/rgw_op.cc

rgw: clean up temp object if failed to PUT

rgw: remove include of curl/types.h

test/rgw_escape: add JSON + utf-8 test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/rgw_escape.cc: add utf-8 test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

CodingStyle: Typos, whitespace.

rgw: fix escaping clobbering utf8

crushtool: Add call to global_init() to avoid dout segfault

Signed-off-by: Jim Schutt <jaschut@sandia.gov>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Revert "mds: handle importer failure on exporter"

This reverts commit bf768cb146b23d345efcc51badfe668bf37d4046.

This won't actually happen with f44cffdb8ccc8cf258b418146dbbfc20a9b685f9

obsync: Depend on python-pyxattr instead of python-xattr

Signed-off-by: Wido den Hollander <wido@widodh.nl>
Signed-off-by: Sage Weil <sage@newdream.net>

mds: verify deferred messages aren't stale

We may defer processing of some messages because we are laggy (in hearing
from the monitor).  When we eventually get to those messages, make sure
they haven't since become stale (i.e., the source mds isn't now down).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: handle importer failure on exporter

If the export is reversed but we still receive the ack, behave.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rados: fix warning

warning: rados.cc:319: large integer implicitly truncated to unsigned type

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: print ino along with &fh for ll_ file ops

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rados: fix warning

warning: rados.cc:319: large integer implicitly truncated to unsigned type

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: bucket deletion updates caches

mime.c: avoid compiler warning

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/mime: test invalid decodes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mime.h: better comments

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add mime encoding

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: fix lru

rgw: restructure runtime config env

mds: only issue xlocker_caps if we are also the loner

We cannot issue caps to a client purely because they have something
xlocked, because we do not revoke caps when we drop the xlock.  However,
if they are a loner AND have the object xlocked, we can; this is why the
xlock release code either moves to LOCK or EXCL state.

Remember, the goal here is to issue caps when we do operations on objects
that xlock them (e.g. setattr, mknod) and move directly to the EXCL state
afterward.  That only works (or makes sense) when we are the lone client
with caps.

Fix the get_caps_allowed_by_type() helper to do this properly.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: rely on master to do anchor locks for slave_link_prep

The replica can't take all these locks without confusing things, since it
maybe need to unlock/relock, may screw up auth_pins, and worse.  The master
can take the locks.

The only problem is that the master may not know if the inode has already
been anchored if the lock hasn't cycled since then.  In that case, we take
more locks than we need to.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: set I_COMPLETE after adding cap

This avoids immediately clearing it because we are newly (re)issued the
Fs cap (a generic check in the cap add code).

Signed-off-by: Sage Weil <sage@newdream.net>

client: reenable local ENOENT

This was accidentally disabled by commit:c7e248b.

Signed-off-by: Sage Weil <sage@newdream.net>