librbd: make read_iterate mirror AioCompletion::complete

This makes testlibrbd and testlibrbdpp pass again, and
now both I/O paths have the same logic.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

testlibrbd: remove useless print statements

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

testlibrbd: NUL-terminate orig_data

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

testlibrbd(pp): accept standard command line arguments

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: don't request heartbeats from down peers

We try to add sources for anything in peer_info, which can include down
OSDs.  Skip them.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'debian-deps'

debian: explicitly bind library users to matching version

We are cheating with the shared libs by making small API changes without
bumping the soname.  Bind users to a matching version to minimize user
pain.  When the APIs become fully stable these will need to go away.

Fixes: #1354
Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-heartbeats'

Conflicts:
src/osd/OSD.cc

Merge branch 'wip-config'

osd: missing ! in ifs on bad map after osd boot

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix Migrator::audit() for IMPORT_ABORTING

Make audit match reality.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objecter: halt compound operation only if result < 0

Remove unused dyn_sprintf

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config.cc: clean up includes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: chain rename subtree projections

We can have two renames for the same file in flight to the journal.  Stack
them up in a list.  The old project_subtree_rename() should have asserted
that the item wasn't already in the map before inserting it to catch this
at the front end.  Now it doesn't matter; it's a list.

Signed-off-by: Sage Weil <sage@newdream.net>

client: whitespace cleanup

Signed-off-by: Sage Weil <sage@newdream.net>

client: send renames to the correct node

AFAICS we got this wrong in b794aeb39, which is annoying given I remember
thinking reasonably carefully about it.

The current reasoning: the request operates on the dentry target
directory from the MDS's perspective; that's where the request needs to go.

Signed-off-by: Sage Weil <sage@newdream.net>

config: more thread-safety stuff

* Don't allow parse_argv, parse_env, or parse_config_files to be used
after threads have been started.

* Don't allow set_val to be used to change unsafe variables after
threads have been started.

* Test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

injectargs: forbid unsafe string modifications

Don't allow string-valued configuration items to be changed using
injectargs unless they have observers. Otherwise, we could have
crashes, since one thread could be reading the std::string's internal
buffer after another thread frees that buffer during assignment.

Write a unit test to validate this behavior.

Also test that we can turn on and off the log_file using injectargs.
This is something that injectargs often gets used for in practice.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/daemon_config: add injectargs test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add daemon_config, libceph_config, etc.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

add config unit test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

librados.h: fix out-of-date comment

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

debian: don't explicitly list deps; let debian figure it out

Signed-off-by: Sage Weil <sage@newdream.net>

config: more cleanup

* any place where we know a set_val can't fail should be set_val_or_die

* in private md_config_t functions that need the mutex, assert that the
mutex is locked

* make md_config_t::expand_meta private, since nobody uses it outside of
the class.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

libceph.h: fix C compilation

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

config: add proper locking, fix comments

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'rgw-formatter'

In RGW, use the common/Formatter

Implement a new subclass of Formatter, RGWFormatter_Plain, using code
from the old RGWFormatter and RGWFormatter_Plain classes.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

filestore: remove unused var

Signed-off-by: Sage Weil <sage@newdream.net>

Formatter: add open_object_section_in_ns, etc.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/formatter: test stream clearing

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Formatter: must use str("") to clear ostreams

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

More changes towards using common/Formatter in rgw

* rename RgwFormatter::init to RgwForamtter::reset.
Roll old reset() into base_reset().

* add write_raw_data to Formatter, to support DTDs.
Remove other DTD support code.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd: expect heartbeats from anyone peering depends on

We were getting heartbeats from just acting replicas.  That's really not
enough if we want to be sure to detect failures of OSDs we depend on,
which includes any stray or up OSDs as well.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: restructure heartbeats

Primary sends explicit message to replicas asking for heartbeats to start,
or to stop.  Replicas never send heartbeats unless explicitly requested
to.

Do not try to mark down old heartbeat relationships; that gets messy, and
the overhead of open sockets is too small to worry about right now.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: add hooks to test multimds link/unlink failures

Signed-off-by: Sage Weil <sage@newdream.net>

qa: move fsx.sh

Revert "qa: Remove fsx workunit."

This reverts commit a66da4af2571f27f55e8ee819f22e70355fb42b0.  Still useful
to have this around.

os/FileStore: print out BTRFS_IOC_SNAP_CREATE errs

Print out the errno for BTRFS_IOC_SNAP_CREATE errors if they happen.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_rest: don't leak formatters

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

XmlFormatter: add DTD option

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Formatter.cc: add get_len method

Add get_len method because rgw needs it.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

RGWFormatter: get rid of one flush variant

This flush variant won't exist in common/Formatter.cc.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_formats: introduce dump_unsigned, dump_int,etc

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: rename dump_value_str to dump_format

rename dump_value_str to dump_format for compatibility with Formatter.cc

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados tool: fix cmdline parsing for --block-size

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Formatter: add XMLFormatter::reset

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Formatter.cc: whitespace fix

whitespace fix to conform to the coding standard

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw:rename open_obj_section to open_object_section

For compatibility with common/Formatter.cc

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add XMLFormatter and unit test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Formatter: remove protected data, methods

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd, objecter, librados: add SRC_CMPXATTR operation

and test.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: change src_oid encoding -- FLAG DAY

The old encoding was mutually exclusive with putting any data payload on
the operation.  That was stupid.. we can't, for example, do xattr ops then
on a src_oid.

Fix this by just including the oid in the data payload inline whenever the
bit is set in the op code.  This changes the client protocol in an
incompatible way, which means users of the CLONERANGE operation need to be
upgrade/downgraded in unison.

Signed-off-by: Sage Weil <sage@newdream.net>

testradospp: test cmpxattr (eq, str)

No coverage for the other tests yet.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix cmpxattr string construction

The bufferlist may not contain a terminating '\0'.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter, osd: fix up cmpxattr

Objecter stubs, useful debug output.

Signed-off-by: Sage Weil <sage@newdream.net>

systest: add watch/notify tests with pool and object deletion

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

systest: st_rados_create_pool: optionally wait before creating the pool

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

systest: add object deletion to mirror object creation

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

systest: add st_rados_notify building block for watch/notify tests

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

systest: add st_rados_watch building block for watch/notify tests

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

systest: make pool configurable

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

systest: move pool deletion into its own file

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados: fix notify deadlock

The success of the notify call needs to checked before waiting to
receive a notification. If we try notifying on an object that does not
exist, for example, it should fail with -ENOENT, and not hang.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: document remove_watchers race avoidance

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

pg: remove do_complete_notify

This method has no dependence on the pg.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: put_object_context: tolerate pgs being deleted

PGs that are queued for deletion won't be in the osdmap,
and may not be in the pg_map, but if they are, it's safe to
put object_context. Otherwise, the pg is being deleted and
will clean up the object contexts itself.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

osd, pg: clean up watchers on pg deletion and shutdown

Watchers and their object contexts need to be cleaned up so
they aren't used after the pg is gone. This happened if the
pool was deleted and the connection to the watcher was reset.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

mds: detach replay thread

Since we don't join it.

This fixes a leak of per-thread state.. namely, an ~8MB chunk of virtual
memory (and a handful of real pages).

Signed-off-by: Sage Weil <sage@newdream.net>

thread: detach()

Signed-off-by: Sage Weil <sage@newdream.net>

common/Formatter: add unit test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mdsmon: send commands to all MDSes, not just the in&up ones.

Now we can send messages to standbys via broadcast, even if we
can't yet single them out.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cephtool: accept semicolons in commandline args

Semicolons can now be used to give multiple arguments to cephtool in one
invocation. So a command like this is now possible:

$ ./ceph osd dump --format=json \; pg dump --format=json -o -

The backslash is to prevent the shell from consuming the semicolon.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

testradospp: add version tests

get_last_version
assert_version
assert_src_version

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librados: add assert_src_version

Like set_assert_version, this is an IoCtx operation that affects the next
(and only the next) operation we perform.

Factor out the current assert version check code into a helper that also
handles the src asserts.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objecter: add assert_src_version

as an ObjectOperation

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: add ASSERT_SRC_VERSION operation

Assert a src object has a particular version.  This is analogous to the
ASSERT_VER operation, but operations on a src_oid instead of the target.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'stable'

osd: set reply_version for read operations

This was probably broken by the OSD prepare_transaction refactor a few
months ago.  Or it never worked.  Adding test to testradospp.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'origin/objecter-latest-map'

rgw: quiet down some log messages

escape_json_attr: don't escape single quotes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Formatter.cc: use common/escape.h

* Rename rgw/rgw_escape.h to common/escape.h

* Use escape.h in common/Formatter.cc

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Makefile.am: always #define __STDC_FORMAT_MACROS

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'stable'

mds: request attempt comes from fwd count, not retry flag

mds: fix create_subtree_map for new dirs

Currently mkdir foo ; rmdir foo fails because we can't get_subtree_map()
on a new directory that isn't linked in the committed plane.  Since we are
journaling the projected subtree, it makes sense to use
get_projected_subtree_map() here.

It's easiest to keep in both the old and new directories in the rename
project map instead of looking at the next-to-most-recent parent for the
inode.  The committed version is irrelevant (could conceivably be multiple
renames behind) and the current projected parent is just newdir; we need
olddir too, and we don't project for cross-mds rename anyway.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

vstart: static mapping of names to ranks

a always 0, b always 1, etc. makes multi-mds debugging much easier.

Signed-off-by: Sage Weil <sage@newdream.net>

v0.32

rgw: don't silently ignore bad user/group when setting acl

objecter: rename POOL_DISAPPEARED to POOL_DNE

The pool may never have existed.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

objecter: check for updated osdmap when requesting a non-existent pool

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

objecter: fix error check - error return code is negative

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

monclient: add method to retrieve the latest version of a map

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

mon: add GetVersion message

This allows clients to determine whether they have the latest
mds, mon, or osd map. This is useful for figuring out if a pool
does not exist, or if the osdmap with it simply hasn't been
received yet.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Makefile: include HeartbeatMap.h in dist

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix validation of (slave) request attempts

Verify that slave requests received are not stale.

Verify that slave replies match the currently processing request.

Clean up the code a bit.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: identify slave requests with reqid + attempt number

We need to distinguish between different attempts to process a request, or
else we can get annoying races in the slave request handling code.  E.g.,

- request sent to mds A
- A authpins items on B, B registered slave_request
- A forwards request to C, sends slave finish to B
- C receives request, sends authpin slave request to B
- B receives C's authpin request, discards (*)
- B receives A's finish, closes slave request

First we just add tracking of the attempt number.

Signed-off-by: Sage Weil <sage@newdream.net>

scatterlock: fix flag assignments.

Want |= to set a flag, not &=!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>