client: whitespace cleanup

Signed-off-by: Sage Weil <sage@newdream.net>

client: send renames to the correct node

AFAICS we got this wrong in b794aeb39, which is annoying given I remember
thinking reasonably carefully about it.

The current reasoning: the request operates on the dentry target
directory from the MDS's perspective; that's where the request needs to go.

Signed-off-by: Sage Weil <sage@newdream.net>

libceph.h: fix C compilation

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'rgw-formatter'

In RGW, use the common/Formatter

Implement a new subclass of Formatter, RGWFormatter_Plain, using code
from the old RGWFormatter and RGWFormatter_Plain classes.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

filestore: remove unused var

Signed-off-by: Sage Weil <sage@newdream.net>

Formatter: add open_object_section_in_ns, etc.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

test/formatter: test stream clearing

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Formatter: must use str("") to clear ostreams

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

More changes towards using common/Formatter in rgw

* rename RgwFormatter::init to RgwForamtter::reset.
Roll old reset() into base_reset().

* add write_raw_data to Formatter, to support DTDs.
Remove other DTD support code.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mds: add hooks to test multimds link/unlink failures

Signed-off-by: Sage Weil <sage@newdream.net>

qa: move fsx.sh

Revert "qa: Remove fsx workunit."

This reverts commit a66da4af2571f27f55e8ee819f22e70355fb42b0.  Still useful
to have this around.

os/FileStore: print out BTRFS_IOC_SNAP_CREATE errs

Print out the errno for BTRFS_IOC_SNAP_CREATE errors if they happen.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_rest: don't leak formatters

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

XmlFormatter: add DTD option

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Formatter.cc: add get_len method

Add get_len method because rgw needs it.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

RGWFormatter: get rid of one flush variant

This flush variant won't exist in common/Formatter.cc.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw_formats: introduce dump_unsigned, dump_int,etc

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: rename dump_value_str to dump_format

rename dump_value_str to dump_format for compatibility with Formatter.cc

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rados tool: fix cmdline parsing for --block-size

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Formatter: add XMLFormatter::reset

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Formatter.cc: whitespace fix

whitespace fix to conform to the coding standard

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw:rename open_obj_section to open_object_section

For compatibility with common/Formatter.cc

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Add XMLFormatter and unit test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Formatter: remove protected data, methods

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

osd, objecter, librados: add SRC_CMPXATTR operation

and test.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: change src_oid encoding -- FLAG DAY

The old encoding was mutually exclusive with putting any data payload on
the operation.  That was stupid.. we can't, for example, do xattr ops then
on a src_oid.

Fix this by just including the oid in the data payload inline whenever the
bit is set in the op code.  This changes the client protocol in an
incompatible way, which means users of the CLONERANGE operation need to be
upgrade/downgraded in unison.

Signed-off-by: Sage Weil <sage@newdream.net>

testradospp: test cmpxattr (eq, str)

No coverage for the other tests yet.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix cmpxattr string construction

The bufferlist may not contain a terminating '\0'.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter, osd: fix up cmpxattr

Objecter stubs, useful debug output.

Signed-off-by: Sage Weil <sage@newdream.net>

systest: add watch/notify tests with pool and object deletion

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

systest: st_rados_create_pool: optionally wait before creating the pool

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

systest: add object deletion to mirror object creation

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

systest: add st_rados_notify building block for watch/notify tests

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

systest: add st_rados_watch building block for watch/notify tests

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

systest: make pool configurable

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

systest: move pool deletion into its own file

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados: fix notify deadlock

The success of the notify call needs to checked before waiting to
receive a notification. If we try notifying on an object that does not
exist, for example, it should fail with -ENOENT, and not hang.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: document remove_watchers race avoidance

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

pg: remove do_complete_notify

This method has no dependence on the pg.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: put_object_context: tolerate pgs being deleted

PGs that are queued for deletion won't be in the osdmap,
and may not be in the pg_map, but if they are, it's safe to
put object_context. Otherwise, the pg is being deleted and
will clean up the object contexts itself.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

osd, pg: clean up watchers on pg deletion and shutdown

Watchers and their object contexts need to be cleaned up so
they aren't used after the pg is gone. This happened if the
pool was deleted and the connection to the watcher was reset.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

mds: detach replay thread

Since we don't join it.

This fixes a leak of per-thread state.. namely, an ~8MB chunk of virtual
memory (and a handful of real pages).

Signed-off-by: Sage Weil <sage@newdream.net>

thread: detach()

Signed-off-by: Sage Weil <sage@newdream.net>

common/Formatter: add unit test

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

mdsmon: send commands to all MDSes, not just the in&up ones.

Now we can send messages to standbys via broadcast, even if we
can't yet single them out.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cephtool: accept semicolons in commandline args

Semicolons can now be used to give multiple arguments to cephtool in one
invocation. So a command like this is now possible:

$ ./ceph osd dump --format=json \; pg dump --format=json -o -

The backslash is to prevent the shell from consuming the semicolon.

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

testradospp: add version tests

get_last_version
assert_version
assert_src_version

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librados: add assert_src_version

Like set_assert_version, this is an IoCtx operation that affects the next
(and only the next) operation we perform.

Factor out the current assert version check code into a helper that also
handles the src asserts.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objecter: add assert_src_version

as an ObjectOperation

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: add ASSERT_SRC_VERSION operation

Assert a src object has a particular version.  This is analogous to the
ASSERT_VER operation, but operations on a src_oid instead of the target.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'stable'

osd: set reply_version for read operations

This was probably broken by the OSD prepare_transaction refactor a few
months ago.  Or it never worked.  Adding test to testradospp.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'origin/objecter-latest-map'

rgw: quiet down some log messages

escape_json_attr: don't escape single quotes

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Formatter.cc: use common/escape.h

* Rename rgw/rgw_escape.h to common/escape.h

* Use escape.h in common/Formatter.cc

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Makefile.am: always #define __STDC_FORMAT_MACROS

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Merge branch 'stable'

mds: request attempt comes from fwd count, not retry flag

mds: fix create_subtree_map for new dirs

Currently mkdir foo ; rmdir foo fails because we can't get_subtree_map()
on a new directory that isn't linked in the committed plane.  Since we are
journaling the projected subtree, it makes sense to use
get_projected_subtree_map() here.

It's easiest to keep in both the old and new directories in the rename
project map instead of looking at the next-to-most-recent parent for the
inode.  The committed version is irrelevant (could conceivably be multiple
renames behind) and the current projected parent is just newdir; we need
olddir too, and we don't project for cross-mds rename anyway.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

vstart: static mapping of names to ranks

a always 0, b always 1, etc. makes multi-mds debugging much easier.

Signed-off-by: Sage Weil <sage@newdream.net>

v0.32

rgw: don't silently ignore bad user/group when setting acl

objecter: rename POOL_DISAPPEARED to POOL_DNE

The pool may never have existed.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

objecter: check for updated osdmap when requesting a non-existent pool

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

objecter: fix error check - error return code is negative

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

monclient: add method to retrieve the latest version of a map

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

mon: add GetVersion message

This allows clients to determine whether they have the latest
mds, mon, or osd map. This is useful for figuring out if a pool
does not exist, or if the osdmap with it simply hasn't been
received yet.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Makefile: include HeartbeatMap.h in dist

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix validation of (slave) request attempts

Verify that slave requests received are not stale.

Verify that slave replies match the currently processing request.

Clean up the code a bit.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: identify slave requests with reqid + attempt number

We need to distinguish between different attempts to process a request, or
else we can get annoying races in the slave request handling code.  E.g.,

- request sent to mds A
- A authpins items on B, B registered slave_request
- A forwards request to C, sends slave finish to B
- C receives request, sends authpin slave request to B
- B receives C's authpin request, discards (*)
- B receives A's finish, closes slave request

First we just add tracking of the attempt number.

Signed-off-by: Sage Weil <sage@newdream.net>

scatterlock: fix flag assignments.

Want |= to set a flag, not &=!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osdmap: in json dump, dump out/in, up/down status

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: get current utc epoch differently

beforehand tm.tm_isdst was returning random results which happened
to work correctly most of the time since we're currently in dst

rgw: init correctly req_state->{bucket, object}

rgw: fix total time reporting in rgw_admin

rgw: tweak content-md5 handling

heartbeatmap: fix/clarify the commenting

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

scatterlock: compress boolean flags into a set of state flags

While we're at it, unify the naming structure a bit and remove
the unused stale flag.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

scatterlock: rename scatter_flags -> state_flags

We want to use this for all the bools, not just the scatter ones.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Makefile: remove from libglobal

Signed-off-by: Sage Weil <sage@newdream.net>

Add -ltr to libcommon

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

Makefile: -lrt for libglobal.la only

Debugging linking is a pita.

Signed-off-by: Sage Weil <sage@newdream.net>

unittest_bufferlist: change include order

fixes a build error (int type conflicts) for me on fatty.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix log trimming races

trim() would iterate over segments.  It would take the *p segment, ++p,
then call try_expire().  But the _expired() function would also clean up
and (if possible) retire subsequent segments on the list if they were on
the expired list, invalidating the p iterator.

Untangle the mess by making expired segment trimming (i.e. removing from
segment list) a separate operation performed only by trim() (probably a
good idea anyway).  This keeps the iterator safe/stable.

Signed-off-by: Sage Weil <sage@newdream.net>

mds: separate type for gratuitous debug ESubtreeMaps

Give these a different type so they are not interpreted as subtree
boundaries during replay.  Otherwise we break the truncate_finish code,
which references the truncate_start logsegment by offset.  Probably other
stuff too.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: 'ceph mon dump [--format=json]'

Signed-off-by: Sage Weil <sage@newdream.net>

heartbeatmap: unit test

Signed-off-by: Sage Weil <sage@newdream.net>

heartbeatmap: we don't care about pthread_t

Workers don't have to be threads.

Signed-off-by: Sage Weil <sage@newdream.net>

client: open session with all mds targets

If we have an open session with an mds, we need to have an open session.

The problem is if we, say,

- client has old mdsmap
- mds A adds B as target in mdsmap
- send request to mds A
- A exports to B
- we get the EXPORT, but B isn't listed as a target for A in client map
- client gets updated map

At the time we receive the map we need to open the session to B.   We can't
really do it when we get the EXPORT because we don't know the target MDS.

We can either track which exports are pending to do it, or just blindly
open sessions with targets for any MDSs we have caps with.  Which is
basically every session we have open.  That's simplest for now.

Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: fix unittest_ceph_argparse build

Signed-off-by: Sage Weil <sage@newdream.net>

injectargs: complain about unparsed args

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

injectargs: print out what is changing

Signed-off-by: Colin McCabe <colin.mccabe@dreamhost.com>

rgw: fix base64 check

rgw: check content md5 validity when doing auth

rgw: fix date checks

rgw: fix authentication

scatterlock: convert [un]scatter_wanted to a bitfield

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>