osd: move object_locator_t to osd_types.{h,cc}

That's a better home.  Also add to ceph-dencoder.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: add osd_reqid_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: add hobject_t

Move to a separate file in os/, since this is an ObjectStore related
object.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: add/use pg_create_t ctor

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: add pg_create_t (formerly MOSDPGCreate::create_rec)

Signed-off-by: Sage Weil <sage@newdream.net>

test/encoding/readable.sh: no \t

Not sure why this sometimes works and sometimes doesn't.  Maybe it's a
bashism?

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-corpus'

ceph-dencoder: add Log{Entry,EntryKey,Summary}

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: wait to start op if other ops are in line

We can have a sequence like:

- commit_start, blocked=true
- op_start thread A gets in line
- op_start thread B gets in line
- commit finished, blocked=false
- thread A goes
- op_start thread C sees blocked=false and continues
-> order broken

If there are people in line from a previous block, we need to get in line,
even if blocked == false.

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: fix op queue quiesce during commit

When I added the ordering constraint fix back in 259c509a I got the
check backwards.  We want to wait if we are blocked OR we are not in the
front of the line (i.e., proceed if we are not blocked AND first in line).

Fixes: #2046
Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'gh/wip-types'

Merge remote branch 'gh/wip-stuck-in-backfill'

Reviewed-by: Sage Weil <sage.weil@dreamhost.com>

ReplicatedPG: don't count deletions as ops

Counting them as ops but not requeueing the pg for recovery causes
backfill to stall when only deletions are sent in
recover_backfill(). Deletions are cheap and don't need to be acked, so
we can simply stop counting them as ops.

Fixes: #2044
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: don't remove pg from recovery queue if not enough recovery ops started

The pg has already been dequeued at the beginning of do_recovery(),
and it requeues itself only if it starts a new recovery op.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: don't treat plus as a space in url decode

Any special character encoding should be done through %hex. The
plus sign is a valid character in object names, and in user id
(when used in signed urls).

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

osd: discard waiting ops when pg mapping changes

If the pg mapping changes away from us, we can safely discard messages we
have waiting for the PG to be created.

Fixes: #2013
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: flush on activate

PG::activate() can make lots of changes, most notably clean_up_local()
which deletes lots of local objects.  Those changes need to be flushed
to the fs before we start servicing requests or else we risk processing a
client read on those objects.

Fixes: #1974
Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: check readability of object corpus on 'make check'

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

add ceph-object-corpus.git submodule

mon: fix [near]full_ratio conf update

Already a value in [0,1].  Interpret as a percentage if > 1.0.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: better MonCaps test cases

Move MonCaps to libcommon.la.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: MonCap[s]

Need some better test instances for MonCaps...

Signed-off-by: Sage Weil <sage@newdream.net>

mon: fix PGMap::generate_test_instances()

Apply an incremental instead of futzing directly with members.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: refactor calc_stats()

Signed-off-by: Sage Weil <sage@newdream.net>

osd: is_zero() method for stat structs

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix ScrubMap::object ctor

Signed-off-by: Sage Weil <sage@newdream.net>

mon: make [near]full_ratio config options floats

ratio implies a real number, not a percentage.  Correct, though, if it is
> 1.0.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: initialize [near]full_ratio during create_initial(), not ctor

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: MonMap

Signed-off-by: Sage Weil <sage@newdream.net>

mon: uninline Monmap encode/decode

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: PGMap[::Incremental]

Signed-off-by: Sage Weil <sage@newdream.net>

update amazonaws xmlns to correct url

Signed-off-by: michael rodriguez <michael@newdream.net>

mds: remove IntervalTree code

Not used, not tested.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

trivial_libceph: need O_RDWR

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: -EINVAL write if not opened writable

Fixes: #1827
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: clean up ctor a bit

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: initialize initialized

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: always send scrub errors to cluster log

Some errors were going to the cluster log, some weren't.  Normalize the
output format and send them all.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: drop election messages with bad rank

The bad message came from old code pre-bfbeae68c045de76ede86ca4f72d2a760a19c84b.

Fixes: #1909
Signed-off-by: Sage Weil <sage@newdream.net>

rgw: cleanup url_decode usage

we now url_decode the relevant strings at initialization,
thus it's clear whether we need to url_decode or not later on.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: url_decode object name

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

osd: re-take the osd lock in the init error path where it's not held

The Mutex::Locker will unlock it once the function exits.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ceph-dencoder: ScrubMap[::object]

Signed-off-by: Sage Weil <sage@newdream.net>

osd: uninline osd_stat_t methods

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: coll_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: pg_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: SnapSet

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: SnapContext, SnapRealmInfo

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

move SnapContext, SnapRealmInfo to common/snap_types.{h,cc}

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: filepath

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: CompatSet

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

kill unused tstring

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

kill useless [cn]string.h

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: escape and list correctly objects that start with underscore

This should fix bug #2025.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

crush: don't BUG_ON

Fail gracefully on map errors; only BUG on code errors.

Signed-off-by: Sage Weil <sage@newdream.net>

crush: don't BUG_ON within crush_choose

It's very hard to recover from an invalid crushmap if mons fail
assertions while processing the map, and osds crash while advancing
past an already-fixed map.  Skip such broken rules instead of
aborting.

Signed-off-by: Alexandre Oliva <oliva@lsd.ic.unicamp.br>

client: init/shutdown objecter in init/shutdown

Not in mount/unmount, and don't do shutdown() twice!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: show full status in ceph health

HEALTH_WARN when nearfull, HEALTH_ERROR when full.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>
Signed-off-by: Sage Weil <sage@newdream.net>

osd: signal dispatch_cond on ms_dispatch completion

There may be another dispatch thread waiting on this cond; we do need to
signal it!

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: use request uri if script name is empty

this was required for some nginx configuration

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

osd: reorder PG recovery_state initialization

The state machine state constructors print stuff to the logs, and the
PG::gen_prefix() includes all kinds of PG fields in that.  Move the
recovery_state member to the end of the class so that all previous fields
are initialized when that happens.  This makes valgrind shut up.

Some more deliberate tidying of the PG members and methods would also
be nice...

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test/encoding/readable.sh: nicer output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: more helpful error message for messages

If the type doesn't match, share what it was vs what you expected.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

messages: set type in default constructor

ceph-dencoder wants this to verify it decoded the correct message type.
Not that it is likely to happen, but let's be pendantic about it anyway.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

pick object from random osd for primary recovery

When recovering a primary, try the osds that have a copy of the object
in random order, rather than preferring the lowest-numbered.

Signed-off-by: Alexandre Oliva <oliva@lsd.ic.unicamp.br>
Suggested-by: Samuel Just <samuel.just@dreamhost.com>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: fix message leak on receipt of undecodable message

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: add test/encoding/types.h

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip-encoding'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>
Conflicts:
src/msg/Message.h
src/osd/OSD.cc
src/osd/ReplicatedPG.cc
src/osd/ReplicatedPG.h

encoding: document ENCODE/DECODE macros

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test/encoding/readable.sh: nicer output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test/encoding/readable.sh: check all version

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix another repop->ctx->op deref

Ok this time I actually looked for more and didn't see any.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'gh/wip-objecter-initialized'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: avoid null deref of repop->ctx->op

It's optional.

Signed-off-by: Sage Weil <sage@newdream.net>

encoding: document ENCODE_DUMP throttling weirdness

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

encoding: fix DECODE_START macro

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

encoding: add DECODE_OLDEST macro

So we can (gracefully) fail to decode very old encoded versions we no
longer support.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix another issue_repop() ctx->op null deref

Signed-off-by: Sage Weil <sage@newdream.net>

check-generated.sh: do self-decode test first

This way we get a helpful error instead of silent failure on later.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

check-generated.sh: nicer output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: print errors to stderr

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: do not dereference ctx->op when NULL

We may not have an OpRequest.  Make the later check do the cast properly
when it is needed.

Signed-off-by: Sage Weil <sage@newdream.net>

encoding: better DECODE_START_LEGACY_COMPAT_LEN

- let you specify whether to decode compat and/or len
- put the argument order in the macro name so you know when you get it
  wrong

This is useful for FileJournal::header_t.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

buffer: iterator::get_remaining()

It's helpful to know how much data is remaining.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote-tracking branch 'gh/wip-osd-op-tracking'

Reviewed-by: Sage Weil <sage@newdream.net>

Merge branch 'master' into wip-encoding

Conflicts:
src/osd/OSD.cc
src/osd/PG.cc
src/osd/PG.h

common/Throttle: throttle in FIFO order

Under heavy write load from many clients, many reader threads will
be waiting in the policy throttler, all on a single condition variable.
When a wakeup is signalled, any of those threads may receive the
signal.  This increases the variance in the message processing
latency, and in extreme cases can significantly delay a message.

This patch causes threads to exit a throttler in the same order
they entered.

Signed-off-by: Jim Schutt <jaschut@sandia.gov>
Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: fix osd_recover_clone_overlap

- we need to populate data_subset
- add check in calc_head_subsets() too

Fixes 2116f012.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: use obc for size in calc_head_subsets()

No need to call stat(2) here; the caller has what we need.

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: remove obsolete fs type check

This isn't a useful check.  xfs and ext4 work too.

Fixes: #1995
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: add initialized flag to client

Do not call init() while initialized; do not call shutdown unless
initialized.

Drop incoming messages if not initialized (we are probably racing with
shutdown).

Signed-off-by: Sage Weil <sage@newdream.net>

client: let set_filer_flags clear flags, toos

ceph-syn does this...

Signed-off-by: Sage Weil <sage@newdream.net>

librados: discard incoming messages when DISCONNECTED

If we are disconnected (probably shutting down, if we are receiving a
message) then ignore anything incoming.  This avoids passing it to
partially torn down subsystems like the objecter.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: track whether initialized; add asserts

init() should be called when not initialized; shutdown() should not be
called unless initialized.  No handle_* method should be called unless
initialized.

Signed-off-by: Sage Weil <sage@newdream.net>

test_filejournal: fix warnings

Signed-off-by: Sage Weil <sage@newdream.net>

osd: mark_started() osd sub ops

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: d'oh again! Make this real exponential, not...ever-linear.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: OpRequest currently_* needs to look at latest, not hit.

D'oh!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

do_autogen.sh: -e <path> to dump encoded objects to a path

Make it easy to build with encode dumping enabled.  This is just a
convenient way to generate a large corpus of encoded objects.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

check-generated.sh: run on 'make check'

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>