osd: new encoding for PG::Interval

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for PG::OndiskLog

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objectstore: new encoding for Transaction

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for ScrubMap

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for object_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for watch_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for SnapSet

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_missing_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_log_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_log_entry_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_history_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_history_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pool_stat_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_stat_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for object_stat_collection_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for object_stat_sum_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_pool_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pool_snap_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for OSDSuperblock

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: move object_locator_t to osd_types.{h,cc}

That's a better home.  Also add to ceph-dencoder.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: add osd_reqid_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: add hobject_t

Move to a separate file in os/, since this is an ObjectStore related
object.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: add/use pg_create_t ctor

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: add pg_create_t (formerly MOSDPGCreate::create_rec)

Signed-off-by: Sage Weil <sage@newdream.net>

test/encoding/readable.sh: no \t

Not sure why this sometimes works and sometimes doesn't.  Maybe it's a
bashism?

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-corpus'

ceph-dencoder: add Log{Entry,EntryKey,Summary}

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: wait to start op if other ops are in line

We can have a sequence like:

- commit_start, blocked=true
- op_start thread A gets in line
- op_start thread B gets in line
- commit finished, blocked=false
- thread A goes
- op_start thread C sees blocked=false and continues
-> order broken

If there are people in line from a previous block, we need to get in line,
even if blocked == false.

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: fix op queue quiesce during commit

When I added the ordering constraint fix back in 259c509a I got the
check backwards.  We want to wait if we are blocked OR we are not in the
front of the line (i.e., proceed if we are not blocked AND first in line).

Fixes: #2046
Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'gh/wip-types'

Merge remote branch 'gh/wip-stuck-in-backfill'

Reviewed-by: Sage Weil <sage.weil@dreamhost.com>

ReplicatedPG: don't count deletions as ops

Counting them as ops but not requeueing the pg for recovery causes
backfill to stall when only deletions are sent in
recover_backfill(). Deletions are cheap and don't need to be acked, so
we can simply stop counting them as ops.

Fixes: #2044
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: don't remove pg from recovery queue if not enough recovery ops started

The pg has already been dequeued at the beginning of do_recovery(),
and it requeues itself only if it starts a new recovery op.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: don't treat plus as a space in url decode

Any special character encoding should be done through %hex. The
plus sign is a valid character in object names, and in user id
(when used in signed urls).

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

osd: discard waiting ops when pg mapping changes

If the pg mapping changes away from us, we can safely discard messages we
have waiting for the PG to be created.

Fixes: #2013
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: flush on activate

PG::activate() can make lots of changes, most notably clean_up_local()
which deletes lots of local objects.  Those changes need to be flushed
to the fs before we start servicing requests or else we risk processing a
client read on those objects.

Fixes: #1974
Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: check readability of object corpus on 'make check'

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

add ceph-object-corpus.git submodule

mon: fix [near]full_ratio conf update

Already a value in [0,1].  Interpret as a percentage if > 1.0.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: better MonCaps test cases

Move MonCaps to libcommon.la.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: MonCap[s]

Need some better test instances for MonCaps...

Signed-off-by: Sage Weil <sage@newdream.net>

mon: fix PGMap::generate_test_instances()

Apply an incremental instead of futzing directly with members.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: refactor calc_stats()

Signed-off-by: Sage Weil <sage@newdream.net>

osd: is_zero() method for stat structs

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix ScrubMap::object ctor

Signed-off-by: Sage Weil <sage@newdream.net>

mon: make [near]full_ratio config options floats

ratio implies a real number, not a percentage.  Correct, though, if it is
> 1.0.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: initialize [near]full_ratio during create_initial(), not ctor

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: MonMap

Signed-off-by: Sage Weil <sage@newdream.net>

mon: uninline Monmap encode/decode

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: PGMap[::Incremental]

Signed-off-by: Sage Weil <sage@newdream.net>

update amazonaws xmlns to correct url

Signed-off-by: michael rodriguez <michael@newdream.net>

mds: remove IntervalTree code

Not used, not tested.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

trivial_libceph: need O_RDWR

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: -EINVAL write if not opened writable

Fixes: #1827
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: clean up ctor a bit

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: initialize initialized

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: always send scrub errors to cluster log

Some errors were going to the cluster log, some weren't.  Normalize the
output format and send them all.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: drop election messages with bad rank

The bad message came from old code pre-bfbeae68c045de76ede86ca4f72d2a760a19c84b.

Fixes: #1909
Signed-off-by: Sage Weil <sage@newdream.net>

rgw: cleanup url_decode usage

we now url_decode the relevant strings at initialization,
thus it's clear whether we need to url_decode or not later on.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: url_decode object name

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

osd: re-take the osd lock in the init error path where it's not held

The Mutex::Locker will unlock it once the function exits.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ceph-dencoder: ScrubMap[::object]

Signed-off-by: Sage Weil <sage@newdream.net>

osd: uninline osd_stat_t methods

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: coll_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: pg_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: SnapSet

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: SnapContext, SnapRealmInfo

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

move SnapContext, SnapRealmInfo to common/snap_types.{h,cc}

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: filepath

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: CompatSet

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

kill unused tstring

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

kill useless [cn]string.h

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: escape and list correctly objects that start with underscore

This should fix bug #2025.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

crush: don't BUG_ON

Fail gracefully on map errors; only BUG on code errors.

Signed-off-by: Sage Weil <sage@newdream.net>

crush: don't BUG_ON within crush_choose

It's very hard to recover from an invalid crushmap if mons fail
assertions while processing the map, and osds crash while advancing
past an already-fixed map.  Skip such broken rules instead of
aborting.

Signed-off-by: Alexandre Oliva <oliva@lsd.ic.unicamp.br>

client: init/shutdown objecter in init/shutdown

Not in mount/unmount, and don't do shutdown() twice!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: show full status in ceph health

HEALTH_WARN when nearfull, HEALTH_ERROR when full.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>
Signed-off-by: Sage Weil <sage@newdream.net>

osd: signal dispatch_cond on ms_dispatch completion

There may be another dispatch thread waiting on this cond; we do need to
signal it!

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: use request uri if script name is empty

this was required for some nginx configuration

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

osd: reorder PG recovery_state initialization

The state machine state constructors print stuff to the logs, and the
PG::gen_prefix() includes all kinds of PG fields in that.  Move the
recovery_state member to the end of the class so that all previous fields
are initialized when that happens.  This makes valgrind shut up.

Some more deliberate tidying of the PG members and methods would also
be nice...

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test/encoding/readable.sh: nicer output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: more helpful error message for messages

If the type doesn't match, share what it was vs what you expected.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

messages: set type in default constructor

ceph-dencoder wants this to verify it decoded the correct message type.
Not that it is likely to happen, but let's be pendantic about it anyway.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

pick object from random osd for primary recovery

When recovering a primary, try the osds that have a copy of the object
in random order, rather than preferring the lowest-numbered.

Signed-off-by: Alexandre Oliva <oliva@lsd.ic.unicamp.br>
Suggested-by: Samuel Just <samuel.just@dreamhost.com>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: fix message leak on receipt of undecodable message

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: add test/encoding/types.h

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip-encoding'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>
Conflicts:
src/msg/Message.h
src/osd/OSD.cc
src/osd/ReplicatedPG.cc
src/osd/ReplicatedPG.h

encoding: document ENCODE/DECODE macros

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test/encoding/readable.sh: nicer output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test/encoding/readable.sh: check all version

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix another repop->ctx->op deref

Ok this time I actually looked for more and didn't see any.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'gh/wip-objecter-initialized'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: avoid null deref of repop->ctx->op

It's optional.

Signed-off-by: Sage Weil <sage@newdream.net>

encoding: document ENCODE_DUMP throttling weirdness

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

encoding: fix DECODE_START macro

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

encoding: add DECODE_OLDEST macro

So we can (gracefully) fail to decode very old encoded versions we no
longer support.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix another issue_repop() ctx->op null deref

Signed-off-by: Sage Weil <sage@newdream.net>

check-generated.sh: do self-decode test first

This way we get a helpful error instead of silent failure on later.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

check-generated.sh: nicer output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: print errors to stderr

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>