osd: new osd_peer_stat_t shell type

We weren't using this, and it had broken (raw) encoding.  The constructor
also didn't initialize fields properly.

Clear out the struct and use the new encoding scheme, so we can cleanly
add fields moving forward.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/btrfs/.gitignore: ignore targets

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: don't use SCRIPT_NAME and QUERY_STRING vars

REQUEST_URI holds everything we need, and it's encoded correctly.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Revert "rgw: don't treat plus as a space in url decode"

This reverts commit a6d7629c177fbab722a7a0c7f861caf91ff92deb.

osd: filter trimming|purged snaps out of op SnapContext

We can receive an op with an old SnapContext that includes snaps that we've
already trimmed or are in the process of trimming.  Filter them out!
Otherwise we will recreate and add links into collections we've already
marked as removed, and we'll get things like ENOTEMPTY when we try to
remove them.  Or just leave them laying around.

Fixes: #1949
Signed-off-by: Sage Weil <sage@newdream.net>

mon: add {mon,quorum}_status admin socket commands

These dump some json with the current monitor/quorum status.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: move quorum_status into helper

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: move mon_status into a helper

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

init-ceph, mkcephfs: try 'btrfs device scan' before 'btrfsctl -a'

Fixes: #2023
Reported-by: Wido den Hollander <wido@widodh.nl>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix MOSDPGCreate version setting

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'gh/wip-osd-encoding'

osd: queue pg removal under pg's epoch

The PG may be doing work relative to a different epoch than what the osd
has.  Make sure the PG removal message is queued under that epoch to avoid
confusing/crashing the recipient like so:

2012-02-10 23:26:35.691793 7f387281f700 osd.3 514 queue_pg_for_deletion: 0.0
osd/OSD.cc: In function 'void OSD::handle_pg_remove(OpRequest*)' thread 7f387281f700 time 2012-02-10 23:26:35.691820
osd/OSD.cc: 4860: FAILED assert(pg->get_primary() == m->get_source().num())

Signed-off-by: Sage Weil <sage@newdream.net>

osd: check for valid snapc _before_ doing op work

Check this early to avoid wasting effort, or causing side-effects from
do_osd_op_effects().

Signed-off-by: Sage Weil <sage@newdream.net>

osd: some cleanup

Signed-off-by: Sage Weil <sage@newdream.net>

mon: validate osmdap input

And clean up some error return paths while we're here.

Fixes: #1493
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: objects can contain '%'

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

mon: fix MMonElection encoding version

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: LingerOp is refcounted

this should fix Bug #2050, where a linger op was used after being freed.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Merge branch 'wip-encoding'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

qa/btrfs/create_async_snap

Stupid tool to call the async snap ioctl.  Until the btrfs tool does it.

Signed-off-by: Sage Weil <sage@newdream.net>

messages: populate header.version in constructor

Define a HEAD_VERSION and COMPAT_VERSION for any versioned message.  Pass
to Message constructor so that it is always initialized, even from the
the default constructor.  That's needed because we use that to check
decoding compatibility when receiving/decoding messages.

If we are conditionally encoding an old version, explicitly set
header.version in encode_payload().

We also set compat_version to demonstrate what will happen for future
revisions.  In this case, it's moot, because no old code understands
compat_version yet: nobody with old decode code will see these values
anyway.  But use this opportunity to demonstrate how it would be used in
the future.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ReplicatedPG: don't put the op on -EAGAIN

EAGAIN indicates that the op is
waiting_for_missing or waiting_for_degraded

Reviewed-by: Greg Farnum <greg.farnum@dreamhost.com>
Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

msg: check compat_version before decoding

If the newly constructed message's version is older than the
compat_version, don't even try to decode; just fail.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: populate compat_version for encoded messages

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: include compat_version in version header

header.version is the version we encoded.
header.compat_version is the oldest version of code that can decode it.

If the value is 0, we don't know anything about backward compatibility.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

new encoding for Log{Entry,Summary}

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

os: new encoding for hobject_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_create_t

There was no version encoding previously, so this is an incompatible
change.  Fortunately this type is only used in one place, MOSDPGCreate,
so we'll rev that encoding and compensate there.  All is well!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for osd_stat_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for object_locator_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for osd_reqid_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new ScrubMap::object encoding

Signed-off-by: Sage Weil <sage@newdream.net>

mon: set last_changed when creating new pgs

This will help us identify PGs that are stuck in creating state.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: set last_unstale when marking PGs stale

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: include state timestamps, mapping_epoch in pg_stat_t

Track the time when the pg state last changed (or was refreshed) in
interesting ways.

Also track the epoch when the mapping last changed (same_interval_since).

Signed-off-by: Sage Weil <sage@newdream.net>

osd: new encoding for PG::Interval

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for PG::OndiskLog

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objectstore: new encoding for Transaction

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for ScrubMap

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for object_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for watch_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for SnapSet

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_missing_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_log_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_log_entry_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_history_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_history_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pool_stat_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_stat_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for object_stat_collection_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for object_stat_sum_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pg_pool_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for pool_snap_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: new encoding for OSDSuperblock

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: move object_locator_t to osd_types.{h,cc}

That's a better home.  Also add to ceph-dencoder.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: add osd_reqid_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: add hobject_t

Move to a separate file in os/, since this is an ObjectStore related
object.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: add/use pg_create_t ctor

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: add pg_create_t (formerly MOSDPGCreate::create_rec)

Signed-off-by: Sage Weil <sage@newdream.net>

test/encoding/readable.sh: no \t

Not sure why this sometimes works and sometimes doesn't.  Maybe it's a
bashism?

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-corpus'

ceph-dencoder: add Log{Entry,EntryKey,Summary}

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: wait to start op if other ops are in line

We can have a sequence like:

- commit_start, blocked=true
- op_start thread A gets in line
- op_start thread B gets in line
- commit finished, blocked=false
- thread A goes
- op_start thread C sees blocked=false and continues
-> order broken

If there are people in line from a previous block, we need to get in line,
even if blocked == false.

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: fix op queue quiesce during commit

When I added the ordering constraint fix back in 259c509a I got the
check backwards.  We want to wait if we are blocked OR we are not in the
front of the line (i.e., proceed if we are not blocked AND first in line).

Fixes: #2046
Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'gh/wip-types'

Merge remote branch 'gh/wip-stuck-in-backfill'

Reviewed-by: Sage Weil <sage.weil@dreamhost.com>

ReplicatedPG: don't count deletions as ops

Counting them as ops but not requeueing the pg for recovery causes
backfill to stall when only deletions are sent in
recover_backfill(). Deletions are cheap and don't need to be acked, so
we can simply stop counting them as ops.

Fixes: #2044
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: don't remove pg from recovery queue if not enough recovery ops started

The pg has already been dequeued at the beginning of do_recovery(),
and it requeues itself only if it starts a new recovery op.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: don't treat plus as a space in url decode

Any special character encoding should be done through %hex. The
plus sign is a valid character in object names, and in user id
(when used in signed urls).

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

osd: discard waiting ops when pg mapping changes

If the pg mapping changes away from us, we can safely discard messages we
have waiting for the PG to be created.

Fixes: #2013
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: flush on activate

PG::activate() can make lots of changes, most notably clean_up_local()
which deletes lots of local objects.  Those changes need to be flushed
to the fs before we start servicing requests or else we risk processing a
client read on those objects.

Fixes: #1974
Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: check readability of object corpus on 'make check'

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

add ceph-object-corpus.git submodule

mon: fix [near]full_ratio conf update

Already a value in [0,1].  Interpret as a percentage if > 1.0.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: better MonCaps test cases

Move MonCaps to libcommon.la.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: MonCap[s]

Need some better test instances for MonCaps...

Signed-off-by: Sage Weil <sage@newdream.net>

mon: fix PGMap::generate_test_instances()

Apply an incremental instead of futzing directly with members.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: refactor calc_stats()

Signed-off-by: Sage Weil <sage@newdream.net>

osd: is_zero() method for stat structs

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix ScrubMap::object ctor

Signed-off-by: Sage Weil <sage@newdream.net>

mon: make [near]full_ratio config options floats

ratio implies a real number, not a percentage.  Correct, though, if it is
> 1.0.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: initialize [near]full_ratio during create_initial(), not ctor

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: MonMap

Signed-off-by: Sage Weil <sage@newdream.net>

mon: uninline Monmap encode/decode

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: PGMap[::Incremental]

Signed-off-by: Sage Weil <sage@newdream.net>

update amazonaws xmlns to correct url

Signed-off-by: michael rodriguez <michael@newdream.net>

mds: remove IntervalTree code

Not used, not tested.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

trivial_libceph: need O_RDWR

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: -EINVAL write if not opened writable

Fixes: #1827
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: clean up ctor a bit

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: initialize initialized

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: always send scrub errors to cluster log

Some errors were going to the cluster log, some weren't.  Normalize the
output format and send them all.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: drop election messages with bad rank

The bad message came from old code pre-bfbeae68c045de76ede86ca4f72d2a760a19c84b.

Fixes: #1909
Signed-off-by: Sage Weil <sage@newdream.net>

rgw: cleanup url_decode usage

we now url_decode the relevant strings at initialization,
thus it's clear whether we need to url_decode or not later on.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: url_decode object name

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

osd: re-take the osd lock in the init error path where it's not held

The Mutex::Locker will unlock it once the function exits.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ceph-dencoder: ScrubMap[::object]

Signed-off-by: Sage Weil <sage@newdream.net>

osd: uninline osd_stat_t methods

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: coll_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: pg_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>