do_autogen.sh: -e <path> to dump encoded objects to a path

Make it easy to build with encode dumping enabled.  This is just a
convenient way to generate a large corpus of encoded objects.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

check-generated.sh: run on 'make check'

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: pg_stat_t: fix member initialization

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix PG::Interval member initialization

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: finalize crush after building simple map

This ensures that max_devices gets calculated (and thus encoded) properly.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: make test instnaces deterministic

current time can vary

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

import-generated.sh: fix to use ceph-dencoder syntax

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: fix ctor

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph_context: initialize member var

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PGLSResponse -> pg_ls_response_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Missing -> pg_missing_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Log -> pg_log_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Log::Entry -> pg_log_entry_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Query -> pg_query_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Info -> pg_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Info::History -> pg_history_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Info[::History] dump, test instances

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: remove message type dups

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: generate test instances on heap

Some objects aren't copyable (OSDMap contains CrushWrapper), but we'd
still like to programmatically generate test instances.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: dump, test instances for PG::Interval

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: dump, instances for PG::OndiskLog

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: use g_ceph_context

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: OSDMap and osd_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: test instances for osd_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: test instances

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: normalist encode/decode

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: add all message types

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: add missing #includes for messages

And remove that unused max() macro.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: dump messages via build option

Dump encoded messages to ENCODE_DUMP when it is defined, just as we do with
the regular encode function.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: handle messages

Dump for now uses the string rendering function, and that's it.  Maybe
we'll write proper dump methods for all of these messages someday.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: implement Message::dump()

Just wrap print() for now.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: go const-crazy on messages

- get_type_name()
- print()
and all the random crap they call.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

paxos: explicitly pass in send timestamp

This is cleaner.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: no cct for decode_payload

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: use absolute times for message encoding

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: make decode cct optional

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: no cct needed for message encoding

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mdsmap: move member initialization to monitor create_initial()

The dependence on cct/conf here was totally wrong.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: use explicit feature argument instead of Connection*

Use the new argument.  Don't rely on Connection *connection being defined.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: pass features explicitly into message encoders

Avoid using the connection reference; pass it in explicitly instead.  This
will make ceph-dencoder's life a bit easier.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'gh/master' into wip-encoding

mon: trim old auth states

These aren't exposed outside the monitor, so we really only keep them
around to assist in mon recovery.  Give ourselves a healthy margin over
the max join drift for that.

Fixes: #2000
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: fix rollback when current/ missing entirely

This can happen when we are starting, rolling back, remove current/, and
then fail before we snapshot a snap_ into place.

Most of the logic was already in place for this; we tried to fix it in
cd2dedd7d190a43a6be50a7f18849fe0123c72bc but missed this piece.

Fixes: #1999
Signed-off-by: Sage Weil <sage@newdream.net>

osd: reset pgstats timer when we reopen monitor session

Otherwise we'll reopen every second from here on out, without giving the
new session a chance to start up and do it's thing.

Signed-off-by: Sage Weil <sage@newdream.net>

clock: ignore clock_offset if cct is NULL

This is helpful e.g. from assert.

Signed-off-by: Sage Weil <sage@newdream.net>

client: do not send release to down mds

We can have a session with state where the mds is not up; don't blindly
send a message or we can get

./mds/MDSMap.h: In function 'const entity_inst_t MDSMap::get_inst(int)', in thread '0x7f092aad1910'
./mds/MDSMap.h: 465: FAILED assert(up.count(m))
 ceph version 0.35-6-g6eb8862 (commit:6eb8862e91d142451e256aaa02b34c81a4f21dea)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x70) [0x71f11a]
 2: (MDSMap::get_inst(int)+0x4b) [0x6dc191]
 3: (Client::flush_cap_releases()+0x94) [0x677e60]
 4: (Client::tick()+0x1f0) [0x690adc]
 5: (C_C_Tick::finish(int)+0x1c) [0x6f3fbe]
 6: (SafeTimer::timer_thread()+0x2c5) [0x6fbfe5]
 7: (SafeTimerThread::entry()+0x19) [0x6fe399]
 8: (Thread::_entry_func(void*)+0x20) [0x72e944]
 9: /lib/libpthread.so.0 [0x7f092dea573a]
 10: (clone()+0x6d) [0x7f092cba169d]

with a map like

$ ./ceph mds dump 85
2012-01-28 09:37:19.251946 mon <- [mds,dump,85]
2012-01-28 09:37:19.252618 mon.1 -> 'dumped mdsmap epoch 85' (0)
epoch   85
flags   0
created 2012-01-28 09:24:42.411202
modified        2012-01-28 09:28:45.093301
tableserver     0
root    0
session_timeout 60
session_autoclose       300
last_failure    0
last_failure_osd_epoch  18
compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
max_mds 1
in      0
up      {}
failed  0
stopped
data_pools      [0]
metadata_pool   1

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'stable'

signal: use _exit() on SIGTERM

No need to call onexit handlers, static dtors, whatever.

This may help with #1996 and #1549.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test: add script for checking admin socket 'objecter_requests' output

Just a couple internal consistency checks for now. More specific ones
would depend on workload.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

objecter: add an admin socket command to get in-flight requests

Fixes: #1881
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

admin socket: increase debug level for successful requests

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

admin socket: add include guard

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

CephContext: add method for retrieving admin socket

This is needed to allow higher layers in the stack to add admin socket
commands.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge branch 'wip-pg-stale'

mon: stale pgs -> HEALTH_WARN

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: mark pgs stale in pg_map if primary osd is down

This alerts the administrator when all OSDs for a PG have failed and the
monitor doesn't receive any further updates.  Otherwise we may continue
to think a pg is active+clean when it is in fact offline.

Fixes: #1993
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: add STALE pg state bit

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

v0.41

objector: document Objecter::init_ops()

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objecter: fix out_* initialization

This looks more like the real cause for #1986.  Op ctor gets a vector of
ops but out_* aren't initialized to match.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: dump offending transaction on any error

Clean this code up to explicitly whitelist what is ok so that the flow is
less annoying to follow/maintain, and so that we dump the transaction
contents on whitelisted errors.

Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

objecter: warn when OSD returns mismatched op vector

The osd shouldn't do this (even though we should tolerate it).

Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

objecter: fix bounds checking on op reply demuxing

We can't assume that the size of out_ops (from the reply) matches the
op->out_* vectors from our request state.  In particular, the out_ops might
be shorter than what we sent the OSD if the OSD was sloppy.  Check them.

We can assume that op->ops and op->out_* all match; assert as much in
op_submit().

Fixes: #1986
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: remove test assert

Grr!

Signed-off-by: Sage Weil <sage@newdream.net>

assert: include timestamp

Also drop quotes around thread id.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'master' into wip-encoding

Conflicts:
src/osd/osd_types.h

filestore: fix typo

Grr

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip-kb'

Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

filestore: zero btrfs vol_args prior to ioctl

Just to be paranoid.  Nothing we haven't set *should* affect the ABI,
but...

Always do this immediately after declaration so that we catch everything.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'upstream/wip-osd-clone-obc'

mon: num_kb -> num_bytes in cluster perfcounters

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: remove num_kb from object_stat_sum_t stats

This is redundant--we can just use num_bytes.  If we're worried about the
per-object overhead or rounding, we can factor in some overhead based on
num_objects.

And, the kb accounting has a bug (#1988).

Avoid changing the encoding at all for now.  Next time the encoding changes
we'll drop the old field.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: improve object context debug output

Include pointer.  This may help with #1979.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: track obc for clone from log replay

We need to keep an in-memory obc to track the state of the in-flight io
to disk.  This is analogous to when an object is pushed + written, and we
can share the same completion function.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: set object_info_t::oid properly when recovering clones

I saw a case (#1973) where the clone had the oid set to the head.  That is
clearly wrong.  Not sure what damage this caused.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'gh/wip-filestore-errors'

package *.py* files

Some post-install rpmbuild defaults byte-compile all packaged python
files, so don't bother removing the .pyc files, and package .py* to
get both .pyo and .pyc.  It wastes a tiny little bit of space, but it
makes the spec file portable across a wider range of rpm and python
configurations.

Signed-off-by: Alexandre Oliva <oliva@lsd.ic.unicam.br>
Signed-off-by: Sage Weil <sage@newdream.net>

librbd: don't infinite loop when header is too large

Since snapshots are currently stored at the end of the header, having
many snapshots made the header larger than the read size, resulting in
an infinite loop when the offset was not changed.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: data_subset may be empty during sub_op_push

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

filestore: fix non-::-prefixed close

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

filestore: add debugging to each error case in lfn_open

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

filestore: TEMP_FAILURE_RETRY on ::close(2)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: return -errno from lfn_open

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: audit + clean up error checks

- use temp var for errno
- in general return -errno from helpers

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge commit '9dc7b9233b985bf859751fc89a5b02253e829836'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: fix warning

rgw/rgw_rest.cc:258: warning: comparison between signed and unsigned integer expressions

Signed-off-by: Sage Weil <sage@newdream.net>

ceph: bail out on first failing command

Signed-off-by: Sage Weil <sage@newdream.net>

ceph: don't write output on error

Accumulate all output, and write it at the end.  This way we can avoid
writing it if any of the commands fail.

Fixes: #1954
Signed-off-by: Sage Weil <sage@newdream.net>

.gitignore: ceph-dencoder

Signed-off-by: Sage Weil <sage@newdream.net>

osd: ignore MInfoRec, MNotifyRec in WaitActingChange

We should ignore logs, infos, and notifies while we are waiting for the
map to change.  Peering has reached a dead-end (we need acting to change)
and we will redo our work when that happens.  That includes the replicas
resending notifies.

Fixes: #1958
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

rgw: fix warning in 32bit arch

ceph-dencoder: needs ceph_ver.h dependency

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

pg: unindex entries when clearing or removing from the log

Leaving the index around could cause use of the indexes to access
freed memory.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: do not clobber log on backfill progress update

This is unnecessary and counterproductive, since the log is used to detect
dup ops.  It's an artifact of an earlier backfill iteration that didn't
preserve the log on the backfill target.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

features: #include ceph_features directly where needed

Less rebuild time when touched.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objectstore: remove unused setattr variants

No callers.  Fugly.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: initialize fields in watch_info_t constructor

Go-go unit tests!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: fix up usage a bit

- verb_noun
- "in-memory"
- 1-based test index

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objectstore: drop unused Transaction::p

This should have gone away when we added the iterator.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: reenable generated types

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>