test/encoding/readable.sh: check all version

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

encoding: document ENCODE_DUMP throttling weirdness

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

encoding: fix DECODE_START macro

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

encoding: add DECODE_OLDEST macro

So we can (gracefully) fail to decode very old encoded versions we no
longer support.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

check-generated.sh: do self-decode test first

This way we get a helpful error instead of silent failure on later.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

check-generated.sh: nicer output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: print errors to stderr

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

encoding: better DECODE_START_LEGACY_COMPAT_LEN

- let you specify whether to decode compat and/or len
- put the argument order in the macro name so you know when you get it
  wrong

This is useful for FileJournal::header_t.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

buffer: iterator::get_remaining()

It's helpful to know how much data is remaining.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'master' into wip-encoding

Conflicts:
src/osd/OSD.cc
src/osd/PG.cc
src/osd/PG.h

filestore: remove obsolete fs type check

This isn't a useful check.  xfs and ext4 work too.

Fixes: #1995
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test_filejournal: fix warnings

Signed-off-by: Sage Weil <sage@newdream.net>

do_autogen.sh: -e <path> to dump encoded objects to a path

Make it easy to build with encode dumping enabled.  This is just a
convenient way to generate a large corpus of encoded objects.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

check-generated.sh: run on 'make check'

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: pg_stat_t: fix member initialization

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix PG::Interval member initialization

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: finalize crush after building simple map

This ensures that max_devices gets calculated (and thus encoded) properly.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: make test instnaces deterministic

current time can vary

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

import-generated.sh: fix to use ceph-dencoder syntax

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: fix ctor

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph_context: initialize member var

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PGLSResponse -> pg_ls_response_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Missing -> pg_missing_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Log -> pg_log_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Log::Entry -> pg_log_entry_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Query -> pg_query_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

cls_rgw: update bucket index when deleting object (with pending)

Bug #2012. Racing delete with other operations (update or another
delete) failed to update the bucket index.

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

osd: PG::Info -> pg_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Info::History -> pg_history_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Info[::History] dump, test instances

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: remove message type dups

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: generate test instances on heap

Some objects aren't copyable (OSDMap contains CrushWrapper), but we'd
still like to programmatically generate test instances.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'gh/wip-divergent-backfill'

Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

osd: dump, test instances for PG::Interval

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: dump, instances for PG::OndiskLog

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: use g_ceph_context

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: OSDMap and osd_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: test instances for osd_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: test instances

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: normalist encode/decode

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: add all message types

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: add missing #includes for messages

And remove that unused max() macro.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: dump messages via build option

Dump encoded messages to ENCODE_DUMP when it is defined, just as we do with
the regular encode function.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix assignment in PG::rewind_divergent_log()

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote-tracking branch 'gh/wip-journal-crc'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

qa: test_backfill.sh: take osd.0 down

Mark this down to
1- trigger the WaitActingChange vs osd down race, and
2- help trigger a divergnet log when osd.2 is blackholed+restarted during
   backfill.  e.g.,

./ceph -- tell osd.1 injectargs '--filestore-blackhole' ; sleep 10 ; ./init-ceph restart osd.1

Signed-off-by: Sage Weil <sage@newdream.net>

osd: restart peering if requesting acting osd goes down

If we request an acting set, we need to restart peering if one of the
requested nodes goes down.  This prevents a deadlock where we get stuck
in WaitActingChange because we have [a,b], want [a,b,c], but c is down and
our up and acting don't actually change.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: rename recovery event NeedNewMap -> NeedActingChange

This is more precise.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: use RecoveryContext transaction, finishers on recovery completion

We should use the enclosing transaction and finisher list here.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: test_backfill.sh: limit pg log length so we trigger backfill

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix divergent backfill targets

During peering, a previous backfill target may have a slightly newer
last_update than the other options, but it will not be chosen because it
is incomplete.  That caused a failed assert during activate() (#1983).

To fix, we remove the bad assert, and then fix merge_log() so that the
replica/backfill target will trim its divergent entries when it gets the
activation MLogRec.  We also fix the handling of MInfoRec, as that can
trigger the same analogous condition.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: implement filestore_blackhole hook

If true, we'll drop any new transactions on the floor. Useful for
triggering failure conditions (e.g., prior to killing ceph-osd itself, to
ensure some operations don't reach the local disk).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: should remove bucket dir instead of sending intent

that was really useless, and also bucket cleanup was broken anyway.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

librados: fix a leak

watch notification message was missing a ->put()

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

osd: disable clone overlap for push/pull

There is a bug in the push/pull code.  Disable the recovery smarts by
default until we fix #2002.

There is currently a race (in the callers) where:
 - an adjacent clone is missing
 - we (calculate some clone overlap? and) start pulling
 - we get adjacent clone
 - we get push, calc a different overlap, and then get confused.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'gh/wip-warnings'

mon: make 'osd [out|in|down]' succeed if already whatever

If we want something out and it is already out, succeed.  This makes the
client command succeed if there is a transient error and it gets resent.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: handle messages

Dump for now uses the string rendering function, and that's it.  Maybe
we'll write proper dump methods for all of these messages someday.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: implement Message::dump()

Just wrap print() for now.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: go const-crazy on messages

- get_type_name()
- print()
and all the random crap they call.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

paxos: explicitly pass in send timestamp

This is cleaner.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: no cct for decode_payload

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: use absolute times for message encoding

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: make decode cct optional

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: no cct needed for message encoding

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mdsmap: move member initialization to monitor create_initial()

The dependence on cct/conf here was totally wrong.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: use explicit feature argument instead of Connection*

Use the new argument.  Don't rely on Connection *connection being defined.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: pass features explicitly into message encoders

Avoid using the connection reference; pass it in explicitly instead.  This
will make ceph-dencoder's life a bit easier.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'gh/master' into wip-encoding

qa: encoding: silence warning

This is cheating, but we always use this class with int types, so it makes
this go away:

warning: test/encoding.cc:79:20: ‘*((void*)(& tu)+4).ConstructorCounter::data’ may be used uninitialized in this function [-Wuninitialized]

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: test/gather fix warning

warning: test/gather.cc:29:222: passing NULL to non-pointer argument 3 of ‘static testing::AssertionResult testing::internal::EqHelper::Compare(const char*, const char*, const T1&, T2*) [with T1 = long int, T2 = C_Gather]’ [-Wconversion-null]

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: test/rados-api/list fix warning

warning: test/rados-api/list.cc:43:156: converting ‘false’ to pointer type for argument 1 of ‘char testing::internal::IsNullLiteralHelper(testing::internal::Secret*)’ [-Wconversion-null]

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test_ipaddr: reverse ASSERT_EQ order

Make these warnings go away:

warning: test/test_ipaddr.cc:217:156: converting ‘false’ to pointer type for argument 1 of ‘char testing::internal::IsNullLiteralHelper(testing::internal::Secret*)’ [-Wconversion-null]

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: remove unused var

warning: osd/PG.cc:1331:20: variable 'plu' set but not used [-Wunused-but-set-variable]

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

admin_socket: fix uninit warning

warning: common/admin_socket_client.cc:166:19: 'socket_fd' may be used uninitialized in this function [-Wuninitialized]

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: trim old auth states

These aren't exposed outside the monitor, so we really only keep them
around to assist in mon recovery.  Give ourselves a healthy margin over
the max join drift for that.

Fixes: #2000
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: fix rollback when current/ missing entirely

This can happen when we are starting, rolling back, remove current/, and
then fail before we snapshot a snap_ into place.

Most of the logic was already in place for this; we tried to fix it in
cd2dedd7d190a43a6be50a7f18849fe0123c72bc but missed this piece.

Fixes: #1999
Signed-off-by: Sage Weil <sage@newdream.net>

osd: reset pgstats timer when we reopen monitor session

Otherwise we'll reopen every second from here on out, without giving the
new session a chance to start up and do it's thing.

Signed-off-by: Sage Weil <sage@newdream.net>

clock: ignore clock_offset if cct is NULL

This is helpful e.g. from assert.

Signed-off-by: Sage Weil <sage@newdream.net>

filejournal: add corruption test to check crc checking code

Verify that the journal replay rejects a corrupted journal entry.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filejournal: assume gibberish flags imply none

Old journals didn't properly initialize the flags (oops).  Assume that
any bits besides the first 2 imply no flags.

Make note that this hack needs to be removed after some time has passed,
but well before these new flags are used.  Or, such use should be
accompanied by a full header format rev and incompatibility.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filejournal: include crc in entry header/footer

Use the unused flags field for this.  Previously it was always 0, so this
lets us skip old entries on old journals and only worry about missing one
out of 2^32 corruptions.  New journals get a flag that strictly enforces
the crc check.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: test_filejournal: test lots of small writes too

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: add test_filejournal

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filejournal: fix header initialization

Make sure it's zeros to start with.  Currently flags might be gibberish!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filejournal: clean up some errno checks

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filejournal: assert submit_entry gets >0 bytes

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filejournal: initialize header before writing

Avoid writing uninitialized crap.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filejournal: move zero_buf allocation

We need header.alignment to be defined.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: do not send release to down mds

We can have a session with state where the mds is not up; don't blindly
send a message or we can get

./mds/MDSMap.h: In function 'const entity_inst_t MDSMap::get_inst(int)', in thread '0x7f092aad1910'
./mds/MDSMap.h: 465: FAILED assert(up.count(m))
 ceph version 0.35-6-g6eb8862 (commit:6eb8862e91d142451e256aaa02b34c81a4f21dea)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x70) [0x71f11a]
 2: (MDSMap::get_inst(int)+0x4b) [0x6dc191]
 3: (Client::flush_cap_releases()+0x94) [0x677e60]
 4: (Client::tick()+0x1f0) [0x690adc]
 5: (C_C_Tick::finish(int)+0x1c) [0x6f3fbe]
 6: (SafeTimer::timer_thread()+0x2c5) [0x6fbfe5]
 7: (SafeTimerThread::entry()+0x19) [0x6fe399]
 8: (Thread::_entry_func(void*)+0x20) [0x72e944]
 9: /lib/libpthread.so.0 [0x7f092dea573a]
 10: (clone()+0x6d) [0x7f092cba169d]

with a map like

$ ./ceph mds dump 85
2012-01-28 09:37:19.251946 mon <- [mds,dump,85]
2012-01-28 09:37:19.252618 mon.1 -> 'dumped mdsmap epoch 85' (0)
epoch   85
flags   0
created 2012-01-28 09:24:42.411202
modified        2012-01-28 09:28:45.093301
tableserver     0
root    0
session_timeout 60
session_autoclose       300
last_failure    0
last_failure_osd_epoch  18
compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
max_mds 1
in      0
up      {}
failed  0
stopped
data_pools      [0]
metadata_pool   1

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'stable'

signal: use _exit() on SIGTERM

No need to call onexit handlers, static dtors, whatever.

This may help with #1996 and #1549.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test: add script for checking admin socket 'objecter_requests' output

Just a couple internal consistency checks for now. More specific ones
would depend on workload.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

objecter: add an admin socket command to get in-flight requests

Fixes: #1881
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

admin socket: increase debug level for successful requests

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

admin socket: add include guard

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

CephContext: add method for retrieving admin socket

This is needed to allow higher layers in the stack to add admin socket
commands.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge branch 'wip-pg-stale'

mon: stale pgs -> HEALTH_WARN

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: mark pgs stale in pg_map if primary osd is down

This alerts the administrator when all OSDs for a PG have failed and the
monitor doesn't receive any further updates.  Otherwise we may continue
to think a pg is active+clean when it is in fact offline.

Fixes: #1993
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>