move SnapContext, SnapRealmInfo to common/snap_types.{h,cc}

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: filepath

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: CompatSet

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

kill unused tstring

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

kill useless [cn]string.h

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: escape and list correctly objects that start with underscore

This should fix bug #2025.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

crush: don't BUG_ON

Fail gracefully on map errors; only BUG on code errors.

Signed-off-by: Sage Weil <sage@newdream.net>

crush: don't BUG_ON within crush_choose

It's very hard to recover from an invalid crushmap if mons fail
assertions while processing the map, and osds crash while advancing
past an already-fixed map.  Skip such broken rules instead of
aborting.

Signed-off-by: Alexandre Oliva <oliva@lsd.ic.unicamp.br>

client: init/shutdown objecter in init/shutdown

Not in mount/unmount, and don't do shutdown() twice!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: show full status in ceph health

HEALTH_WARN when nearfull, HEALTH_ERROR when full.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>
Signed-off-by: Sage Weil <sage@newdream.net>

osd: signal dispatch_cond on ms_dispatch completion

There may be another dispatch thread waiting on this cond; we do need to
signal it!

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: use request uri if script name is empty

this was required for some nginx configuration

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

osd: reorder PG recovery_state initialization

The state machine state constructors print stuff to the logs, and the
PG::gen_prefix() includes all kinds of PG fields in that.  Move the
recovery_state member to the end of the class so that all previous fields
are initialized when that happens.  This makes valgrind shut up.

Some more deliberate tidying of the PG members and methods would also
be nice...

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test/encoding/readable.sh: nicer output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: more helpful error message for messages

If the type doesn't match, share what it was vs what you expected.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

messages: set type in default constructor

ceph-dencoder wants this to verify it decoded the correct message type.
Not that it is likely to happen, but let's be pendantic about it anyway.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

pick object from random osd for primary recovery

When recovering a primary, try the osds that have a copy of the object
in random order, rather than preferring the lowest-numbered.

Signed-off-by: Alexandre Oliva <oliva@lsd.ic.unicamp.br>
Suggested-by: Samuel Just <samuel.just@dreamhost.com>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: fix message leak on receipt of undecodable message

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: add test/encoding/types.h

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip-encoding'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>
Conflicts:
src/msg/Message.h
src/osd/OSD.cc
src/osd/ReplicatedPG.cc
src/osd/ReplicatedPG.h

encoding: document ENCODE/DECODE macros

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test/encoding/readable.sh: nicer output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test/encoding/readable.sh: check all version

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix another repop->ctx->op deref

Ok this time I actually looked for more and didn't see any.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'gh/wip-objecter-initialized'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: avoid null deref of repop->ctx->op

It's optional.

Signed-off-by: Sage Weil <sage@newdream.net>

encoding: document ENCODE_DUMP throttling weirdness

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

encoding: fix DECODE_START macro

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

encoding: add DECODE_OLDEST macro

So we can (gracefully) fail to decode very old encoded versions we no
longer support.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix another issue_repop() ctx->op null deref

Signed-off-by: Sage Weil <sage@newdream.net>

check-generated.sh: do self-decode test first

This way we get a helpful error instead of silent failure on later.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

check-generated.sh: nicer output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: print errors to stderr

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: do not dereference ctx->op when NULL

We may not have an OpRequest.  Make the later check do the cast properly
when it is needed.

Signed-off-by: Sage Weil <sage@newdream.net>

encoding: better DECODE_START_LEGACY_COMPAT_LEN

- let you specify whether to decode compat and/or len
- put the argument order in the macro name so you know when you get it
  wrong

This is useful for FileJournal::header_t.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

buffer: iterator::get_remaining()

It's helpful to know how much data is remaining.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote-tracking branch 'gh/wip-osd-op-tracking'

Reviewed-by: Sage Weil <sage@newdream.net>

Merge branch 'master' into wip-encoding

Conflicts:
src/osd/OSD.cc
src/osd/PG.cc
src/osd/PG.h

common/Throttle: throttle in FIFO order

Under heavy write load from many clients, many reader threads will
be waiting in the policy throttler, all on a single condition variable.
When a wakeup is signalled, any of those threads may receive the
signal.  This increases the variance in the message processing
latency, and in extreme cases can significantly delay a message.

This patch causes threads to exit a throttler in the same order
they entered.

Signed-off-by: Jim Schutt <jaschut@sandia.gov>
Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: fix osd_recover_clone_overlap

- we need to populate data_subset
- add check in calc_head_subsets() too

Fixes 2116f012.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: use obc for size in calc_head_subsets()

No need to call stat(2) here; the caller has what we need.

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: remove obsolete fs type check

This isn't a useful check.  xfs and ext4 work too.

Fixes: #1995
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

client: add initialized flag to client

Do not call init() while initialized; do not call shutdown unless
initialized.

Drop incoming messages if not initialized (we are probably racing with
shutdown).

Signed-off-by: Sage Weil <sage@newdream.net>

client: let set_filer_flags clear flags, toos

ceph-syn does this...

Signed-off-by: Sage Weil <sage@newdream.net>

librados: discard incoming messages when DISCONNECTED

If we are disconnected (probably shutting down, if we are receiving a
message) then ignore anything incoming.  This avoids passing it to
partially torn down subsystems like the objecter.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: track whether initialized; add asserts

init() should be called when not initialized; shutdown() should not be
called unless initialized.  No handle_* method should be called unless
initialized.

Signed-off-by: Sage Weil <sage@newdream.net>

test_filejournal: fix warnings

Signed-off-by: Sage Weil <sage@newdream.net>

osd: mark_started() osd sub ops

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: d'oh again! Make this real exponential, not...ever-linear.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: OpRequest currently_* needs to look at latest, not hit.

D'oh!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

do_autogen.sh: -e <path> to dump encoded objects to a path

Make it easy to build with encode dumping enabled.  This is just a
convenient way to generate a large corpus of encoded objects.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

check-generated.sh: run on 'make check'

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'origin/master' into wip-osd-op-tracking

Conflicts:
src/osd/ReplicatedPG.h

osd: pg_stat_t: fix member initialization

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: add check_ops_in_flight()

By default it warns on requests that are more than 30 seconds old,
using an exponential backoff of that interval.
Also add state name retrieval to OpRequest.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: fix PG::Interval member initialization

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: finalize crush after building simple map

This ensures that max_devices gets calculated (and thus encoded) properly.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: make test instnaces deterministic

current time can vary

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

import-generated.sh: fix to use ceph-dencoder syntax

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: fix ctor

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph_context: initialize member var

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: "mark" OpRequests as they move through the system.

Right now these are just informational flags which can be read out. Later
they might extend to timing information, separate lists for more precise
control over latency warnings, etc.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

PG: switch op passing interface to use OpRequest

This is all the PG/ReplicatedPG internals and the few remaining OSD callers.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: switch op passing interface to use OpRequest instead of raw Messages

This doesn't handle the PG internals yet.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: add new OpRequest struct and an xlist to track it

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: PGLSResponse -> pg_ls_response_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Missing -> pg_missing_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Log -> pg_log_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Log::Entry -> pg_log_entry_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Query -> pg_query_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

cls_rgw: update bucket index when deleting object (with pending)

Bug #2012. Racing delete with other operations (update or another
delete) failed to update the bucket index.

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

osd: PG::Info -> pg_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Info::History -> pg_history_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: PG::Info[::History] dump, test instances

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: remove message type dups

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: generate test instances on heap

Some objects aren't copyable (OSDMap contains CrushWrapper), but we'd
still like to programmatically generate test instances.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'gh/wip-divergent-backfill'

Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

osd: dump, test instances for PG::Interval

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: dump, instances for PG::OndiskLog

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: use g_ceph_context

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: OSDMap and osd_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: test instances for osd_info_t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: test instances

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdmap: normalist encode/decode

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-dencoder: add all message types

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: add missing #includes for messages

And remove that unused max() macro.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: dump messages via build option

Dump encoded messages to ENCODE_DUMP when it is defined, just as we do with
the regular encode function.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix assignment in PG::rewind_divergent_log()

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote-tracking branch 'gh/wip-journal-crc'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

msgr: Document recv_stamp and add a dispatch_stamp and throttle_wait.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

qa: test_backfill.sh: take osd.0 down

Mark this down to
1- trigger the WaitActingChange vs osd down race, and
2- help trigger a divergnet log when osd.2 is blackholed+restarted during
   backfill.  e.g.,

./ceph -- tell osd.1 injectargs '--filestore-blackhole' ; sleep 10 ; ./init-ceph restart osd.1

Signed-off-by: Sage Weil <sage@newdream.net>

osd: restart peering if requesting acting osd goes down

If we request an acting set, we need to restart peering if one of the
requested nodes goes down.  This prevents a deadlock where we get stuck
in WaitActingChange because we have [a,b], want [a,b,c], but c is down and
our up and acting don't actually change.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: rename recovery event NeedNewMap -> NeedActingChange

This is more precise.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: use RecoveryContext transaction, finishers on recovery completion

We should use the enclosing transaction and finisher list here.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: test_backfill.sh: limit pg log length so we trigger backfill

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix divergent backfill targets

During peering, a previous backfill target may have a slightly newer
last_update than the other options, but it will not be chosen because it
is incomplete.  That caused a failed assert during activate() (#1983).

To fix, we remove the bad assert, and then fix merge_log() so that the
replica/backfill target will trim its divergent entries when it gets the
activation MLogRec.  We also fix the handling of MInfoRec, as that can
trigger the same analogous condition.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: implement filestore_blackhole hook

If true, we'll drop any new transactions on the floor. Useful for
triggering failure conditions (e.g., prior to killing ceph-osd itself, to
ensure some operations don't reach the local disk).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: should remove bucket dir instead of sending intent

that was really useless, and also bucket cleanup was broken anyway.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

librados: fix a leak

watch notification message was missing a ->put()

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

osd: disable clone overlap for push/pull

There is a bug in the push/pull code.  Disable the recovery smarts by
default until we fix #2002.

There is currently a race (in the callers) where:
 - an adjacent clone is missing
 - we (calculate some clone overlap? and) start pulling
 - we get adjacent clone
 - we get push, calc a different overlap, and then get confused.

Signed-off-by: Sage Weil <sage@newdream.net>