hobject_t: decode json

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

add libjson_spirit.la

This is lightweight and relies on boost spirit, which we already use, so
there are no new dependencies.

There were some other libraries that also looked good, but they weren't
already packages for existing Debian distros like squeeze or even wheezy.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: pass in data to do_command

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: 'tell osd.N mark_unfound_lost revert' -> 'pg <pgid> mark_unfound_lost revert'

More consistent interface.

Fixes: #2030
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

lockdep: warn on stderr (via derr), not stdout

Otherwise we screw up ceph-conf output and the like.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

do_autogen.sh: -T for --without-tcmalloc

Signed-off-by: Sage Weil <sage@newdream.net>

ceph: fix help.t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-tool: remove reference to "stop" command

This doesn't exist any more, and I don't think it
ever "cleanly shut down the filesystem" -- certainly not
within my recent lifetime!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
Reviewed-by: Dan Mick <dan.mick@dreamhost.com>

mds: remove unused MDBalancer dump_pop_map() function.

Commenting it out is not the right answer. ;)

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
Reviewed-by: Dan Mick <dan.mick@dreamhost.com>

Merge remote branch 'origin/wip-mds-old-inodes'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote branch 'origin/wip-dencoder'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote branch 'origin/wip-1820'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: only set CLEAN when we are not remapped (up == acting)

If we have a temporary mapping for this PG, consider that unclean.  This
makes CLEAN and REMAPPED mutually exclusive.  For example, a 2 node cluster
with 2x replication and one osd marked out will make the pgs all
active+remapped, not active+clean+remapped.

Fixes: #2094
Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/wip-pg-query'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

Merge remote-tracking branch 'gh/wip-dump-ops-in-flight'

Reviewed-by: Sage Weil <sage@newdream.net>

mon: use pending_mdsmap for deactivate

We should always look at the proposed map to avoid weird races.

Signed-off-by: Sage Weil <sage@newdream.net>

doc: 'deactivate mds' instead of 'stop mds'

Signed-off-by: Sage Weil <sage@newdream.net>

mon: mds "stop" -> "deactivate"

See #1820.

Signed-off-by: Sage Weil <sage@newdream.net>

test: add basic test for the OSD's dump_ops_in_flight adminsocket command

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: add "dump_ops_in_flight" to the AdminSocket.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mon: refuse to stop mds if max_mds will make it rejoin

Otherwise the MDS will leave the cluster and immediately rejoin, which is
useless and confusing to users.  See #1820.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-2090'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

mon: unlock mon before msgr shutdown

The ceph_mon.cc main() will delete mon when the msgr dispatch thread
completes.  Make sure we unlock before we shut down the messenger, and
avoid touching this after messenger->shutdown().

Fixes: #2090
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: deprecate mon 'stop' command

Send SIGTERM.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: join dispatch_thread after it completes

This is just for completeness.  No change in behavior, since we don't
get here until the thread has signaled it is done.

Drop the destroy() overload, since we join earlier.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote-tracking branch 'gh/wip-stop'

filestore: use IOC_CLONERANGE intead of IOC_CLONE ioctl

This is functionally equivalent, except that valgrind doesn't complain
about a bad pointer passed to an ioctl.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

osd: drop "stop" command

Send SIGINT.

Fixes: #1820
Signed-off-by: Sage Weil <sage@newdream.net>

osd: drop unused "stop" check

This is never reached: both callers handle "stop" explicitly.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: don't complete recovery if unfound

Otherwise we fail the !needs_recovery() assert.  Because we aren't
recovered.  For example,

2012-02-21 16:16:13.104665 1685c700 osd.5 1217 pg[0.16( v 1215'337 lc 19'2 (0'0,1215'337] n=25 ec=1 les/c 0/1061 1210/1210/1210) [5,3] r=0 lpr=1210 mlcod 0'0 active m=23 u=23 snaptrimq=[1~99,9b~e,aa~72,11d~3d,15b~e,16a~f,17a~5,180~4,185~1a,1a0~a,1ac~10,1bd~4,1c2~8,1cb~1,1cd~1,1cf~1a,1ea~10,1fb~6,202~2,205~2,209~2,20c~8,215~2,218~5,21e~1,220~1,222~9,22c~4,231~3,235~2,238~3,23e~2,241~4,246~1,248~1,24b~1,24d~9,257~6,25e~1,263~1,265~2,268~3,26e~1,273~1,275~5,27e~1,280~2]] needs_recovery osd.3 has 23 missing
osd/PG.cc: In function 'boost::statechart::result PG::RecoveryState::Active::react(const PG::RecoveryState::RecoveryComplete&)' thread 1685c700 time 2012-02-21 16:16:13.108923
osd/PG.cc: 4070: FAILED assert(!pg->needs_recovery())
 ceph version 0.42-70-g0e4367a (commit:0e4367aaac88b99c36386b6ce5e8d816fdd4ada0)
 1: (PG::RecoveryState::Active::react(PG::RecoveryState::RecoveryComplete const&)+0x1b3) [0x6a1173]
 2: (boost::statechart::simple_state<PG::RecoveryState::Active, PG::RecoveryState::Primary, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x121) [0x6c7301]
 3: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x6b) [0x6bfc6b]
 4: (PG::RecoveryState::handle_recovery_complete(PG::RecoveryCtx*)+0x10c) [0x67c03c]
 5: (ReplicatedPG::start_recovery_ops(int, PG::RecoveryCtx*)+0x241) [0x4f83c1]
 6: (OSD::do_recovery(PG*)+0x345) [0x54b3e5]
 7: (ThreadPool::worker()+0xa26) [0x619e66]
 8: (ThreadPool::WorkThread::entry()+0xd) [0x57ad5d]
 9: (()+0x7971) [0x5037971]
 10: (clone()+0x6d) [0x679f92d]

Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

mds: make EMetaBlob::fullbit::old_inodes non-ptr

No need to put this separately on the heap, as a static map<> isn't much
more expensive than a pointer.  Also, this ensures we unconditonally
reset in->old_inodes to a potentially empty value if we replay the same
inode multiple times and lose old inodes in subsequent versions.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: Add old_inodes to emetablob

Add information about old inodes to the mds journal.

Signed-off-by: Alexandre Oliva <oliva@lsd.ic.unicamp.br>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: include encoding check scripts in dist tarball

This makes 'make distcheck' happy.  Well, more happy at least; it's still
cranky but I can't tell why.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

debian: add ceph-dencoder

Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec.in: add ceph-dencoder

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: man page

Signed-off-by: Sage Weil <sage@newdream.net>

osd: make object_info_t::dump  using hobject_t and object_locator_t dumpers

Makes the output more readable.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/wip-dump-stuck-pgs'

Reviewed-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/wip-osd-write'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

osdmap: dump embedded crush map in Incremental::dump()

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-crush'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

crush: write CrushWrapper:dump()

Signed-off-by: Sage Weil <sage@newdream.net>

test/rados-api/misc: fix LibRadosMisc.Operate1PP test

It's a mutation, so we get a result of 0 (or error).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: refuse to return data payload if request wrote anything

Write operations aren't allowed to return a data payload because
we can't do so reliably. If the client has to resend the request
and it has already been applied, we will return 0 with no
payload.  Non-deterministic behavior is no good.

See #1765.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip-osdmap'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

osdmap: dump fullmap from dump()

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-1821'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: accepted access key chars should be url safe

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

ceph: if 'pg <pgid> ..' doesn't parse a pgid, send to mon

E.g., 'pg dump'.  Sigh.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: fix misplaced unit tests

These weren't run on make check because they were defined in the wrong
spot.

Signed-off-by: Sage Weil <sage@newdream.net>

hobject_t: remove unused back_up_to_bounding_key()

This was a path not taken in the backfill code.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: sched_scrub() outside of map_lock

Inside sched_scrub() we call _lookup_lock_pg(), which takes
map_lock.get_read().  That's technically okay because RWLock read side is
recursive, but lockdep doesn't know that, and we don't need map_lock
because we hold osd_lock.

Signed-off-by: Sage Weil <sage@newdream.net>

global: resurrect lockdep

Add 'lockdep' config option, and initialize g_lockdep from that in
global_init().

Signed-off-by: Sage Weil <sage@newdream.net>

mon: disable pg_num adjustment

Until #1515 is fixed/reimplemented.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: use encode function for new Incremental

When we encode an Incremental, use the encode wrapper function, so that
we can capture the encoded struct when building with ENCODE_DUMP.  Set
all features (the default when encode() is called directly).

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: successfully decode short map

When we send (old) maps to the kclient, we omit the extended section.  Lets
decode those (old, abbreviated maps) successfully, too.

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: use FEATURE encoder macro

This generates encode/decode functions that pass feature bits into the
encoder, allowing us to encode old formats.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/btrfs/test_rmdir_async_snap

Attempt to reproduce btrfs bug when rmdirs race with an async snap.
Unsuccessful.  Best guess is that we need multiple threads to trigger.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: add OSDMap::Incremental

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: add Incremental::dump()

Signed-off-by: Sage Weil <sage@newdream.net>

osd: don't count SNAPDIR as a clone during backfill

When we are backfilling, we add in objects as we push them.  Do not count
the snapdir object as a clone, or else we'll screw up the count.

Fixes: #2080
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crush: fix CrushCompiler warning

warning: crush/CrushCompiler.cc:595: ‘r’ may be used uninitialized in this function

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test/encoding/readable.sh: sh, not dash

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crushtool: fix clitests

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'stable'

msgr: fix shutdown race again

Only unlock once.  Sigh.

Signed-off-by: Sage Weil <sage@newdream.net>

v0.42

msgr: fix accept shutdown race fault

Need to hold pipe_lock.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: test injected crush map

Run a bunch of inputs through an injected crush map to make sure it isn't
broken.

Fixes: #1932
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crush: move crushtool --test into CrushTester

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crush: move (de)compile into CrushCompiler class

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: fix message discard on shutdown

Return true, so the messenger is happy, and drop the message reference.

Avoids an assert like

2012-02-19T12:36:05.102 INFO:teuthology.task.ceph.mon.2.err:ms_deliver_dispatch: fatal error: unhandled message 0x1b7b280 paxos(auth lease_ack lc 8 fc 1 pn 0 opn 0) v1 from mon.2 10.3.14.197:6789/0msg/Messenger.h: In function 'void Messenger::ms_deliver_dispatch(Message*)' thread 7fd7fe360700 time 2012-02-19 12:36:05.094713
2012-02-19T12:36:05.102 INFO:teuthology.task.ceph.mon.2.err:msg/Messenger.h: 143: FAILED assert(0)

Signed-off-by: Sage Weil <sage@newdream.net>

crush: uninline encode/decode

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crush: cleanup: use temp var for curstep

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: use want_state to indicate shutdown

State gets DNE when we receive the first map.  And want_ makes more sense
anyway.  Fixes MDS startup.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: implement 'pg <pgid> query'

Dump a blob of json about the pg state.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: dispatch 'pg <pgid> ...' commands to PG::do_command()

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph: direct 'pg <pgid> ...' to primary osd for given pgid

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix up argument to PG::init()

Commit cefa55b288b40e17ade9875493dd94de52ac22bf moved PG initialization
into init(), but passed acting for both up and acting args.  This lead to
confusion between primary and replica.

Also fix debug print so that the output is useful.

Fixes: #2075, #2070
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

SimpleMessenger: drop unused sigint()

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: promote SimpleMessenger::Policy to Messenger::Policy

This is part of the generic interface, not specific to the implementation.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: ignore all msgr callbacks on shutdown, not just dispatch

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: discard messages while shutting down

Add SHUTDOWN state.  Ignore any msgr callbacks if set.

Fixes crash like

2012-02-18T21:57:58.912 INFO:teuthology.task.ceph:Shutting down mon daemons...
2012-02-18T21:57:58.912 DEBUG:teuthology.task.ceph.mon.a:waiting for process to exit
2012-02-18T21:57:58.913 INFO:teuthology.task.ceph.mon.a.err:2012-02-18 21:57:58.927759 7fe98dfa1700 mon.a@1(peon) e1 *** Got Signal Terminated ***
2012-02-18T21:57:59.014 INFO:teuthology.task.ceph.mon.a.err:*** Caught signal (Segmentation fault) **
2012-02-18T21:57:59.014 INFO:teuthology.task.ceph.mon.a.err: in thread 7fe98d7a0700
2012-02-18T21:57:59.014 INFO:teuthology.task.ceph.mon.a.err: ceph version 0.41-382-gc1db900 (commit:c1db9009c2cde9dc7ab8857b0d28a1b6d931e98a)
2012-02-18T21:57:59.015 INFO:teuthology.task.ceph.mon.a.err: 1: /tmp/cephtest/binary/usr/local/bin/ceph-mon() [0x5b0871]
2012-02-18T21:57:59.015 INFO:teuthology.task.ceph.mon.a.err: 2: (()+0xfb40) [0x7fe991a1eb40]
2012-02-18T21:57:59.015 INFO:teuthology.task.ceph.mon.a.err: 3: (PerfCounters::set(int, unsigned long)+0x1a) [0x52008a]
2012-02-18T21:57:59.015 INFO:teuthology.task.ceph.mon.a.err: 4: (PGMonitor::update_logger()+0x96) [0x4d4bf6]
2012-02-18T21:57:59.015 INFO:teuthology.task.ceph.mon.a.err: 5: (PGMonitor::update_from_paxos()+0xa70) [0x4e0980]
2012-02-18T21:57:59.016 INFO:teuthology.task.ceph.mon.a.err: 6: (Monitor::_ms_dispatch(Message*)+0x143b) [0x47bd6b]
2012-02-18T21:57:59.016 INFO:teuthology.task.ceph.mon.a.err: 7: (Monitor::ms_dispatch(Message*)+0x90) [0x489210]
2012-02-18T21:57:59.016 INFO:teuthology.task.ceph.mon.a.err: 8: (SimpleMessenger::dispatch_entry()+0x89a) [0x53959a]
2012-02-18T21:57:59.016 INFO:teuthology.task.ceph.mon.a.err: 9: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x46358c]
2012-02-18T21:57:59.016 INFO:teuthology.task.ceph.mon.a.err: 10: (()+0x7971) [0x7fe991a16971]
2012-02-18T21:57:59.017 INFO:teuthology.task.ceph.mon.a.err: 11: (clone()+0x6d) [0x7fe9902a592d]

which is analogous to #2014.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: fix shutdown vs accept race

This is a kludge.  The real fix is to rewrite SimpleMessenger as a state
machine.

Fixes: #2073
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: drop all messages during suicide

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'gh/wip-pg-states'

mon: fix STUCK_STALE check

Look at last_unstale if STALE bit is not set.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: add dump_stuck command

This will help monitoring transient pg states at a coarse level.

Fixes: #2005
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

mon: constify functions needed to use dout from a const function

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

PGMap: extract method for outputting plain pg stats

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

PGMap: fix else indentation

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: update_stats() in GetInfo state start

This is the first stage of peering.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: don't update_stats() on prec_replica_info

Nothing changes here...

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: hold journal_lock during

Hold journal_lock during replay so that we don't stomp on variables like
op_seq and open_ops that the the commit thread cares about.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: only complete/deregister repop once

It's now possible to send the ack and deregister the repop before the
op_applied() happens.  And when that happens, we'll call eval_repop() once
more.  Don't do anything in that case.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'next'

man: regenerate man pages

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

man: move man page fixes to rst

83cf1b62fde525d068bc292c4a1ccc42199657ae and
e5f49104ab62ba7bc42cf6ecf41c9257b46585f7 updated the nroff output
but not the rst source.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

doc: fix snapshot creation/deletion syntax in rbd man page (trivial)

Creating a snapshot requires using "rbd snap create",
as opposed to just "rbd create". Also for purposes of
clarification, add note that removing a snapshot similarly
requires "rbd snap rm".

Thanks to Josh Durgin for the explanation on IRC.

Signed-off-by: Florian Haas <florian@hastexo.com>
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

PGMap: add indent settings header

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>