v0.43

librados: only shutdown objecter after it's initialized

The objecter is only initialized once the RadosClient state is
CONNECTED from the perspective of a RadosClient::shutdown()
caller. Error paths in RadosClient::connect() may call shutdown while
still in the CONNECTING state.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

Makefile: add json_spirit headers to tarball

Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: drop separate libjson_spirit.la

automake seems to have difficulty with the .la dependency on another .la.
Since libjson_spirit.la is only used by libcommon.la anyway, just build it
directly into that.  Sigh.

...
CXXLD libjson_spirit.la
AR libmds.a
CXXLD libcls_rbd.la
CXXLD libcls_rgw.la
CXXLD cephfs
CCLD test_ioctls
CC libcommon_la-ceph_ver.lo
CXX libcommon_la-version.lo
CXX ceph_dencoder.o
CCLD mount.ceph
CC ceph_ver.o
CXX test_libhadoopcephfs_build-version.o
CXXLD test_libhadoopcephfs_build
CXXLD libcommon.la
libtool: link: cannot find the library `libjson_spirit.la' or unhandled argument `libjson_spirit.la'

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-osd: clarify error messages

So we know where the error came from.  And use real error codes in init().

Signed-off-by: Sage Weil <sage@newdream.net>

init: Actually do start the daemons when 'service ceph start <type>' is specified

A bug in my previous patch prevented any daemon with auto_start set to false from starting.

This patch allows:
* /etc/init.d/ceph start osd|mds|mon
* service ceph start osd|mds|mon

It however does not start daemons if auto_start is disabled when you invoke:
* /etc/init.d/ceph start
* service ceph start

Signed-off-by: Wido den Hollander <wido@widodh.nl>

filestore: make less noise on ENOENT

Don't generate high-level log spam on every open error.

Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

pg: use get_cluster_inst instead of get_inst in activate

This was mistakenly broken in 4b3bb5ab37a05fa001d59f24da7d9c30d650321b

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
Reviewed-by: Sam Just <sam.just@dreamhost.com>

journaler: log on unexpected objecter error

This will help with #2110, #1796, #1640.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix recursive map_lock via check_replay_queue()

Also drop activate_pg() helper while we're at it, so it's clear that we
are the only user.

recursive lock of OSD::map_lock (33)
 ceph version 0.42-146-g7ad35ce (commit:7ad35ce489cc5f9169eb838e1196fa2ca4d6e985)
2012-02-24 12:30:16.541416 1: (PG::lock(bool)+0x2a) [0xa09348]
2012-02-24 12:30:16.541424 2: (OSD::_lookup_lock_pg(pg_t)+0xbd) [0x84b8df]
2012-02-24 12:30:16.541431 3: (OSD::activate_pg(pg_t, utime_t)+0x9f) [0x87463b]
2012-02-24 12:30:16.541442 4: (OSD::check_replay_queue()+0x12f) [0x87452d]
2012-02-24 12:30:16.541450 5: (OSD::tick()+0x23c) [0x8535ea]
2012-02-24 12:30:16.541456 6: (OSD::C_Tick::finish(int)+0x1f) [0x881671]
2012-02-24 12:30:16.541462 7: (SafeTimer::timer_thread()+0x2d5) [0x8f8211]
2012-02-24 12:30:16.541468 8: (SafeTimerThread::entry()+0x1c) [0x8f923c]
2012-02-24 12:30:16.541475 9: (Thread::_entry_func(void*)+0x23) [0x9c8109]
2012-02-24 12:30:16.541485 10: (()+0x68ba) [0x7f9dbed838ba]
2012-02-24 12:30:16.541491 11: (clone()+0x6d) [0x7f9dbd66f02d]
2012-02-24 12:30:16.541495 common/lockdep.cc: In function 'int lockdep_will_lock(const char*, int)' thread 7f9db9d98700 time 2012-02-24 12:30:16.541504

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Sam Just <samuel.just@dreamhost.com>

init-ceph: stick with /var/run for the time being

/run isn't present on older systems.  Stick with the old location until it
is more pervasive, or we add an autoconf option to control it.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

debian: /var/run/ceph -> /run/ceph

/run/ceph should exists for creating UNIX domain sockets
ceph uses UNIX domain sockets for internal communication. Create their
directory on startup as /run is on a virtual filesystem.

Last-Update: <2012-02-26>
Bug-Debian: http://bugs.debian.org/660238
Forwarded: <ceph-devel@vger.kernel.org>
Signed-off-by: Laszlo Boszormenyi (GCS) <gcs@debian.hu>

debian: build-{indep,arch}

Signed-off-by: Laszlo Boszormenyi <gcs@debian.hu>

debian: sdparm|hdparm, new standards version

Signed-off-by: Laszlo Boszormenyi <gcs@debian.hu>

rgw: initialize bucket_id in bucket structure

might make valgrind a little bit less noisy.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: _exit(0) on SIGTERM

We need to do something a bit smarter to get coverage information, but this
is a start.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'gh/wip-crush-adjust'

Reviewed-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Merge remote branch 'gh/wip-mds-resetter'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge branch 'wip-pg-query'

Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'stable'

v0.42.2

Merge remote-tracking branch 'gh/stable' into stable

Merge branch 'stable'

osd: fix array index

Signed-off-by: Sage Weil <sage@newdream.net>

lockdep: don't make noise on startup

Who cares!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

formatter: fix trailing dump_stream()

Flush a previous dump_stream() if it was the last thing prior to a
close_section().

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: include timestamps in state json dumps

Include the time we entered this state in the dump.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip-2007'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: use blocks for readability in list_missing

Signed-off-by: Sage Weil <sage@newdream.net>

osd: dump recovery_state states in json

Use a formatter.  Present a vector of states, inner to outer.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: query Peering substates

Signed-off-by: Sage Weil <sage@newdream.net>

osd: query recovery state machine

For now, just append this to the end of the pg <pgid> query json dump.
We definitely want to do something smarter here, but I'm not sure whether
json or plaintext is the way to go.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: add tunable for number of records in osd command replies

e.g., 'pg <pgid> list_missing [offset]'.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: 'pg <pgid> list_missing <json hobject_t offset>'

Dump missing objects in json.  If more key is non-zero, user should ask for
more by passing the last object as the offset for the next request.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

hobject_t: decode json

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

add libjson_spirit.la

This is lightweight and relies on boost spirit, which we already use, so
there are no new dependencies.

There were some other libraries that also looked good, but they weren't
already packages for existing Debian distros like squeeze or even wheezy.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: pass in data to do_command

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: 'tell osd.N mark_unfound_lost revert' -> 'pg <pgid> mark_unfound_lost revert'

More consistent interface.

Fixes: #2030
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

lockdep: warn on stderr (via derr), not stdout

Otherwise we screw up ceph-conf output and the like.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

do_autogen.sh: -T for --without-tcmalloc

Signed-off-by: Sage Weil <sage@newdream.net>

ceph: fix help.t

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

v0.42.1

debian: add ceph-dencoder

Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec.in: add ceph-dencoder

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: man page

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-tool: remove reference to "stop" command

This doesn't exist any more, and I don't think it
ever "cleanly shut down the filesystem" -- certainly not
within my recent lifetime!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
Reviewed-by: Dan Mick <dan.mick@dreamhost.com>

mds: remove unused MDBalancer dump_pop_map() function.

Commenting it out is not the right answer. ;)

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
Reviewed-by: Dan Mick <dan.mick@dreamhost.com>

mds: clean up useless block

Signed-off-by: Sage Weil <sage@newdream.net>

mds: fix Resetter locking

We need to hold the lock for ms_dispatch, esp calls into objecter.  We
should only drop it when blocking; use distinct naming for the on-stack
mutex used for that.

Reported-by: Alexandre Oliva <oliva@lsd.ic.unicamp.br>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'origin/wip-mds-old-inodes'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote branch 'origin/wip-dencoder'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote branch 'origin/wip-1820'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: only set CLEAN when we are not remapped (up == acting)

If we have a temporary mapping for this PG, consider that unclean.  This
makes CLEAN and REMAPPED mutually exclusive.  For example, a 2 node cluster
with 2x replication and one osd marked out will make the pgs all
active+remapped, not active+clean+remapped.

Fixes: #2094
Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/wip-pg-query'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: conditionally encode old pg_pool_t when no CEPH_FEATURE_OSDENC

This fixes OSDMap compatibility between v0.42 and <v0.42.

For MOSDMap, reencode maps if OSDENC feature is missing.  Also rev the
message version.  We don't use COMPAT version here because v3 can't be
understood by v2 (that's why we're checking feature bits).  (It will be
possible to do that later when our constituent types can be decoded by
multiple versions.)

Fixes: #2095
Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote-tracking branch 'gh/wip-dump-ops-in-flight'

Reviewed-by: Sage Weil <sage@newdream.net>

mon: use pending_mdsmap for deactivate

We should always look at the proposed map to avoid weird races.

Signed-off-by: Sage Weil <sage@newdream.net>

doc: 'deactivate mds' instead of 'stop mds'

Signed-off-by: Sage Weil <sage@newdream.net>

mon: mds "stop" -> "deactivate"

See #1820.

Signed-off-by: Sage Weil <sage@newdream.net>

test: add basic test for the OSD's dump_ops_in_flight adminsocket command

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: add "dump_ops_in_flight" to the AdminSocket.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mon: refuse to stop mds if max_mds will make it rejoin

Otherwise the MDS will leave the cluster and immediately rejoin, which is
useless and confusing to users.  See #1820.

Signed-off-by: Sage Weil <sage@newdream.net>

crushtool: add --reweight-item cli tests

Test list, tree, and straw buckets.

Signed-off-by: Sage Weil <sage@newdream.net>

crush: fix weight adjust for list, tree buckets

Fix the typo.  Code now matches that for straw buckets.

Reported-by: ZhuRongze <zrz4ceph@gmail.com>
Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-2090'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

mon: unlock mon before msgr shutdown

The ceph_mon.cc main() will delete mon when the msgr dispatch thread
completes.  Make sure we unlock before we shut down the messenger, and
avoid touching this after messenger->shutdown().

Fixes: #2090
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: deprecate mon 'stop' command

Send SIGTERM.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: join dispatch_thread after it completes

This is just for completeness.  No change in behavior, since we don't
get here until the thread has signaled it is done.

Drop the destroy() overload, since we join earlier.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote-tracking branch 'gh/wip-stop'

filestore: use IOC_CLONERANGE intead of IOC_CLONE ioctl

This is functionally equivalent, except that valgrind doesn't complain
about a bad pointer passed to an ioctl.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

osd: drop "stop" command

Send SIGINT.

Fixes: #1820
Signed-off-by: Sage Weil <sage@newdream.net>

osd: drop unused "stop" check

This is never reached: both callers handle "stop" explicitly.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: don't complete recovery if unfound

Otherwise we fail the !needs_recovery() assert.  Because we aren't
recovered.  For example,

2012-02-21 16:16:13.104665 1685c700 osd.5 1217 pg[0.16( v 1215'337 lc 19'2 (0'0,1215'337] n=25 ec=1 les/c 0/1061 1210/1210/1210) [5,3] r=0 lpr=1210 mlcod 0'0 active m=23 u=23 snaptrimq=[1~99,9b~e,aa~72,11d~3d,15b~e,16a~f,17a~5,180~4,185~1a,1a0~a,1ac~10,1bd~4,1c2~8,1cb~1,1cd~1,1cf~1a,1ea~10,1fb~6,202~2,205~2,209~2,20c~8,215~2,218~5,21e~1,220~1,222~9,22c~4,231~3,235~2,238~3,23e~2,241~4,246~1,248~1,24b~1,24d~9,257~6,25e~1,263~1,265~2,268~3,26e~1,273~1,275~5,27e~1,280~2]] needs_recovery osd.3 has 23 missing
osd/PG.cc: In function 'boost::statechart::result PG::RecoveryState::Active::react(const PG::RecoveryState::RecoveryComplete&)' thread 1685c700 time 2012-02-21 16:16:13.108923
osd/PG.cc: 4070: FAILED assert(!pg->needs_recovery())
 ceph version 0.42-70-g0e4367a (commit:0e4367aaac88b99c36386b6ce5e8d816fdd4ada0)
 1: (PG::RecoveryState::Active::react(PG::RecoveryState::RecoveryComplete const&)+0x1b3) [0x6a1173]
 2: (boost::statechart::simple_state<PG::RecoveryState::Active, PG::RecoveryState::Primary, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x121) [0x6c7301]
 3: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x6b) [0x6bfc6b]
 4: (PG::RecoveryState::handle_recovery_complete(PG::RecoveryCtx*)+0x10c) [0x67c03c]
 5: (ReplicatedPG::start_recovery_ops(int, PG::RecoveryCtx*)+0x241) [0x4f83c1]
 6: (OSD::do_recovery(PG*)+0x345) [0x54b3e5]
 7: (ThreadPool::worker()+0xa26) [0x619e66]
 8: (ThreadPool::WorkThread::entry()+0xd) [0x57ad5d]
 9: (()+0x7971) [0x5037971]
 10: (clone()+0x6d) [0x679f92d]

Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

mds: make EMetaBlob::fullbit::old_inodes non-ptr

No need to put this separately on the heap, as a static map<> isn't much
more expensive than a pointer.  Also, this ensures we unconditonally
reset in->old_inodes to a potentially empty value if we replay the same
inode multiple times and lose old inodes in subsequent versions.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: Add old_inodes to emetablob

Add information about old inodes to the mds journal.

Signed-off-by: Alexandre Oliva <oliva@lsd.ic.unicamp.br>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Fix ceph-mds --journal-reset

Complete configuration initialization for special actions, and
hold Resetter lock while running reset.

Signed-off-by: Alexandre Oliva <oliva@lsd.ic.unicamp.br>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: include encoding check scripts in dist tarball

This makes 'make distcheck' happy.  Well, more happy at least; it's still
cranky but I can't tell why.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

debian: add ceph-dencoder

Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec.in: add ceph-dencoder

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dencoder: man page

Signed-off-by: Sage Weil <sage@newdream.net>

osd: make object_info_t::dump  using hobject_t and object_locator_t dumpers

Makes the output more readable.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/wip-dump-stuck-pgs'

Reviewed-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/wip-osd-write'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

osdmap: dump embedded crush map in Incremental::dump()

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-crush'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

crush: write CrushWrapper:dump()

Signed-off-by: Sage Weil <sage@newdream.net>

test/rados-api/misc: fix LibRadosMisc.Operate1PP test

It's a mutation, so we get a result of 0 (or error).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: refuse to return data payload if request wrote anything

Write operations aren't allowed to return a data payload because
we can't do so reliably. If the client has to resend the request
and it has already been applied, we will return 0 with no
payload.  Non-deterministic behavior is no good.

See #1765.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip-osdmap'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

osdmap: dump fullmap from dump()

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-1821'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: accepted access key chars should be url safe

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

ceph: if 'pg <pgid> ..' doesn't parse a pgid, send to mon

E.g., 'pg dump'.  Sigh.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: fix misplaced unit tests

These weren't run on make check because they were defined in the wrong
spot.

Signed-off-by: Sage Weil <sage@newdream.net>

hobject_t: remove unused back_up_to_bounding_key()

This was a path not taken in the backfill code.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: sched_scrub() outside of map_lock

Inside sched_scrub() we call _lookup_lock_pg(), which takes
map_lock.get_read().  That's technically okay because RWLock read side is
recursive, but lockdep doesn't know that, and we don't need map_lock
because we hold osd_lock.

Signed-off-by: Sage Weil <sage@newdream.net>

global: resurrect lockdep

Add 'lockdep' config option, and initialize g_lockdep from that in
global_init().

Signed-off-by: Sage Weil <sage@newdream.net>

mon: disable pg_num adjustment

Until #1515 is fixed/reimplemented.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: use encode function for new Incremental

When we encode an Incremental, use the encode wrapper function, so that
we can capture the encoded struct when building with ENCODE_DUMP.  Set
all features (the default when encode() is called directly).

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: successfully decode short map

When we send (old) maps to the kclient, we omit the extended section.  Lets
decode those (old, abbreviated maps) successfully, too.

Signed-off-by: Sage Weil <sage@newdream.net>