osd: more heartbeat debug

Signed-off-by: Sage Weil <sage@newdream.net>

osd: don't fail new heartbeat peers

last_tx may be 0 because we just added this peer; don't mark them down
yet!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: ignore peer epoch of 0 on ping reply

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: discard heartbeat_peer in note_down_osd

Discard the heartbeat_peer as soon as we find out, along with queued
failures, or else the heartbeat_check may come along (without map_lock)
and requeue a failure.  And then later, when we try to report it, we'll
osdmap->get_inst() on a now-down OSD and fail miserably.

Reported-by: Wido den Hollander <wido@widodh.nl>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: rename hbin -> hbclient, hbout -> hbserver

This is way less confusing.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: send pings from hbin

Fixes: #2212
Signed-off-by: Sage Weil <sage@newdream.net>

osd: simplify heartbeat logic

Simplify heartbeats to use a simple request/reply model.

 - avoid any weirdness with map update timing
 - no from/to distinction
 - lossy client/server model

Signed-off-by: Sage Weil <sage@newdream.net>

ceph: --concise by default, add --verbose option

It's time.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'upstream/wip_latency'

ReplicatedPG: ctx might not contain an OpRequest

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileJournal: optionally zero journal on create

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileJournal: use DSYNC for directio path

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileStore: Pass OpRequestRef into filestore in queue_transaction

This allow us to track op progress through the filestore.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd/: OpRequest implements TrackedOp for passing into filestore

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: replace dout with ldout

librgw can't use g_ceph_context

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Merge branch 'stable'

throttle: fix off by one issue

We were blocking only if we exceeded max count, not if
we reached it.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: throttle incoming requests

Don't accept more than the number of threads, otherwise if cluster is
backed up for any reason we'd end up exhausting resources.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Merge branch 'wip-intent-fixes'

Merge branch 'master' of ssh://github.com/ceph/ceph

rgw: minor style fixes

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: all intent log operations are now async

That includes removing a directory index object, and the removal of
the actual intent log object.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

osd: increase default heartbeat_interval to 6 seconds

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: remove pool_list(), can't list_objects() on system buckets

pool_list() was broken, replaced now with pool_iterate(). list_objects()
shouldn't be used any more with system buckets (raw pools), we can't
have it return sorted list of objects without reading the entire list.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: intent log processing uses new pool_iterate()

intead of pool_list(), which is broken (assuming pgls results are
sorted, which are not).

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Merge branch 'misc-fixes-for-review'

mon: Paxos needs to store the latest version permanently on-disk.

Previously it was only storing this m->latest_value in the stash,
which of course got overwritten. And then when somebody tried to read
it back, it failed!
Instead, require that the message include the regular version (not
just the stashed version), which the previous commit provides. And then
write the regular version to disk alongside the stash.

This set of procedures still suffers from some of the same disk consistency
issues as we recently fixed in slurping, but it's better than it was, and
fixing those would require a good deal more work.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

paxos: share_state sends every unknown value, including the stashed one

Sage points out that the stashed object might not be the same as the
one we actually archive. For instance, OSDMonitor stashes the full
OSDMap but the items it stores in the regular machine_name dir are
incrementals.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

uclient: We want to release cache when we lose the CACHE cap, not gain it!

Looks like this was detected as a problem back in
84644dc56183b67050793a1b8da07850508b29d6 but the fix wasn't completed.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

doc: add a short thing on kernel client troubleshooting.

I just noticed this sitting uncommitted in my tree.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

v0.44.1

objecter: don't call op_throttle_ops.take(1) unconditionally

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

objecter: add in-flight ops throttling

In addition to ops length, we also want to throttle it by
actual number of ops.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

msg: assert pipe->msgr == msgr

Fixes: #2216
Signed-off-by: Sage Weil <sage@newdream.net>

doc: include crush in toctree

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

doc: few notes on manipulating the crush map

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

doc/dev/peering.rst: fix typo

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd/: OpRequest is no longer a RefCountedObject, remove puts/gets

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd/: Convert OpRequest* to OpRequestRef

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

OSD: Add typedef for shared_ptr<OpRequest>

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd/: add mark_event to OpRequest and move tracking into OpTracker

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

vstart: enable omap for xattrs

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'gh/wip-doc-peering'

don't override CFLAGS

leveldb adds -I flags to CFLAGS and CXXFLAGS, but if these macros are
overridden in the make command line, the flags are dropped, and the
build fails.  leveldb should probably use AM_CFLAGS instead, but the
spec file can specify the preferred CFLAGS in the configure command
line, and then everything will work as expected.

Signed-off-by: Alexandre Oliva <oliva@lsd.ic.unicamp.br>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Makefile: fix modules that cannot find pk11pub.h when compiling with NSS on RHEL6

Signed-off-by: Jim Schutt <jaschut@sandia.gov>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: don't #include fcgi from rgw_common.h

ceph-dencoder #includes rgw_common.h, and needs to build even when
--without-radosgw is specified and libfcgi isn't installed.

Reported-by: vandemar in #ceph
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rados_bench: generate_object_name now takes a buffer length

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

doc: dev/peering.rst edits from Greg

Signed-off-by: Sage Weil <sage@newdream.net>

doc: update dev/peering document

- fix discussion of last epoch started
- define terms for current and past intervals
- describe role of pg info
- remove mention of the backlog
- fix discussion of up_thru
- etc.

Signed-off-by: Sage Weil <sage@newdream.net>

msgr: fix tcp.cc linkage

Signed-off-by: Sage Weil <sage@newdream.net>

cephtool: don't prefix log items

This just makes it hard to read them.

Signed-off-by: Sage Weil <sage@newdream.net>

Objecter: resend linger_ops on any change

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

ObjectStore: Add collection_move to generate_instances

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileStore: remove src on EEXIST during collection_move replay

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileStore: whitelist COLLECTION_MOVE on replay

Signed-off-by: Samuel Just <rexludorum@gmail.com>

ObjectStore: add COLLECTION_MOVE to dump

Signed-off-by: Samuel Just <rexludorum@gmail.com>

doc: update list of debian dists

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'stable'

Objecter: resend linger_ops on any change

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

TestRados: Add watch

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: keep pool placement info also in cacheable location

Mirror the pools placement info, so that we can cache it.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: remove unused definition

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Revert "Objecter: add op->resend_on_any_change"

This reverts commit c53194d75390dd6d5aa4a9a33f741cbd106e3338.

recalc_linger_op_target is used for linger_ops

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: return -EBUSY on delete for objects with watchers

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Objecter: add op->resend_on_any_change

lingers must be resent even if the primary does not change.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: osd_max_notify_timeout -> osd_default_notify_timeout

This setting should not override user specified timeout.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: replace bucket_id generation

bucket_id is now string: <global instance id>.<num> where
num is increasing monotonically within the current rgw
instance.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

v0.44

rgw: process default alt args before processing conf file

this fixes #2189

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: process default alt args before processing conf file

this fixes #2189

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: incrase socket backlog

20 is too small

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: fix internal cache api

This fixes issue #2190

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

OSD: do not hold obc lock in disconnect_session_watches

ObjectContext::lock is used only for implementing read_lock and
write_lock.  PG::lock is used to protect the ObjectContexts
belonging to a pg.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

ReplicatedPG: there should be no object_contexts during on_activate

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'next'

osd: fix object_info.size mismatch file due to truncate_seq on new object

If the first write that creates an object includes a truncate_seq and
truncate_size, we were taking the truncte patch and doing a truncate op
in our transaction prior to the write, and then setting the object_info
size appropriately.  However, if the object doesn't exist, the truncate
op fails even though the oi.size gets set.

Later, this turns up as a scrub error (see #2080).

Fix this by skipping the truncate if it is a new object.  Instead, we
should just initialize our truncate_{seq,size} metadata so that we're all
up to date for any later writes.

Alternatively, we could touch the object and then truncate it (up) to the
large size, but this is sort of a waste; data beyond a short object eof is
defined to be zeros, so all we would accomplish is making recovery work
harder by copying zeros around.

Fixes: #2080
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

.gitignore: xattr_bench

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

heartbeatmap: use utimes(2) instead of futimens(2)

For poor users with ancient glibc.  We don't much care about rename races
here anyway.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: remove special handline for head recovery from clone

This breaks because:

 - we don't have the head or current snapset
 - get_object_context() creates a new snapset, which is wrong

We probably can only do this if we are certain we can construct/modify
the old snapset and end up with the correct one.

Signed-off-by: Sage Weil <sage@newdream.net>

config: fix recursive locking of md_config_t::lock

Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: explicitly create new object,snap contexts on push

We specifically want to use this during recovery to avoid loading the obc
or ssc for a previous version of the object and populating the watchers.
We know we won't have any existing obc here because it is missing (old or
dne).

For the snapset context, we provide it explicitly when we recover the head
or snapset object (which we always do first).  For clones, we re-use the
existing get_snapset_context(), which will either have the ssc open or
can load it from the head/snapset object.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: re-use create_object_context() in get_object_context()

Signed-off-by: Sage Weil <sage@newdream.net>

osd: ReplicatedPG::create_object_context()

New helper that creates a new object context.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: create_snapset_context()

Signed-off-by: Sage Weil <sage@newdream.net>

osd: ensure we don't clobber other *contexts when registering new ones

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip_omap_xattrs'

ReplicatedPG,FileStore: clone should copy xattrs as well

_make_clone (called from make_writeable) and _rollback_to included
attr reads from head or a clone.  In that case, an ondisk read
lock would be necessary.  Now, clone also handles xattrs, so the
attr read should not be necessary.

Signed-off-by: Samuel Just <sam.just@dreamhost.com>

RadosModel: test xattrs with omap

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileStore: add support for omap xattrs

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'wip-msgr4'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

ObjectMap: use Index object for locking rather than path object

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

tests/: Added xattr bench

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

config_opts.h: opts for omap_xattrs

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

test_object_map: update unit test for xattr

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

DBObjectMap: implement xattr interface

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ObjectMap: add interface for storing xattrs

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

DBObjectMap: add support for storing xattrs

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

configure: fix warnings

Finally!

Signed-off-by: Sage Weil <sage@newdream.net>

ReplicatedPG: populate_object_context during handle_pull_response

A cached objectcontext should always have its watchers populated.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

leveldb: .gitignore TAGS

Signed-off-by: Sage Weil <sage@newdream.net>