mon: adjust or decay laggy probabilities on osd boot

On each osd boot, determine whether the osd was laggy (wrongly marked down)
or newly booted.  Either update the laggy probability and interval or
decay the values, as appropriate.

Signed-off-by: Sage Weil <sage@inktank.com>

osdmap: include osd_xinfo_t to track laggy probabilities, timestamps

Track information about laggy probabilities for each OSD.  That is, the
probability that if it is marked down it is because it is laggy, and
the expected interval over which it will take to recovery if it is laggy.

We store this in the OSDMap because it is not convenient to keep it
elsewhere in the monitor.  Yet.  When the new mon infrastructure is in
place, there is a bunch of stuff that can be moved out of the OSDMap
'extended' section into other mon data structures.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: include boot_epoch in MOSDBoot

This will let the monitor infer whether we were wrongly marked down or
the daemon restarted.

Signed-off-by: Sage Weil <sage@inktank.com>

osd: include failed_for in MOSDFailure reports

The monitor will need this to dynamically adjust the heartbeat grace.

Closes: #3044
Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/wip-crush'

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

upstart: Add ceph-create-keys.conf to package.

Signed-off-by: Tommi Virtanen <tv@inktank.com>

:doc: Fixed typo.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

obsync: if OrdinaryCallingFormat fails, try SubdomainCallingFormat

This blindly tries the Subdomain calling format if the ordinary method
fails.  In particular, this works around buckets that present a
PermanentRedirect message.

See bug #3128.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Matthew Wodrich <matthew.wodrich@dreamhost.com>

Merge remote-tracking branch 'upstream/next'

Conflicts:
src/osd/ReplicatedPG.cc

ReplicatedPG: do not start_recovery_op if we are already pushing

Should fix bug #2761.

If we are already pushing soid, recovery_ops will only be decremented once for
all current pushes, so only increment recovery_ops if we are not currently
pushing it.

This bug causes us to leak a recovery op and get stuck in backfill.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

osd: fill in user log entry last after snapdir tran

Reorder the snapdir logic and ctx->at_version adjustments prior to filling
in the object_info_t and user_versions and all that stuff.  Adjust
at_version after appending the log entry (so that it points to the next
position/version we will write at.. culminating in the actual user
event).

The user log entry contains the request id, which will be used
by replay ops to put themselves in the correct place in the
waiting_for_commit/ack maps.  Thus, the repop needs to be tagged
with the same version as the log entry with the request id.
Thus, the request id bearing log entry should be the last in
the log entry vector.

This should fix #3072, wherein a replay which should wait on
the repop tagged as version '36 will instead wait on '35.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

:doc: Added tunables to cruch-map.rst.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Removed old pg tuning. New section was added.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Trimmed the old ops tree. Will remove when all porting verified.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Trimmed the tree for failures/troubleshooting. RGW remains.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: removed. RBD now has its own section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

mon: adjust number of req args for loc

At least one loc key/value pair is required to do anything useful with
these commands.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: move loc map parsing into a helper

Signed-off-by: Sage Weil <sage@inktank.com>

crush: constify loc map arguments

Signed-off-by: Sage Weil <sage@inktank.com>

crush: add const string& versions of accessors

Signed-off-by: Sage Weil <sage@inktank.com>

doc/control.rst: add 'osd crush create-or-move ...'

Signed-off-by: Sage Weil <sage@inktank.com>

doc: make note of crush usage change

Even tho it is compatible.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: make redundant osd.NNN argument optional

Instead of 'osd crush set NNN osd.NNN weight loc...', make the second
osd.NNN option optional, and allow either NNN or osd.NNN to specify the
osd id.  This makes the usage much more sane, but maintains backward
compatibility.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph tool: add 'osd crush create-or-move ...' to help

Signed-off-by: Sage Weil <sage@inktank.com>

:doc: Deleting this. Wrote a new one, but will be revised a bit soon.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Removed old ops pool section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Removed old authentication section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Removed old resize OSD section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Removed old mon resize section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Removed from old ops doc. Still needs to be composed though.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: New cluster ops section addresses the todo.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Removed old OSD troubleshooting. New version to be updated shortly.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Removed old monitor troubleshooting. New version to be revised shortly.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Removed old mds troubleshooting. Still needs to be composed.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Removed old cephfs disucssion.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Trimmed toctree to last bits of legacy data.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Updated FAQ with a friendlier message.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

mon: parse '<id>' or 'osd.<id>' for 'osd crush create-or-move ...'

Signed-off-by: Sage Weil <sage@inktank.com>

mon: fail on trailing characters after parsing numbers

parse '8' but not '8asdf'.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: 'osd crush create-or-move <id> <initial-weight> <loc ...>'

Create an item in the tree with the given weight, or move it (without
touching the weight) if it is already present.

Closes: #3101
Signed-off-by: Sage Weil <sage@inktank.com>

crush: create_or_move_item()

Create an item if it doesn't exist, with the specified weight.  If it is
already in the tree, move it, but do not adjust the weight.

Signed-off-by: Sage Weil <sage@inktank.com>

crush: get_item_weight[f]()

Get the weight for an item, or at least the first instance of an item in
the tree.

Signed-off-by: Sage Weil <sage@inktank.com>

osdmap: 4 decimal places for osd tree

This makes it print out more nicely for reasonable weights.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'wip-swift-manifest'

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add missing ret code check

Also, for some reason we used cerr instead of dout so fix
that too.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge branch 'wip-3060'

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

objecter: remove the now-unused "registering" flag on LingerOps

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

rgw: encode total usage for backwards compatibility

Adding the encoding of the total usage allows us to
make the format change backwards compatible.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: configurable swift auth entry point

Fixes: #1776
Swift auth entry point is not hard coded anymore.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

osd: fill in user log entry last after snapdir tran

Reorder the snapdir logic and ctx->at_version adjustments prior to filling
in the object_info_t and user_versions and all that stuff.  Adjust
at_version after appending the log entry (so that it points to the next
position/version we will write at.. culminating in the actual user
event).

The user log entry contains the request id, which will be used
by replay ops to put themselves in the correct place in the
waiting_for_commit/ack maps.  Thus, the repop needs to be tagged
with the same version as the log entry with the request id.
Thus, the request id bearing log entry should be the last in
the log entry vector.

This should fix #3072, wherein a replay which should wait on
the repop tagged as version '36 will instead wait on '35.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

:doc: Fixed broken hyperlinks.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Addresses Documentation #3096. Also added new information.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

rgw: rgw-admin filter usage by categories

rgw can now filter usage output by specific op
categories.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

gitignore: Remove "nbproject", no idea what that even is.

Looks like this line was accidentally introduced in commit
af4d8db55f7268ab68ee5a7e17ac58c993528566.

Signed-off-by: Tommi Virtanen <tv@inktank.com>

rgw: usage now account categories

Instead of just keeping a flat usage info per bucket, we
now maintain a list of categories for which requests
usage is aggregated in. Ops are put in categories based
on their names.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix usage

Fixes: #3085
usage was showing 'bucket info' command that never
existed.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

:doc: Minor tweak to heading text.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge branch 'master' of github.com:ceph/ceph

:doc: Modified the index page to point to the new cluster-ops section.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Made minor changes to restructuredText headers.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Added comment redirecting editors to new page location.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Added index/toctree page for cluster ops.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Added new auth settings to reference doc.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Refactored and moved control.rst page.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Consolidated file system recommendations.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Minor syntax update.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: restructuredText syntax corrections.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Added index page. This will be refactored again soon.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Refactored and moved to ceph/docs/cluster-ops/pools.rst

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Removed. New section is in ceph/doc/cluster-ops/authentication.rst

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Fixed heading syntax.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

PG: clear want_acting in choose_acting if want == acting

Otherwise, a pg_temp from a previous peering sequence
(but not a different peering_interval) might leak through
into Active and incorrectly trip the
Active::react(AdvMap&) asserts regarding want_acting.
Those asserts assume that want_acting is either empty or is
a results of recovery completion.  In the latter case, the
want_acting set much consist only of elements of up and
acting.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge remote-tracking branch 'gh/wip_deep_scrub_final'

osd: initialize pg_log_entry_t::invalid_pool in default ctor

Signed-off-by: Sage Weil <sage@inktank.com>

doc: explain what scrub, deep-scrub, and repair actually do

Signed-off-by: Mike Ryan <mike.ryan@inktank.com>

osd: deep scrub, read file contents from disk and compare digest

Deep scrub reads the contents of every file from the store and computes
a crc32 digest. The primary compares the digest of all replicas and will
mark the PG inconsistent if any don't match.

OSDs that do not support deep scrub simply perform an ordinary chunky
scrub. Any subset of OSDs that do support deep scrub will have their
digests compared.

Signed-off-by: Mike Ryan <mike.ryan@inktank.com>

buffer: class for efficiently calculating CRC32 of >= 1 bufferlist

Signed-off-by: Mike Ryan <mike.ryan@inktank.com>

pg: store scrubber state in its own object

Signed-off-by: Mike Ryan <mike.ryan@inktank.com>

osd: chunky scrub, scrub PGs a chunk of objects at a time

Chunky scrub is a more efficient scrub. It blocks writes on a subset of
objects and scrubs those, allowing writes through to the rest of the PG.

The scrub takes longer to complete than a classic scrub, but improves
overall write throughput.

This feature is backward-compatible with classic scrub. If the primary
detects that any replica does not have the chunky scrub feature, it
falls back to the less efficient classic scrub.

Signed-off-by: Mike Ryan <mike.ryan@inktank.com>

crush: change default type from 'pool' to 'root'

The 'pool=default' in the default crush maps is confusing wrt rados pools.
'root' makes more sense given that we are talking about hierarchies/trees.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'upstream/wip-kvstore'

FileStore: get objects whose names fall within a range

Signed-off-by: Mike Ryan <mike.ryan@inktank.com>

pg: change _scrub() to take out parameters as pointers

Signed-off-by: Mike Ryan <mike.ryan@inktank.com>

Monitor.cc:  Added include for limits.h.

This include is needed on Centos.  It seems to be included implicitly
on other platforms.

Merge branch 'master' of github.com:ceph/ceph

:doc: Added a section for adding and removing monitors. Singificantly re-factored.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

:doc: Incorporated Joao's feedback into the reference material.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

docs: Add CloudStack documentation

The basic documentation about how you can use RBD with CloudStack

Signed-off-by: Wido den Hollander <wido@widodh.nl>

:doc: Added recovering from OSD failures. Will be re-factored again soon.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added monitor failure recovery. Will be re-factored again soon.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Re-factored adding an OSD.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Minor updates.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge branch 'master' of github.com:ceph/ceph

doc: Added admonishments for Ceph FS per http://tracker.newdream.net/issues/3077

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Updated to incorporate Sage's changes.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added anchor references.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Refactored the debug section to point back to reference.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added QA reference to --valgrind option.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Revert "ReplicatedPG: fill in user log entry last after snapdir tran"

This reverts commit 0aad5462eb79be0427004f2442903bb56c2057c1.

This gives us two events with the same version, and crashes like so:

osd/PG.cc: In function 'void PG::add_log_entry(pg_log_entry_t&, ceph::bufferlist&)' thread 7fd21b187700 time 2012-09-04 15:10:39.475385
osd/PG.cc: 2181: FAILED assert(e.version > info.last_update)

 ceph version 0.51-411-g40fd6ba (commit:40fd6ba8ed9ba70c8d20a79936f53f10f2dfe839)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x95) [0x1025139]
 2: (PG::add_log_entry(pg_log_entry_t&, ceph::buffer::list&)+0xb0) [0xe47552]
 3: (PG::append_log(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >&, eversion_t, ObjectStore::Transaction&)+0x1cd) [0xe47939]
 4: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>)+0x41f2) [0xcb5c84]
 5: (PG::do_request(std::tr1::shared_ptr<OpRequest>)+0x1f3) [0xe42329]
 6: (OSD::dequeue_op(PG*)+0x286) [0xd9a214]
 7: (OSD::OpWQ::_process(PG*)+0x27) [0xda20e7]
 8: (ThreadPool::WorkQueue<PG>::_void_process(void*)+0x2e) [0xdf1ab4]
 9: (ThreadPool::worker()+0x4ce) [0x101c762]
 10: (ThreadPool::WorkThread::entry()+0x1c) [0xda049e]
 11: (Thread::_entry_func(void*)+0x23) [0x1016a49]
 12: (()+0x7e9a) [0x7fd22b7fce9a]
 13: (clone()+0x6d) [0x7fd229db14bd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

rgw: use prefetched buffer even when ofs != 0

Beforehand we only used the prefetched buffer if read ofs
was 0.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>