osd: fix up PG::Missing methods a bit

Pass in iterators when possible.  Stack methods instead of duplicating
functionality.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: factor out recover_primary_got() helper

This handles the missing set and lsat_complete adjustment when we recover
an object on the primary.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: make C_OSD_CommittedPushedObject::op optional

This lets us reuse this helper for commiting recovery ops that aren't a
result of a push.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: pass version explicitly to pull

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix share_pg_log()

We need to handle a log message in the ReplicaActive state.  And set the
epoch properly when we send it.

Signed-off-by: Sage Weil <sage@newdream.net>

messages/MOSDPG*: clean up output a bit

Signed-off-by: Sage Weil <sage@newdream.net>

osd: remove superfluous write_info calls

- merge_log() will write_info (and log) as needed
- Activate() will do the same

Signed-off-by: Sage Weil <sage@newdream.net>

osd: all_unfound_are_queried_or_lost

The check to make isn't whether all locations are lost, but whether all
locations are either lost or have been queried and don't have the object(s)
we want.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: adjust LOST log entry types; simplify log entry type strings

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix up mark_all_unfound_lost so that it actually works

Well, it works given our weak definition of LOST.

- use ObjectContexts properly
- move into ReplicatedPG
- no need for _as_ in name

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: implement 'flush_pg_stats' command

This flushes the current pg stats to the monitor, and blocks until the
monitor commits it.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: process commands in a workqueue

This lets us do commands that can potentially block.  For example:

 - flush pg stats to osd
 - request (and wait for) latest osdmap

Currently the threadpool only has 1 thread.  i.e., one concurrent command.
That should be fine, methinks.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: feed MPGStats tids back through the MPGStatsAck

Signed-off-by: Sage Weil <sage@newdream.net>

osd: remove some pg stats debug cruft

Signed-off-by: Sage Weil <sage@newdream.net>

osd: handle (and reply to) direct MCommands

Signed-off-by: Sage Weil <sage@newdream.net>

cephtool: ability to send commands directly to osds

This makes commands beginning with 'tell <target>' magic in that they go
to the given target instead of to the monitor.  This is slightly odd, but
I think it gives the most natural interface for the user, with the tool
Doing The Right Thing for you.  E.g.,

 ceph tell <someone> something      (direct to some daemon)
 ceph do something                  (goes to monitor to do X)

Signed-off-by: Sage Weil <sage@newdream.net>

msg: entity_name_t::parse()

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msg: add MCommand, MCommandReply message types

These are similar to MMonCommand[Ack], but aren't PaxosServiceMessage
children, don't include the command in the reply (useless), have a more
generic name.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: assert on any unexpected error

Right now, the only errors we expect out of the underlying filesystem are
-ENOENT, -ENODATA, or (as a workaround for extN xattr suckage) -ENOSPC
for certain setxattr operations.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: make number of old paxos states configurable

Currently settable on osdmaps, pgmaps, and log.  Still need MDSMap and
authmap trimming.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: share oldest_map info with peers

This helps OSDs trim their old maps even when they don't get MOSDMap
messages directly from the monitor.

It also feeds that information to clients, although they don't use it yet.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: send full map if we don't have sufficiently old incremental

If the peer has a really old map, send a full map instead of crashing
because we are missing the needed incremental.

Signed-off-by: Sage Weil <sage@newdream.net>

paxos: trim extra state dirs

OSDMonitor, for instance, stores both an "osdmap" and "osdmap_full" for
each state.  Trim them both.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: remove old osdmaps

When the monitor removes old maps, we should too.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

PG: call set_last_peering_reset in Started contructor

Calling it here should cover all possible replica and primary peering
resets.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: Fix log.empty confusion

Previously, log.empty meant that the log.head was everion_t().  However,
it was in a few places used to mean that log.head == log.tail.  Now,
log.empty means log.head == log.tail and log.null() indicates that
log.head is eversion_t().

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

makefile changes for interval tree

Added unit test case for interval tree to the makefile template.

Signed-off-by: Jojy George Varghese <jvarghese@scalecomputing.com>

mds: Unit tests for interval tree

Provides usage scenarios and test cases for interval tree
implementation.

Tests include:
 - testing addInterval interface
 - testing removeInterval interfaces
 - testing with various template parameters

Signed-off-by: Jojy George Varghese <jvarghese@scalecomputing.com>

mds: Interval tree implementation

Interval tree is an optimized data structure for representing and
querying intervals. Elementary intervals are represented as nodes of an
avl tree and the corresponding data is stored on these nodes based on a
concept of span. This representation allows log(n) (where n is the
number of data) storage. The balanced avl tree allows a log(n) query.
The implementation is a template class that is instantiated based on
parameters : - Interval type - Data type

Signed-off-by: Jojy George Varghese <jvarghese@scalecomputing.com>

auth: remove global instance of auth_supported

Wrap it in a class.

Instantiate locally, or keep a copy around if we'll need it often.

Factor out the protocol selection into an AuthSupported method.  Prefer
larger ids, for lack of a better policy.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: bound generate_past_intervals() by oldest map

The oldest osdmap we maintain is a lower bound on last_epoch_clean for the
entire system (assuming the monitor is doing it's job right).  We can stop
generating past intervals when we hit it.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

cls_rgw: remove the write_bucket_dir function.

It's no longer called anywhere. Hurray, we don't do our own
read-modify-write cycle any more (and can exploit the power of
btrees later)!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cls_rgw: rewrite rgw_bucket_complete_op to use update.

Unfortunately we can't do multiple writes via the interface -- the
second one will clobber the first one. So use the update functionality
and go through that pain instead.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cls_rgw: refactor rgw_bucket_complete_op in terms of TMAP

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cls_rgw: refactor rgw_bucket_prepare_op in terms of tmap

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cls_rgw: refactor rgw_bucket_init_index in terms of tmap

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cls_rgw: refactor read_bucket_dir in terms of tmap.

This function won't be called often once refactoring is done, but
its functionality will be needed for listing, if nothing else.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

objclass: add map interfaces.

Right now, they implement the TMAP functions, plus a few obvious
extras to read/write select keys and the header. In the future it
should be easy to switch them to better mapping implementations.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ReplicatedPG: remove unused tmap implementation.

If it's surrounded by an if(0), it shouldn't still be in the code.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

test_librbd: expect copy to succeed

0 is the success return code. These were accidentally changed in the
conversion to gtest.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: return errors when read_iterate fails during copy

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

PG: merge_old_entry: merged delete might not be in missing

If the new log does not contain an entry for that oid, it might not yet
be in missing, and we would need to add it.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: swift key removal

Revert "config: base default libdir, sysconfdir off autoconf values"

This reverts commit 7e5dee907a8218647a88d1c7d3316cc277e1c44b.

config: base default libdir, sysconfdir off autoconf values

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: implement some missing swift api, other swift fixes

rgw: encoded swift key contains full user name

rgw: multiple swift keys per user

PG,OSD: clean up xlist::items on pg removal

Should take care of #1599.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd: fix race between op requeueing and _dispatch

If a message is working it's way through _dispatch, and another thread
requeues waiting messages under pg->lock (e.g.
osd->take_waiting(waiting_for_active)), the requeued ops are processed
after the one _dispatch() is chewing on, breaking client ordering.

Instead, add a new OSD::requeue_ops() that reinjects ops back into the
op queue by feeding them to the _handle_*() helpers.  Those do last minute
checks before enqueuing the ops.

Fixes: #1490 (again)
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

thread: whitespace

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: clean up error output

No color codes in the logs!

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: tolerate missing current/

We remove it anyway.  If it's missing entirely, just continue and roll
back to the latest snap_ when the user passes --osd-use-stale-snap.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec.in: fix radosgw files

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: find fcgi headers in /usr/include or /usr/include/fastcgi

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph.spec.in: more updates to make build.opensuse.org happy

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph.spec.in: add init-radosgw back

Accidentally took this out a few patches back.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: push log list|show|rm into RGWStore

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

radosgw-admin: push log listing into RGWRados

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: chdir to dir in conf (default /)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: push intent log processing into RGWRados layer

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librbd: don't return >32bit value from copy().

Signed-off-by: Sage Weil <sage@newdream.net>

.gitignore: test_libhadoopcephfs build

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge branch 'master' of github.com:NewDreamNetwork/ceph

librbd: fix overflow on large large image copy

Fixes: #1611
Signed-off-by: Sage Weil <sage@newdream.net>

librbd: slightly cleaner

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: fix copy progress

Signed-off-by: Sage Weil <sage@newdream.net>

hadoop: get the right class member, and ask for it properly.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

auth: move AuthAuthorizeHandler registry into class

Static classes with constructors and destructors are dangerous.  Explicitly
manage these as part of the server components (OSD, MDS).

Fixes: #1608
Signed-off-by: Sage Weil <sage@newdream.net>

OSD,ReplicatedPG: expire and cleanup unconnected watchers

During handle_notify_timeout or ms_handle_reset, watchers are now marked
unconnected via pg->register_unconnected_watcher. A safe timer event has
been added to trigger OSD::handle_watch_timeout.
remove_watchers_and_notifies (called on role change) cleans up these
events before peering.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

move ceph_mount call

osd: discard requests that from disconnected clients

If we can't reply, throw out the request; they'll need to resend it anyway.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec.in: handle docdir properly

Pass correct path to configure (fixes SuSE builds).

Use %doc command to install sample.ceph.conf and sample.fetch_conf.

Signed-off-by: Sage Weil <sage@newdream.net>

clitests: add test case for crush exhaustive search

This detects the bug that caused #1594.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Makefile: only reset docdir if it's not already defined

This is only for old versions of automake, and won't work for ancient
SuSe.

Signed-off-by: Sage Weil <sage@newdream.net>

crushtool: fix --weight arg parsing

Signed-off-by: Sage Weil <sage@newdream.net>

crushtool: add --force N

Signed-off-by: Sage Weil <sage@newdream.net>

crush: try _all_ bucket items when doing exhaustive search

N-1 isn't exhaustive.

Fixes: #1594
Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec.in: build on suse

Signed-off-by: Sage Weil <sage@newdream.net>

mon: tolerate 50ms (instead of 10ms) of clock drift

NTP on sepia can't manage 10ms, it seems.  Sigh.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: fix printf format warning

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: make log object naming configurable

Use date(1) codes for object name, plus %i and %n for bucket id/name, and
make UTC vs localtime configurable.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: make other send_incremental variant handle map discontinuity

Signed-off-by: Sage Weil <sage@newdream.net>

auth: fix authorizer leak fix

We were never reusing the authorizer, so make it local.  This fixes a crash
(double-free) on reset().

Signed-off-by: Sage Weil <sage@newdream.net>

mon: fix osdmap trimming unsigned overflow

This was trimming way to aggressively for early epochs due to the unsigned
version overflowing.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: cope with missing osdmaps in history

Signed-off-by: Sage Weil <sage@newdream.net>

osd, pg: ignore responses to obsolete queries

This adds a query_epoch to notify and log messages, which are
sent in response to queries from the primary during peering. To
guarantee we don't try to process old logs and notifies after
restarting peering, query_epoch is set to the epoch at which the
query was sent. If query_epoch is less than last_peering_reset,
the primary discards the message.

This caused a "bad state machine event" crash in the following
scenario:

1. Primary tells a stray to generate a backlog at epoch 199.
2. The up set changes because a stray goes up.
3. Primary restarts peering at epoch 200.
4. Stray gets new map for epoch 200, sees that acting set did not
change, and sends log to primary.
5. Primary crashes.

Related to #1403, #1449
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

pg: rename warm_restart and last_warm_restart

These names make more sense, since last_warm_restart was updated
outside of the warm_restart function.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

.gitignore: add multi_stress_watch

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: assert *_FLUSHING unreachable for AccessMode

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

test/multi_stress_watch.cc: add watch notify tester for teuthology

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

qa: rados-api: try harder to make these pool names unique

Still getting -17 sometimes on sepia... :/

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

xlist: more assertive

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Revert "osd: simplify the session ref-counting branches"

This reverts commit f002144ef7d314e80d3d19a5cd2c4cef38851056.

osd: boot with map of oldest-1

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: UTC for time in 'log show', local time in separate field

Signed-off-by: Sage Weil <sage@newdream.net>

utime: add gmtime() output function

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: for date filtering for 'log list'

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-leaks'

cephx: don't leak Authorizers on each request

It's not clear to me why this is a class member -- it's only
written to or read from in this function, which allocates a fresh
one each time it's called.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>