cls_rgw: remove the write_bucket_dir function.

It's no longer called anywhere. Hurray, we don't do our own
read-modify-write cycle any more (and can exploit the power of
btrees later)!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cls_rgw: rewrite rgw_bucket_complete_op to use update.

Unfortunately we can't do multiple writes via the interface -- the
second one will clobber the first one. So use the update functionality
and go through that pain instead.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cls_rgw: refactor rgw_bucket_complete_op in terms of TMAP

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cls_rgw: refactor rgw_bucket_prepare_op in terms of tmap

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cls_rgw: refactor rgw_bucket_init_index in terms of tmap

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cls_rgw: refactor read_bucket_dir in terms of tmap.

This function won't be called often once refactoring is done, but
its functionality will be needed for listing, if nothing else.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

objclass: add map interfaces.

Right now, they implement the TMAP functions, plus a few obvious
extras to read/write select keys and the header. In the future it
should be easy to switch them to better mapping implementations.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ReplicatedPG: remove unused tmap implementation.

If it's surrounded by an if(0), it shouldn't still be in the code.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

test_librbd: expect copy to succeed

0 is the success return code. These were accidentally changed in the
conversion to gtest.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: return errors when read_iterate fails during copy

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

PG: merge_old_entry: merged delete might not be in missing

If the new log does not contain an entry for that oid, it might not yet
be in missing, and we would need to add it.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: swift key removal

Revert "config: base default libdir, sysconfdir off autoconf values"

This reverts commit 7e5dee907a8218647a88d1c7d3316cc277e1c44b.

config: base default libdir, sysconfdir off autoconf values

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: implement some missing swift api, other swift fixes

rgw: encoded swift key contains full user name

rgw: multiple swift keys per user

PG,OSD: clean up xlist::items on pg removal

Should take care of #1599.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd: fix race between op requeueing and _dispatch

If a message is working it's way through _dispatch, and another thread
requeues waiting messages under pg->lock (e.g.
osd->take_waiting(waiting_for_active)), the requeued ops are processed
after the one _dispatch() is chewing on, breaking client ordering.

Instead, add a new OSD::requeue_ops() that reinjects ops back into the
op queue by feeding them to the _handle_*() helpers.  Those do last minute
checks before enqueuing the ops.

Fixes: #1490 (again)
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

thread: whitespace

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: clean up error output

No color codes in the logs!

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: tolerate missing current/

We remove it anyway.  If it's missing entirely, just continue and roll
back to the latest snap_ when the user passes --osd-use-stale-snap.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec.in: fix radosgw files

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: find fcgi headers in /usr/include or /usr/include/fastcgi

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph.spec.in: more updates to make build.opensuse.org happy

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph.spec.in: add init-radosgw back

Accidentally took this out a few patches back.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: push log list|show|rm into RGWStore

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

radosgw-admin: push log listing into RGWRados

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: chdir to dir in conf (default /)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: push intent log processing into RGWRados layer

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librbd: don't return >32bit value from copy().

Signed-off-by: Sage Weil <sage@newdream.net>

.gitignore: test_libhadoopcephfs build

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge branch 'master' of github.com:NewDreamNetwork/ceph

librbd: fix overflow on large large image copy

Fixes: #1611
Signed-off-by: Sage Weil <sage@newdream.net>

librbd: slightly cleaner

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: fix copy progress

Signed-off-by: Sage Weil <sage@newdream.net>

hadoop: get the right class member, and ask for it properly.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

auth: move AuthAuthorizeHandler registry into class

Static classes with constructors and destructors are dangerous.  Explicitly
manage these as part of the server components (OSD, MDS).

Fixes: #1608
Signed-off-by: Sage Weil <sage@newdream.net>

OSD,ReplicatedPG: expire and cleanup unconnected watchers

During handle_notify_timeout or ms_handle_reset, watchers are now marked
unconnected via pg->register_unconnected_watcher. A safe timer event has
been added to trigger OSD::handle_watch_timeout.
remove_watchers_and_notifies (called on role change) cleans up these
events before peering.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

move ceph_mount call

osd: discard requests that from disconnected clients

If we can't reply, throw out the request; they'll need to resend it anyway.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec.in: handle docdir properly

Pass correct path to configure (fixes SuSE builds).

Use %doc command to install sample.ceph.conf and sample.fetch_conf.

Signed-off-by: Sage Weil <sage@newdream.net>

clitests: add test case for crush exhaustive search

This detects the bug that caused #1594.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Makefile: only reset docdir if it's not already defined

This is only for old versions of automake, and won't work for ancient
SuSe.

Signed-off-by: Sage Weil <sage@newdream.net>

crushtool: fix --weight arg parsing

Signed-off-by: Sage Weil <sage@newdream.net>

crushtool: add --force N

Signed-off-by: Sage Weil <sage@newdream.net>

crush: try _all_ bucket items when doing exhaustive search

N-1 isn't exhaustive.

Fixes: #1594
Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec.in: build on suse

Signed-off-by: Sage Weil <sage@newdream.net>

mon: tolerate 50ms (instead of 10ms) of clock drift

NTP on sepia can't manage 10ms, it seems.  Sigh.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: fix printf format warning

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: make log object naming configurable

Use date(1) codes for object name, plus %i and %n for bucket id/name, and
make UTC vs localtime configurable.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: make other send_incremental variant handle map discontinuity

Signed-off-by: Sage Weil <sage@newdream.net>

auth: fix authorizer leak fix

We were never reusing the authorizer, so make it local.  This fixes a crash
(double-free) on reset().

Signed-off-by: Sage Weil <sage@newdream.net>

mon: fix osdmap trimming unsigned overflow

This was trimming way to aggressively for early epochs due to the unsigned
version overflowing.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: cope with missing osdmaps in history

Signed-off-by: Sage Weil <sage@newdream.net>

osd, pg: ignore responses to obsolete queries

This adds a query_epoch to notify and log messages, which are
sent in response to queries from the primary during peering. To
guarantee we don't try to process old logs and notifies after
restarting peering, query_epoch is set to the epoch at which the
query was sent. If query_epoch is less than last_peering_reset,
the primary discards the message.

This caused a "bad state machine event" crash in the following
scenario:

1. Primary tells a stray to generate a backlog at epoch 199.
2. The up set changes because a stray goes up.
3. Primary restarts peering at epoch 200.
4. Stray gets new map for epoch 200, sees that acting set did not
change, and sends log to primary.
5. Primary crashes.

Related to #1403, #1449
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

pg: rename warm_restart and last_warm_restart

These names make more sense, since last_warm_restart was updated
outside of the warm_restart function.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

.gitignore: add multi_stress_watch

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: assert *_FLUSHING unreachable for AccessMode

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

test/multi_stress_watch.cc: add watch notify tester for teuthology

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

qa: rados-api: try harder to make these pool names unique

Still getting -17 sometimes on sepia... :/

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

xlist: more assertive

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Revert "osd: simplify the session ref-counting branches"

This reverts commit f002144ef7d314e80d3d19a5cd2c4cef38851056.

osd: boot with map of oldest-1

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: UTC for time in 'log show', local time in separate field

Signed-off-by: Sage Weil <sage@newdream.net>

utime: add gmtime() output function

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: for date filtering for 'log list'

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-leaks'

cephx: don't leak Authorizers on each request

It's not clear to me why this is a class member -- it's only
written to or read from in this function, which allocates a fresh
one each time it's called.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: put session ref in ms_handle_reset

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: simplify the session ref-counting branches

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: fix chdir result warning

warning: rgw/rgw_main.cc:265: ignoring return value of ‘int chdir(const char*)’, declared with attribute warn_unused_result

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: fix rgw_switch_url_previx printf arg

Signed-off-by: Sage Weil <sage@newdream.net>

osd: same_acting_since -> same_interval_since

Interval is (currently) defined as same acting and up sets.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix same_acting_since update

same_acting_since actually means "same acting AND up set since".  This is
what the other same_acting_since update does (see a22511d).

We should rename it same_interval_since or something similar.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: use conf for rgw_swift_url[_prefix] instead of fcgi env

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: use conf for rgw_print_continue instead of fcgi environment

Also remove unused expect100cont bool.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: use conf for rgw_dns_name instead of fcgi environment

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: add conf rgw_remote_addr_param instead of using fcgi environment

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix pg stats encoding for unfound

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

vstart: use normal name for .asok files

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: fix initscript default user

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: daemonize by default

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: set locator key before calling get_obj_state()

get_obj_state assumes the io_ctx is ready to go.  Set the locator key
earlier rather than later.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: normalize oloc.key

If oloc.key == oid, clear it.  Users can set it to the oid or leave it
blank; they are equivalent.

Signed-off-by: Sage Weil <sage@newdream.net>

cli: fix test contents again.

I don't know why this didn't come across properly from my wip branch...

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: fix map epoch check in handle_osd_map

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

paxos: update first_committed

Otherwise this stays at 0, which isn't accurate.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: fix mis-merge issues.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge branch 'wip-rgw-pools'

Conflicts:
src/rgw/rgw_bucket.cc
src/rgw/rgw_main.cc

monclient: add an initialized bool to guard shutdown.

The addition of a Finisher in 9c56070bc20878e87fcb4715b0a3559dd1aaf9ff
broke shutdown in the case where MonClient::init() was never called, so
add a guard variable to keep track.

I'm not sure this is actually the best solution (Timer guard itself,
for instance; maybe Finisher should too?), but I don't want to change
the Finisher interface without looking at it more carefully than I'm
going to right now.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove the rgw_buckets files

We don't need them any more. Hurrah!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove the rgw_get_bucket_info functions

Push them down into the RGWAccess layers.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: move rgw_store_bucket_info into RGWRados.

create_bucket is the only user, so just make it a simple helper.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove select_bucket_placement from RGWAccess interface.

RGWRados::create_bucket is the only user now, so make it private
and make the interface a little tighter.

(We are going to need to handle placement at some point in the future,
but the interface needs to be designed a lot more carefully than this
one [wasn't].)

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove rgw_create_bucket.

Push all its extra functionality down into RGWRados::create_bucket. Convert
callers to the different interface (there's no reason to pass in the
bucket name apart from the bucket, and all callers know if they're
using a system bucket or not).

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: new command to add pools to the list of available places for storage.

Do this by adding an add_bucket_placement function to RGWAccess. Hurray
leaky abstractions! :(

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge branch 'stable'

osd: fix PG::up calculation on startup

We weren't filling in pg->up, which meant we booked an incorrect past
interval in PG::warm_restart(), leading to unknown strangeness down the
line.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: fix maps sent after osd boot

This was a bad hunk that crept in while I was rebasing wip-oldmaps.  :(

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>