radosgw-admin: push log listing into RGWRados

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: chdir to dir in conf (default /)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: push intent log processing into RGWRados layer

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

librbd: don't return >32bit value from copy().

Signed-off-by: Sage Weil <sage@newdream.net>

.gitignore: test_libhadoopcephfs build

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge branch 'master' of github.com:NewDreamNetwork/ceph

librbd: fix overflow on large large image copy

Fixes: #1611
Signed-off-by: Sage Weil <sage@newdream.net>

librbd: slightly cleaner

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: fix copy progress

Signed-off-by: Sage Weil <sage@newdream.net>

hadoop: get the right class member, and ask for it properly.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

auth: move AuthAuthorizeHandler registry into class

Static classes with constructors and destructors are dangerous.  Explicitly
manage these as part of the server components (OSD, MDS).

Fixes: #1608
Signed-off-by: Sage Weil <sage@newdream.net>

OSD,ReplicatedPG: expire and cleanup unconnected watchers

During handle_notify_timeout or ms_handle_reset, watchers are now marked
unconnected via pg->register_unconnected_watcher. A safe timer event has
been added to trigger OSD::handle_watch_timeout.
remove_watchers_and_notifies (called on role change) cleans up these
events before peering.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

move ceph_mount call

osd: discard requests that from disconnected clients

If we can't reply, throw out the request; they'll need to resend it anyway.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec.in: handle docdir properly

Pass correct path to configure (fixes SuSE builds).

Use %doc command to install sample.ceph.conf and sample.fetch_conf.

Signed-off-by: Sage Weil <sage@newdream.net>

clitests: add test case for crush exhaustive search

This detects the bug that caused #1594.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Makefile: only reset docdir if it's not already defined

This is only for old versions of automake, and won't work for ancient
SuSe.

Signed-off-by: Sage Weil <sage@newdream.net>

crushtool: fix --weight arg parsing

Signed-off-by: Sage Weil <sage@newdream.net>

crushtool: add --force N

Signed-off-by: Sage Weil <sage@newdream.net>

crush: try _all_ bucket items when doing exhaustive search

N-1 isn't exhaustive.

Fixes: #1594
Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec.in: build on suse

Signed-off-by: Sage Weil <sage@newdream.net>

mon: tolerate 50ms (instead of 10ms) of clock drift

NTP on sepia can't manage 10ms, it seems.  Sigh.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: fix printf format warning

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: make log object naming configurable

Use date(1) codes for object name, plus %i and %n for bucket id/name, and
make UTC vs localtime configurable.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: make other send_incremental variant handle map discontinuity

Signed-off-by: Sage Weil <sage@newdream.net>

auth: fix authorizer leak fix

We were never reusing the authorizer, so make it local.  This fixes a crash
(double-free) on reset().

Signed-off-by: Sage Weil <sage@newdream.net>

mon: fix osdmap trimming unsigned overflow

This was trimming way to aggressively for early epochs due to the unsigned
version overflowing.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: cope with missing osdmaps in history

Signed-off-by: Sage Weil <sage@newdream.net>

osd, pg: ignore responses to obsolete queries

This adds a query_epoch to notify and log messages, which are
sent in response to queries from the primary during peering. To
guarantee we don't try to process old logs and notifies after
restarting peering, query_epoch is set to the epoch at which the
query was sent. If query_epoch is less than last_peering_reset,
the primary discards the message.

This caused a "bad state machine event" crash in the following
scenario:

1. Primary tells a stray to generate a backlog at epoch 199.
2. The up set changes because a stray goes up.
3. Primary restarts peering at epoch 200.
4. Stray gets new map for epoch 200, sees that acting set did not
change, and sends log to primary.
5. Primary crashes.

Related to #1403, #1449
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

pg: rename warm_restart and last_warm_restart

These names make more sense, since last_warm_restart was updated
outside of the warm_restart function.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

.gitignore: add multi_stress_watch

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: assert *_FLUSHING unreachable for AccessMode

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

test/multi_stress_watch.cc: add watch notify tester for teuthology

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

qa: rados-api: try harder to make these pool names unique

Still getting -17 sometimes on sepia... :/

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

xlist: more assertive

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Revert "osd: simplify the session ref-counting branches"

This reverts commit f002144ef7d314e80d3d19a5cd2c4cef38851056.

osd: boot with map of oldest-1

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: UTC for time in 'log show', local time in separate field

Signed-off-by: Sage Weil <sage@newdream.net>

utime: add gmtime() output function

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: for date filtering for 'log list'

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-leaks'

cephx: don't leak Authorizers on each request

It's not clear to me why this is a class member -- it's only
written to or read from in this function, which allocates a fresh
one each time it's called.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: put session ref in ms_handle_reset

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: simplify the session ref-counting branches

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: fix chdir result warning

warning: rgw/rgw_main.cc:265: ignoring return value of ‘int chdir(const char*)’, declared with attribute warn_unused_result

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: fix rgw_switch_url_previx printf arg

Signed-off-by: Sage Weil <sage@newdream.net>

osd: same_acting_since -> same_interval_since

Interval is (currently) defined as same acting and up sets.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix same_acting_since update

same_acting_since actually means "same acting AND up set since".  This is
what the other same_acting_since update does (see a22511d).

We should rename it same_interval_since or something similar.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: use conf for rgw_swift_url[_prefix] instead of fcgi env

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: use conf for rgw_print_continue instead of fcgi environment

Also remove unused expect100cont bool.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: use conf for rgw_dns_name instead of fcgi environment

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: add conf rgw_remote_addr_param instead of using fcgi environment

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix pg stats encoding for unfound

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

vstart: use normal name for .asok files

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: fix initscript default user

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: daemonize by default

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: set locator key before calling get_obj_state()

get_obj_state assumes the io_ctx is ready to go.  Set the locator key
earlier rather than later.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: normalize oloc.key

If oloc.key == oid, clear it.  Users can set it to the oid or leave it
blank; they are equivalent.

Signed-off-by: Sage Weil <sage@newdream.net>

cli: fix test contents again.

I don't know why this didn't come across properly from my wip branch...

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: fix map epoch check in handle_osd_map

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

paxos: update first_committed

Otherwise this stays at 0, which isn't accurate.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: fix mis-merge issues.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge branch 'wip-rgw-pools'

Conflicts:
src/rgw/rgw_bucket.cc
src/rgw/rgw_main.cc

monclient: add an initialized bool to guard shutdown.

The addition of a Finisher in 9c56070bc20878e87fcb4715b0a3559dd1aaf9ff
broke shutdown in the case where MonClient::init() was never called, so
add a guard variable to keep track.

I'm not sure this is actually the best solution (Timer guard itself,
for instance; maybe Finisher should too?), but I don't want to change
the Finisher interface without looking at it more carefully than I'm
going to right now.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove the rgw_buckets files

We don't need them any more. Hurrah!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove the rgw_get_bucket_info functions

Push them down into the RGWAccess layers.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: move rgw_store_bucket_info into RGWRados.

create_bucket is the only user, so just make it a simple helper.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove select_bucket_placement from RGWAccess interface.

RGWRados::create_bucket is the only user now, so make it private
and make the interface a little tighter.

(We are going to need to handle placement at some point in the future,
but the interface needs to be designed a lot more carefully than this
one [wasn't].)

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove rgw_create_bucket.

Push all its extra functionality down into RGWRados::create_bucket. Convert
callers to the different interface (there's no reason to pass in the
bucket name apart from the bucket, and all callers know if they're
using a system bucket or not).

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: new command to add pools to the list of available places for storage.

Do this by adding an add_bucket_placement function to RGWAccess. Hurray
leaky abstractions! :(

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge branch 'stable'

osd: fix PG::up calculation on startup

We weren't filling in pg->up, which meant we booked an incorrect past
interval in PG::warm_restart(), leading to unknown strangeness down the
line.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: fix maps sent after osd boot

This was a bad hunk that crept in while I was rebasing wip-oldmaps.  :(

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip-rgw-log'

osd: remove some dead code

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: write debug acl output to the debug output (not cout)

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: more sane defaults

Notably, this gives us log_to_stderr = 1.

Signed-off-by: Sage Weil <sage@newdream.net>

common: CINIT_FLAG_NO_BANNER

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: rgw_log -> debug_rgw

Normalize with the rest of the code base.

Also, 'rgw log' might be confused with 'log file'.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: remove useless RGWConf::log_level

Not used for anything.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: set dout condvar to rgw_log

We would normally set DOUT_SUBSYS and get debug_$foo, but the setting in
this case is rgw_log.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: RGW_LOG -> dout

Signed-off-by: Sage Weil <sage@newdream.net>

rados-api/test: don't srand(time())

This will give duplicate results if called more than once per second, which
leads to things like -EEXIST (#1558).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: make MPoolOp*::print() readable

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'wip-oldmaps'

radosgw: run as 'user' field in ceph.conf

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: set xmlns on ListBucketResult element

Certain S3 clients (e.g., s3fs) require the xmlns to parse the
results correctly.

Signed-off-by: Henry C Chang <henry.cy.chang@gmail.com>

osd: get latest osdmaps before booting

- get the latest osdmaps before adding/marking ourselves up
- behave if there is a discontinuity in the osdmap history

This lets us behave sanely if an osd has been down for a very long time,
or if we replace (wipe) an osd, or otherwise take a fresh new osd and add
it to an aged cluster (with lots of old maps or, more likely, an oldest
map that has a large epoch).

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: include addrs in json dump

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: show weight for out osds

This info is reduncant (weight is 0 for out osds), but makes the output
cleaner.

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: show old addrs for down osds

This is useful for admins to find which ip/node to connect to.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: limit maps send on onetime osdmap subscribe

This throttles the load put on the monitor by making the client request
osdmaps in discrete chunks.  Use a feature bit to control this, since the
old kernel clients will only send a single onetime request if they think
they are behind.

Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: include oldest, newest known epoch in message

If non-zero, these are filled in with the oldest and newest map epochs
available.  This helps the client decide whether to ask for newer or older
maps.

Some minor refactoring in the monitor.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: use pointers for handle_advance_map

So that lastmap can be optional.

Signed-off-by: Sage Weil <sage@newdream.net>

objecter: fix locking on is_latest_map checks

We need to take the lock protecting these structures in the completion
context.

Signed-off-by: Sage Weil <sage@newdream.net>

monclient: use an async finisher

Otherwise we can deadlock if any completions call back into MonClient.

Signed-off-by: Sage Weil <sage@newdream.net>

monclient: clean up old version_requests on session reset

This throws out the old contexts.  We assume that the caller will not wait
for them.

Signed-off-by: Sage Weil <sage@newdream.net>

monclient: implement get_version() to get newest _and_ oldest map versions

Reimplement is_latest_map() in terms of more general helper.  Add field
to reply message.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: is_active() instead of !is_booting() for map sharing checks

Signed-off-by: Sage Weil <sage@newdream.net>

osd: do not request osdmap when loner if not booted

Signed-off-by: Sage Weil <sage@newdream.net>