rgw: remove select_bucket_placement from RGWAccess interface.

RGWRados::create_bucket is the only user now, so make it private
and make the interface a little tighter.

(We are going to need to handle placement at some point in the future,
but the interface needs to be designed a lot more carefully than this
one [wasn't].)

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove rgw_create_bucket.

Push all its extra functionality down into RGWRados::create_bucket. Convert
callers to the different interface (there's no reason to pass in the
bucket name apart from the bucket, and all callers know if they're
using a system bucket or not).

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: new command to add pools to the list of available places for storage.

Do this by adding an add_bucket_placement function to RGWAccess. Hurray
leaky abstractions! :(

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: don't specify create_pool and set_marker in create_bucket.

It's wildly inappropriate for that kind of implementation detail to
leak out of the interface. For the moment, leave in a "system_bucket"
parameter, which provides the necessary information. But we should
probably be able to push all of that information down below the
interface layer eventually, and then get rid of that param too.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: move rgw_bucket_select_host_pool behind RGWAccess as select_bucket_placement

This doesn't really belong in front of the interface. Maybe later we
can hide it completely, but for now we can put it behind the RGWAccess
interface in a way that at least pretends to be backing-store-agnostic.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove withdraw_pool function.

I think its use was broken anyway?

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove preallocation of pools

Rename rgw_bucket_allocate_pool to rgw_bucket_select_host_pool, since
that better describes functionality we might actually use someday (and
what it's actually doing now).

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove preallocating pools maintenance tick

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge branch 'wip-pool-sharing'

doc: add documentation for librbd python bindings

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: add class and method documentation

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: use a dict to translate error codes

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: allow setting an image to no snapshot

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: turn EROFS into an exception

This is returned when trying to write to a snapshot.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: don't remove on-disk bucket object in rgw_user::rgw_remove_bucket

This is the same object that rgw_rados removes from the root pool now.
And since nobody calls rgw_remove_bucket_info, kill it.
(Hopefully we can kill all of the pi_buckets functions soon.)

While we're at it, rename rgw_remove_bucket to rgw_remove_user_bucket_info,
since that's what it does.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: revert part of commit:30b814e2677659ddda4109a1421f33a47d83a05b

It's more consistent for RGWRados to delete the bucket object, rather
than the rgw_user functions.
And commenting functions out is not a happy practice.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: don't remove bucket object twice

Or else the second attempt will faile with -ENOENT.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: fix bucket-id assignment

Get the version from the same io_ctx that you write with.

Also, use a private one, so that we don't race with other threads.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: separate bucket vs object owner tracking

- track bucket_owner in session
- separte bucket and object owner fields in log entries

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: fix acl vs content-length check

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: rename clitest dir

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-pool-sharing' of github.com:NewDreamNetwork/ceph into wip-pool-sharing

Conflicts:
src/rgw/rgw_bucket.h

radosgw-admin: fix help clitest

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: set log entry owner to user if s->acl is not set

This is probably wrong, but it fixes the immediate problem of the first
entry in the log not having the owner field set.  The 'log show' looks at
that first entry to spit out key bucket metadata.

A better approach is probably to put that bucket/log metadata (bucket_id,
owner) that doesn't change as an xattr on the object instead of dupilcating
it in every record.  Or something like that...

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: simplify bucket info storage

Use a single method to store bucket metadata.

Store it in both locations, so it never diverges.  (This can still be
cleaned up.. probably one object should just point to the other, so the
data is only in one place.)

Disable to 'pool create' thing for now, which is still confusing me...

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: pull 'log show' owner+bucket from first entry

Don't rely on the bucketinfo object for this, as it may be deleted.  Having
it there most of the time but not always is going to mean breakage for
users.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: fix RGWBucketInfo encoding of owner

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: fix log retry after bucket creation

We were clobbering the object name for some reason.. probably cruft from
some old change.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: store bucket information in the .rgw pool instead of .buckets

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

radosgw_admin: we do want to set markers on these buckets

Commit 25499b644b1c5a7b6f4c8876261334dda10fbec3 introduced the initial
work for pool sharing and markers, but didn't set radosgw_admin to
use that infrastructure. We definitely want to be doing so, though,
since radosgw_admin isn't creating system buckets. Do so.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Cond.h: adjust nlock during waits

Previously, is_locked could erroneously return true if there were any
threads waiting on a cond on that lock.  In particular, any lock
associated with a timer would always return true for is_locked if any
events were scheduled on the timer.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: dump_watchers in do_osd_op_effects

dump_watchers requires the osd watch_lock.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

librados: Make notify a linger op

Notifies must be refreshed when the mapping changes.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

test_async_snap: loop a finite number of times.

Signed-off-by: Sage Weil <sage@newdream.net>

qa: btrfs: test_async_snap

Hammer on the btrfs async sync/snap create/remove ioctls.

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: 'log list' and 'log rm'

The normal object listing API seemed like a real poor match here so I
hacked in something that just lists the raw rados objects.  It also looks
like that may need some fixing.

This patch at least establishes a usable interface that is purely
radosgw-admin based for listing, fetching, and removing logs, so users can
avoid looking at rados pools directly.

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: dump ints, not strings

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: fix bucket stats

- 'usage' object instead of 'categories' array.
- dump ints, not strings.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: protect the id-based accesses from namespace collisions

Users can create buckets named "8", but they can't name them ".8". So
prepend the actual on-disk objects with a period.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove get_bucket_info and convert users.

This just lives in struct rgw_bucket now.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

radosgw-admin: fix 'bucket stats --uid='

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: include 'suspended' in user info output

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: clean up some cout usage

Errors to stderr.  Avoid cout for commands with structured output.

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: use formatter for 'bucket list'

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: 'bucket stats --uid'

List stats for all buckets on for the given user.

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: clean up 'bucket list' output

Signed-off-by: Sage Weil <sage@newdream.net>

rgw_admin: rip out format.empty() special cases

If we want a plain output, we should implement a PlainFormatter of some
sort.  For now, pretty json is good enough.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Conflicts:

src/rgw/rgw_admin.cc

rgw_admin: fix --pretty-format and --purge-data flag parsing

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Conflicts:

src/rgw/rgw_admin.cc

rgw_admin: default to json output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

radosgw-admin: use formatter for 'bucket list'

Signed-off-by: Sage Weil <sage@newdream.net>

rdosgw_admin -> radosgw-admin

Be consistent with other binary tool naming.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: remove now-unused struct RGWPoolInfo

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove rgw_[retrieve|store]_pool_info and ".pool" pool

Convert these to rgw_[get|store]_bucket_info, add the owner field
to bucket info, and store both lookup formats in the ".bucket" pool.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove references to pool_id and replace with new bucket_id.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Use LIBEDIT_LIBS in Makefile.am

The editline library might require some additional libs, so
instead of hardcoding '-ledit' we should be using LIBEDIT_LIBS
in Makefile.am

Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-dotnames'

clientN -> client.N

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

monN -> mon.N

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mdsN -> mds.N

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osdN -> osd.N

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

msgr: print entity_name_t with a .

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix leak of src_obc references

We never moved the src_obc map from the local var into the RepGather
member!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

test-rados-api: test tmap rm

Make sure we get ENOENT if the entry doesn't exist.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: error out current op only on tmap ENOENT

If an individual op in do_osd_ops() fails, set result, and continue to the
end of the loop.  Normally we'll return the error then, but if the ignore
error flag is set we'll continue.

Fix the tmap error handling so that we don't modify the tmap if the op
failed.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: some PgPriorSet comments, trivial cleanups, new FIXME

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: skip replay phase if osd_replay_window=0

The replay phase is only useful if there are clients waiting for ACK (but
not COMMIT) and expect ordering to be preserved across OSD failures.
Currently nobody does that except the client in sync io mode (shared
writers).  For installations where we don't care (e.g., using rados only),
we can disable replay altogether.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ReplicatedPG: remove assert in remove_watcher

notif is not in notif->session->notifs.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: don't encode the bucket name twice in RGWPoolInfo!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

ReplicatedPG: clean up watched objects on role change

Also removes inconsistent watch handling in handle_notify_timeout.
We will need to handle watch timeouts elsewhere.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: put bucketid allocator object in root (.rgw) pool

This keeps the .rgw.control pool disposable.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: use same prefix for locator key as for oid

If the bucket has a prefix for oids, prepend the same prefix to the
locator key.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-authtool: fix -C

Fixes: #1578
Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec.in: radosgw init script

Signed-off-by: Sage Weil <sage@newdream.net>

mon: auth: implement 'auth print-key <name>'

Signed-off-by: Sage Weil <sage@newdream.net>

mon: fix error return codes for 'auth ...'

Signed-off-by: Sage Weil <sage@newdream.net>

ceph.spec.in: add rbd python bindings

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

debian: make python-ceph depend on librbd

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

pybind: fix errors found by pyflakes

Unused imports, typos, and unused variables.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

debian: move ceph and ceph-authtool into ceph-common

Signed-off-by: Sage Weil <sage@newdream.net>

mon: auth: generate random key on 'auth add ...' if no keyring is provided

Signed-off-by: Sage Weil <sage@newdream.net>

debian: ceph-client-tools -> ceph-common, now required, w/ ceph-conf etc.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: list buckets using bucket objects

Don't assume the pools == buckets.  List the bucket objects in the .rgw
pool.

Fixes: #1560
Signed-off-by: Sage Weil <sage@newdream.net>

osdmap: remove dead code

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'origin/master' into wip-pool-sharing

Conflicts:
src/osd/OSD.cc
src/rgw/rgw_rados.cc

Merge branch 'stable'

radosgw: fix content-length checks

Ignore content-length check if the REQUEST_URI has a trailing ?acl.

(The object_str.find() check was wrong because object_str has the ?whatever
stripped off, and because it returns std::string::npos on failure.)

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: tolerate race creating .rgw* metadata pools

There may be multiple people trying to create these pools.  That's just
fine.

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: fix end_block calculation for read_iterate

All end block calculations match now.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

test_rbd.py: Add large read and write tests

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: don't zero negative lengths

They are past the end of the buffer (and wrap around).

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: be paranoid about strings

This prevents bad conversions (i.e. from int to char *).

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: remove useless version object

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: be paranoid about strings

This prevents bad conversions (i.e. from int to char *).

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: explicitly cast strings to c_char_p for type clarity

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: pass 0 for the default order, not NULL

librbd expects the order pointer to be valid - if it points to 0, the
default order is used.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: make exceptions inherit from a base class

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: allow configuring rados through constructor options

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: make Image a context manager

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: make Rados and Ioctx objects context managers

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: make exceptions inherit from a base class

This way you can catch them all.
Also remove the unused NoData exceptions
- this was just useful for xattrs in librados.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>