rgw: separate bucket vs object owner tracking

- track bucket_owner in session
- separte bucket and object owner fields in log entries

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: fix acl vs content-length check

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: rename clitest dir

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-pool-sharing' of github.com:NewDreamNetwork/ceph into wip-pool-sharing

Conflicts:
src/rgw/rgw_bucket.h

radosgw-admin: fix help clitest

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: set log entry owner to user if s->acl is not set

This is probably wrong, but it fixes the immediate problem of the first
entry in the log not having the owner field set.  The 'log show' looks at
that first entry to spit out key bucket metadata.

A better approach is probably to put that bucket/log metadata (bucket_id,
owner) that doesn't change as an xattr on the object instead of dupilcating
it in every record.  Or something like that...

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: simplify bucket info storage

Use a single method to store bucket metadata.

Store it in both locations, so it never diverges.  (This can still be
cleaned up.. probably one object should just point to the other, so the
data is only in one place.)

Disable to 'pool create' thing for now, which is still confusing me...

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: pull 'log show' owner+bucket from first entry

Don't rely on the bucketinfo object for this, as it may be deleted.  Having
it there most of the time but not always is going to mean breakage for
users.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: fix RGWBucketInfo encoding of owner

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: fix log retry after bucket creation

We were clobbering the object name for some reason.. probably cruft from
some old change.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: store bucket information in the .rgw pool instead of .buckets

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

radosgw_admin: we do want to set markers on these buckets

Commit 25499b644b1c5a7b6f4c8876261334dda10fbec3 introduced the initial
work for pool sharing and markers, but didn't set radosgw_admin to
use that infrastructure. We definitely want to be doing so, though,
since radosgw_admin isn't creating system buckets. Do so.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

radosgw-admin: 'log list' and 'log rm'

The normal object listing API seemed like a real poor match here so I
hacked in something that just lists the raw rados objects.  It also looks
like that may need some fixing.

This patch at least establishes a usable interface that is purely
radosgw-admin based for listing, fetching, and removing logs, so users can
avoid looking at rados pools directly.

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: dump ints, not strings

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: fix bucket stats

- 'usage' object instead of 'categories' array.
- dump ints, not strings.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: protect the id-based accesses from namespace collisions

Users can create buckets named "8", but they can't name them ".8". So
prepend the actual on-disk objects with a period.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove get_bucket_info and convert users.

This just lives in struct rgw_bucket now.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

radosgw-admin: fix 'bucket stats --uid='

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: include 'suspended' in user info output

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: clean up some cout usage

Errors to stderr.  Avoid cout for commands with structured output.

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: use formatter for 'bucket list'

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: 'bucket stats --uid'

List stats for all buckets on for the given user.

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: clean up 'bucket list' output

Signed-off-by: Sage Weil <sage@newdream.net>

rgw_admin: rip out format.empty() special cases

If we want a plain output, we should implement a PlainFormatter of some
sort.  For now, pretty json is good enough.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Conflicts:

src/rgw/rgw_admin.cc

rgw_admin: fix --pretty-format and --purge-data flag parsing

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Conflicts:

src/rgw/rgw_admin.cc

rgw_admin: default to json output

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

radosgw-admin: use formatter for 'bucket list'

Signed-off-by: Sage Weil <sage@newdream.net>

rdosgw_admin -> radosgw-admin

Be consistent with other binary tool naming.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: remove now-unused struct RGWPoolInfo

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove rgw_[retrieve|store]_pool_info and ".pool" pool

Convert these to rgw_[get|store]_bucket_info, add the owner field
to bucket info, and store both lookup formats in the ".bucket" pool.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: remove references to pool_id and replace with new bucket_id.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: don't encode the bucket name twice in RGWPoolInfo!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: put bucketid allocator object in root (.rgw) pool

This keeps the .rgw.control pool disposable.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: use same prefix for locator key as for oid

If the bucket has a prefix for oids, prepend the same prefix to the
locator key.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: list buckets using bucket objects

Don't assume the pools == buckets.  List the bucket objects in the .rgw
pool.

Fixes: #1560
Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'origin/master' into wip-pool-sharing

Conflicts:
src/osd/OSD.cc
src/rgw/rgw_rados.cc

Merge branch 'stable'

radosgw: fix content-length checks

Ignore content-length check if the REQUEST_URI has a trailing ?acl.

(The object_str.find() check was wrong because object_str has the ?whatever
stripped off, and because it returns std::string::npos on failure.)

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: tolerate race creating .rgw* metadata pools

There may be multiple people trying to create these pools.  That's just
fine.

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: fix end_block calculation for read_iterate

All end block calculations match now.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

test_rbd.py: Add large read and write tests

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: don't zero negative lengths

They are past the end of the buffer (and wrap around).

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: be paranoid about strings

This prevents bad conversions (i.e. from int to char *).

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: remove useless version object

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: be paranoid about strings

This prevents bad conversions (i.e. from int to char *).

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: explicitly cast strings to c_char_p for type clarity

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: pass 0 for the default order, not NULL

librbd expects the order pointer to be valid - if it points to 0, the
default order is used.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: make exceptions inherit from a base class

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: allow configuring rados through constructor options

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: make Image a context manager

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: make Rados and Ioctx objects context managers

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: make exceptions inherit from a base class

This way you can catch them all.
Also remove the unused NoData exceptions
- this was just useful for xattrs in librados.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Use cpp_strerror or strerror_r when appropriate.

The only remaining uses of strerror are in single-threaded programs
that don't depend on libcommon.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: add partial bindings for librbd

This is all the basic operations, but nothing with callbacks yet.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: raise specific error when write is too long

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: init counter to 0 so errors don't say '-1% done'

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ReplicatedPG, tmap: return -ENOENT if the key to remove does not exist

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: use pre-existing helper methods for tmap operations

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: add debugging for failing to remove a header object

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: improve error messages

One of these was accidentally not negated.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

mon: fix it again

Signed-off-by: Sage Weil <sage@newdream.net>

mon: fix dup pool creation race

Fixes: #1574
Signed-off-by: Sage Weil <sage@newdream.net>

rgw: get bucket markers from the same object at all times

We use an object in the control pool here, rather than an object
local to each pool. This gives us a single value which can be used
for tracking data at all times.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

init-ceph: more c* -> ceph-* cleanup

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix warning more

Argh

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
Signed-off-by: Sage Weil <sage@newdream.net>

radosgw_admin: 'bucket list' or 'buckets list'

The plural looks weird to me.

Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: include init-radosgw in tarball

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: shut down threadpool

Don't crash when run as a non-daemon without apache.. e.g. 'radosgw' with
no args.

Fixes: #1572
Signed-off-by: Sage Weil <sage@newdream.net>

rgw: pass POD c-string instead of string to formatter

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: fix warning more

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote branch 'origin/master' into wip-pool-sharing

osd: fix warning

Signed-off-by: Sage Weil <sage@newdream.net>

debian: include radosgw init script in package

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: init script

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'stable'

filejournal: improve debug error messge

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd/mon: make max maps per MOSDMap message tunable

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: limit size of osdmap messages

Limit size of MOSDMap messages sent from OSDs.  Send many smaller
messages instead of a single large one.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: drop map_cache_keep_from

The purpose here was to avoid trimming cached maps prior to what we have
on disk.  However, now that we have the map_bl cache, this isn't needed:
anything after that epoch will come out of that cache.

Also, it was broken anyway--the value was never read.  So clean it out!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: limit size of OSDMap cache

If we get way way behind on our maps, we may end up with a really large
OSDMap cache because we currently on trim old maps based on
oldest_last_clean, which may be way in the past.  Avoid eating up gobs of
RAM by putting a ceiling on the cache size.  It'll mean more disk IO in
those situations, but it also means that we'll only load up the old maps
that we actually need (not every single one).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: only require content-length to be set on non-acl puts

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: don't finish boot unless instance in map is really us

We were going BOOTING->ACTIVE as soon as we showed up in the map with the
same client_addr.  Also verify that we were up_from an epoch after when
we started or rebound, to avoid the case where we rebind to the same
ports for client_addr (but maybe not others) and get caught in a rebind
loop.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

init-ceph: don't umount btrfs by default

As an admin I find this very annoying, even though it isn't strictly
symmetrical with the mount behavior.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

- Fix atomic set.

rgw: user async exec for index completion op

librados: add async exec

rgw: tmp objects can be removed async

rgw: fix bucket stats

rgw: fix header encoding

rgw: fix compilation warning

rgw: fix compilation

cls_rgw: substract obj stats only if obj exists

rgw: integrate index categories

rgw: only require content-length to be set on non-acl puts

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Revert "rgw: tolerate missing content-length header on canned ACL puts"

This reverts commit 05b06e807ef20c7fe1bd7c7abbeecd656d2280c0.

Don't be stupid when an empty content-length can only be on ACL puts!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote branch 'gh/stable'

man: c* -> ceph-* rename: Regenerate manpages.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Fix reStructuredText formatting warnings.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile: c* -> ceph-* rename: missed cdebugpack in CLEANFILES.

make distcheck said:

  ERROR: files left in build directory after distclean:
  ./src/ceph-debugpack
  make[1]: *** [distcleancheck] Error 1

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile: Add missing header test/unit.h

Used to fail "make distcheck".

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>