Merge remote branch 'origin/master' into wip-pool-sharing

Conflicts:
src/osd/OSD.cc
src/rgw/rgw_rados.cc

Merge branch 'stable'

radosgw: fix content-length checks

Ignore content-length check if the REQUEST_URI has a trailing ?acl.

(The object_str.find() check was wrong because object_str has the ?whatever
stripped off, and because it returns std::string::npos on failure.)

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: tolerate race creating .rgw* metadata pools

There may be multiple people trying to create these pools.  That's just
fine.

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: fix end_block calculation for read_iterate

All end block calculations match now.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

test_rbd.py: Add large read and write tests

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: don't zero negative lengths

They are past the end of the buffer (and wrap around).

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: be paranoid about strings

This prevents bad conversions (i.e. from int to char *).

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: remove useless version object

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: be paranoid about strings

This prevents bad conversions (i.e. from int to char *).

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: explicitly cast strings to c_char_p for type clarity

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: pass 0 for the default order, not NULL

librbd expects the order pointer to be valid - if it points to 0, the
default order is used.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: make exceptions inherit from a base class

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: allow configuring rados through constructor options

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: make Image a context manager

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: make Rados and Ioctx objects context managers

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: make exceptions inherit from a base class

This way you can catch them all.
Also remove the unused NoData exceptions
- this was just useful for xattrs in librados.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Use cpp_strerror or strerror_r when appropriate.

The only remaining uses of strerror are in single-threaded programs
that don't depend on libcommon.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: add partial bindings for librbd

This is all the basic operations, but nothing with callbacks yet.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: raise specific error when write is too long

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: init counter to 0 so errors don't say '-1% done'

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ReplicatedPG, tmap: return -ENOENT if the key to remove does not exist

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: use pre-existing helper methods for tmap operations

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: add debugging for failing to remove a header object

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: improve error messages

One of these was accidentally not negated.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

mon: fix it again

Signed-off-by: Sage Weil <sage@newdream.net>

mon: fix dup pool creation race

Fixes: #1574
Signed-off-by: Sage Weil <sage@newdream.net>

rgw: get bucket markers from the same object at all times

We use an object in the control pool here, rather than an object
local to each pool. This gives us a single value which can be used
for tracking data at all times.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

init-ceph: more c* -> ceph-* cleanup

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix warning more

Argh

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
Signed-off-by: Sage Weil <sage@newdream.net>

radosgw_admin: 'bucket list' or 'buckets list'

The plural looks weird to me.

Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: include init-radosgw in tarball

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: shut down threadpool

Don't crash when run as a non-daemon without apache.. e.g. 'radosgw' with
no args.

Fixes: #1572
Signed-off-by: Sage Weil <sage@newdream.net>

rgw: pass POD c-string instead of string to formatter

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: fix warning more

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote branch 'origin/master' into wip-pool-sharing

osd: fix warning

Signed-off-by: Sage Weil <sage@newdream.net>

debian: include radosgw init script in package

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: init script

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'stable'

filejournal: improve debug error messge

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd/mon: make max maps per MOSDMap message tunable

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: limit size of osdmap messages

Limit size of MOSDMap messages sent from OSDs.  Send many smaller
messages instead of a single large one.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: drop map_cache_keep_from

The purpose here was to avoid trimming cached maps prior to what we have
on disk.  However, now that we have the map_bl cache, this isn't needed:
anything after that epoch will come out of that cache.

Also, it was broken anyway--the value was never read.  So clean it out!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: limit size of OSDMap cache

If we get way way behind on our maps, we may end up with a really large
OSDMap cache because we currently on trim old maps based on
oldest_last_clean, which may be way in the past.  Avoid eating up gobs of
RAM by putting a ceiling on the cache size.  It'll mean more disk IO in
those situations, but it also means that we'll only load up the old maps
that we actually need (not every single one).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: only require content-length to be set on non-acl puts

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: don't finish boot unless instance in map is really us

We were going BOOTING->ACTIVE as soon as we showed up in the map with the
same client_addr.  Also verify that we were up_from an epoch after when
we started or rebound, to avoid the case where we rebind to the same
ports for client_addr (but maybe not others) and get caught in a rebind
loop.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

init-ceph: don't umount btrfs by default

As an admin I find this very annoying, even though it isn't strictly
symmetrical with the mount behavior.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

- Fix atomic set.

rgw: user async exec for index completion op

librados: add async exec

rgw: tmp objects can be removed async

rgw: fix bucket stats

rgw: fix header encoding

rgw: fix compilation warning

rgw: fix compilation

cls_rgw: substract obj stats only if obj exists

rgw: integrate index categories

rgw: only require content-length to be set on non-acl puts

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Revert "rgw: tolerate missing content-length header on canned ACL puts"

This reverts commit 05b06e807ef20c7fe1bd7c7abbeecd656d2280c0.

Don't be stupid when an empty content-length can only be on ACL puts!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote branch 'gh/stable'

man: c* -> ceph-* rename: Regenerate manpages.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Fix reStructuredText formatting warnings.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile: c* -> ceph-* rename: missed cdebugpack in CLEANFILES.

make distcheck said:

  ERROR: files left in build directory after distclean:
  ./src/ceph-debugpack
  make[1]: *** [distcleancheck] Error 1

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile: Add missing header test/unit.h

Used to fail "make distcheck".

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

.gitignore: distcheck temporary directory.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile: include run-cli-tests-maybe-unset-ccache in dist tarball.

Apparently "make distcheck" has been broken for 3 months;
I find that hard to believe, but that's what it looks like.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ceph-run: c* -> ceph-* rename: Use ceph-run as stderr prefix.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ceph-debugpack: c* -> ceph-* rename: use "ceph-debugpack" as temp dir name too.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

udev: c* -> ceph-* rename: missed crbdnamer.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: c* -> ceph-* rename: missed csyn, crbdnamer.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rpm: c* -> ceph-* rename: missed crbdnamer, cdebugpack.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile: c* -> ceph-* rename: missed crun.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: v0.35 is out, remove rpm build workaround.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Add index into the sidebar.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Search page is always in sidebar, no need to have it on page.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Add a space after sudoers "NOPASSWD:", as that seems more common.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Add index entries, remove glossary as it's practically replaced by the index.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Leftovers of the c* -> ceph-* rename.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rgw: update index when doing put_obj_meta

this fixes multipart upload listing

rgw: tolerate missing content-length header on canned ACL puts

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cls_rgw: list entries return upper_bound of marker

rgw: fix a few regressions

mon: remember source client address in routed requests

when we resend_routed_requests, the source client address is lost.
This may cause problems. For example, if we resend an mds beacon
(boot) to a new monitor leader, the new mdsmap will contain a new
mds entry without IP address.

To reproduce this bug:
1. deploy a cluster with 3 mons.
2. let active mds send beacon to mon0; standby mds send becaon to mon2
3. gdb attach to mon2 to make it unresponsive and make standby mds laggy.
4. gdb attach to mon0 to make it unresponsive and make active mds laggy.
5. detach mon2, then the standby mds will become active.
6. ceph mds dump -o - shows the active mds address is :/0

Signed-off-by: Henry C Chang <henry_c_chang@tcloudcomputing.com>
Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix race between handle_osd_ping and handle_osd_map

If handle_osd_map is in progress and handle_osd_ping doesn't have the
map lock, we can't call osdmap->get_inst() in send_still_alive().  Keep
the entity_inst_t in the failure_pending map so that we don't need to.

osd/OSDMap.h: In function 'entity_inst_t OSDMap::get_inst(int)', in thread '0x7fda6a46b710'
osd/OSDMap.h: 477: FAILED assert(is_up(osd))
 ceph version 0.24.1 (commit:e06fb657842379259826f3d9215101fc14575fbd)
 1: (OSD::send_still_alive(int)+0x1b9) [0x4dd1b9]
 2: (OSD::handle_osd_ping(MOSDPing*)+0x716) [0x4f6446]
 3: (OSD::heartbeat_dispatch(Message*)+0x36) [0x4f6666]
 4: (SimpleMessenger::dispatch_entry()+0x882) [0x46f002]
 5: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x465fac]
 6: (()+0x6a3a) [0x7fda7835aa3a]
 7: (clone()+0x6d) [0x7fda76f7777d]

Reported-by: Henry C Chang <henry_c_chang@tcloudcomputing.com>
Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: drop useless all_sources thing

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rename source files c* -> ceph-*

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

misc: use strncpy to prevent buffer overflow

ceph-elaster: add parentheses around macros

test_store: parse args, don't link librados

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-pool-sharing' of ssh://github.com/NewDreamNetwork/ceph into wip-pool-sharing

rgw: fix clone in case state is NULL

rgw: send correct object id to class op

rgw: encode correct timestamp

rgw: fix rgw_admin arg parsing

rgw: generate tag if doesn't exist

crushtool: add unit test for adding items to map

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crushtool: fix argument parsing

Reported by: Henry C Chang <henry.cy.chang@gmail.com>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crushtool: crush_finalize on any map modification

This recalculates max_devices and the parent maps.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Revert "crush: refuse to add item >= max_devices"

This reverts commit 58f98e7afc364d28a385adc7ef0fee2291d70c17.

max_devices only affects the size of the parent arrays, which aren't allocated
until crush_finalize().  No problems here.