rgw: get bucket markers from the same object at all times

We use an object in the control pool here, rather than an object
local to each pool. This gives us a single value which can be used
for tracking data at all times.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: pass POD c-string instead of string to formatter

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: fix warning more

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote branch 'origin/master' into wip-pool-sharing

Makefile: include init-radosgw in tarball

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: shut down threadpool

Don't crash when run as a non-daemon without apache.. e.g. 'radosgw' with
no args.

Fixes: #1572
Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix warning

Signed-off-by: Sage Weil <sage@newdream.net>

debian: include radosgw init script in package

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: init script

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'stable'

filejournal: improve debug error messge

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd/mon: make max maps per MOSDMap message tunable

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: limit size of osdmap messages

Limit size of MOSDMap messages sent from OSDs.  Send many smaller
messages instead of a single large one.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: drop map_cache_keep_from

The purpose here was to avoid trimming cached maps prior to what we have
on disk.  However, now that we have the map_bl cache, this isn't needed:
anything after that epoch will come out of that cache.

Also, it was broken anyway--the value was never read.  So clean it out!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: limit size of OSDMap cache

If we get way way behind on our maps, we may end up with a really large
OSDMap cache because we currently on trim old maps based on
oldest_last_clean, which may be way in the past.  Avoid eating up gobs of
RAM by putting a ceiling on the cache size.  It'll mean more disk IO in
those situations, but it also means that we'll only load up the old maps
that we actually need (not every single one).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: only require content-length to be set on non-acl puts

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: don't finish boot unless instance in map is really us

We were going BOOTING->ACTIVE as soon as we showed up in the map with the
same client_addr.  Also verify that we were up_from an epoch after when
we started or rebound, to avoid the case where we rebind to the same
ports for client_addr (but maybe not others) and get caught in a rebind
loop.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

init-ceph: don't umount btrfs by default

As an admin I find this very annoying, even though it isn't strictly
symmetrical with the mount behavior.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

- Fix atomic set.

rgw: user async exec for index completion op

librados: add async exec

rgw: tmp objects can be removed async

rgw: fix bucket stats

rgw: fix header encoding

rgw: fix compilation warning

rgw: fix compilation

cls_rgw: substract obj stats only if obj exists

rgw: integrate index categories

rgw: only require content-length to be set on non-acl puts

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Revert "rgw: tolerate missing content-length header on canned ACL puts"

This reverts commit 05b06e807ef20c7fe1bd7c7abbeecd656d2280c0.

Don't be stupid when an empty content-length can only be on ACL puts!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote branch 'gh/stable'

man: c* -> ceph-* rename: Regenerate manpages.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Fix reStructuredText formatting warnings.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile: c* -> ceph-* rename: missed cdebugpack in CLEANFILES.

make distcheck said:

  ERROR: files left in build directory after distclean:
  ./src/ceph-debugpack
  make[1]: *** [distcleancheck] Error 1

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile: Add missing header test/unit.h

Used to fail "make distcheck".

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

.gitignore: distcheck temporary directory.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile: include run-cli-tests-maybe-unset-ccache in dist tarball.

Apparently "make distcheck" has been broken for 3 months;
I find that hard to believe, but that's what it looks like.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ceph-run: c* -> ceph-* rename: Use ceph-run as stderr prefix.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ceph-debugpack: c* -> ceph-* rename: use "ceph-debugpack" as temp dir name too.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

udev: c* -> ceph-* rename: missed crbdnamer.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: c* -> ceph-* rename: missed csyn, crbdnamer.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rpm: c* -> ceph-* rename: missed crbdnamer, cdebugpack.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile: c* -> ceph-* rename: missed crun.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: v0.35 is out, remove rpm build workaround.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Add index into the sidebar.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Search page is always in sidebar, no need to have it on page.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Add a space after sudoers "NOPASSWD:", as that seems more common.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Add index entries, remove glossary as it's practically replaced by the index.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Leftovers of the c* -> ceph-* rename.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rgw: update index when doing put_obj_meta

this fixes multipart upload listing

rgw: tolerate missing content-length header on canned ACL puts

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

cls_rgw: list entries return upper_bound of marker

rgw: fix a few regressions

mon: remember source client address in routed requests

when we resend_routed_requests, the source client address is lost.
This may cause problems. For example, if we resend an mds beacon
(boot) to a new monitor leader, the new mdsmap will contain a new
mds entry without IP address.

To reproduce this bug:
1. deploy a cluster with 3 mons.
2. let active mds send beacon to mon0; standby mds send becaon to mon2
3. gdb attach to mon2 to make it unresponsive and make standby mds laggy.
4. gdb attach to mon0 to make it unresponsive and make active mds laggy.
5. detach mon2, then the standby mds will become active.
6. ceph mds dump -o - shows the active mds address is :/0

Signed-off-by: Henry C Chang <henry_c_chang@tcloudcomputing.com>
Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix race between handle_osd_ping and handle_osd_map

If handle_osd_map is in progress and handle_osd_ping doesn't have the
map lock, we can't call osdmap->get_inst() in send_still_alive().  Keep
the entity_inst_t in the failure_pending map so that we don't need to.

osd/OSDMap.h: In function 'entity_inst_t OSDMap::get_inst(int)', in thread '0x7fda6a46b710'
osd/OSDMap.h: 477: FAILED assert(is_up(osd))
 ceph version 0.24.1 (commit:e06fb657842379259826f3d9215101fc14575fbd)
 1: (OSD::send_still_alive(int)+0x1b9) [0x4dd1b9]
 2: (OSD::handle_osd_ping(MOSDPing*)+0x716) [0x4f6446]
 3: (OSD::heartbeat_dispatch(Message*)+0x36) [0x4f6666]
 4: (SimpleMessenger::dispatch_entry()+0x882) [0x46f002]
 5: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x465fac]
 6: (()+0x6a3a) [0x7fda7835aa3a]
 7: (clone()+0x6d) [0x7fda76f7777d]

Reported-by: Henry C Chang <henry_c_chang@tcloudcomputing.com>
Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: drop useless all_sources thing

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rename source files c* -> ceph-*

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

misc: use strncpy to prevent buffer overflow

ceph-elaster: add parentheses around macros

test_store: parse args, don't link librados

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-pool-sharing' of ssh://github.com/NewDreamNetwork/ceph into wip-pool-sharing

rgw: fix clone in case state is NULL

rgw: send correct object id to class op

rgw: encode correct timestamp

rgw: fix rgw_admin arg parsing

rgw: generate tag if doesn't exist

crushtool: add unit test for adding items to map

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crushtool: fix argument parsing

Reported by: Henry C Chang <henry.cy.chang@gmail.com>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crushtool: crush_finalize on any map modification

This recalculates max_devices and the parent maps.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Revert "crush: refuse to add item >= max_devices"

This reverts commit 58f98e7afc364d28a385adc7ef0fee2291d70c17.

max_devices only affects the size of the parent arrays, which aren't allocated
until crush_finalize().  No problems here.

rgw: two phase commit for index updating

Just compiles, not tested, probably wouldn't work.

Merge branch 'stable'

Merge branch 'wip-names'

man: rebuild manpages

Signed-off-by: Sage Weil <sage@newdream.net>

doc: more c* -> ceph-* renames

Signed-off-by: Sage Weil <sage@newdream.net>

rbd: set default pool when none is specified

096e3b6353e5035362cffdcbd2e4a4f5572aa2ba broke this by only using
set_pool_image_name for commands that accept the snapshot
parameter. This whole undocumented format parsing should be reworked
or removed at some point.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

libceph -> libcephfs

Signed-off-by: Sage Weil <sage@newdream.net>

c* -> ceph-*

Hopefully I didn't miss too much...

Signed-off-by: Sage Weil <sage@newdream.net>

mount.ceph: use ":/" instead of just ":" as ip/path delimiter

This is tolerant of colons in the mount subdir.

Signed-off-by: Sage Weil <sage@newdream.net>

addr_parsing: null terminate safe_cat result

Signed-off-by: Sage Weil <sage@newdream.net>

mount.ceph: fix use-after-free

Signed-off-by: Sage Weil <sage@newdream.net>

test/rados-api/test.cc: Add ability to set client id

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

test/test_stress_watch.cc: added simple watch stress test

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: set log.backlog to false on corrupt log

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: set log.backlog to false on corrupt log

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: fix rgw_bucket_dir_entry encoding

rgw: list objects (almost completely) works

rgw: obj listing doesn't stat object

rgw: store more info for each indexed object

vstart: debug filestore = 20

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix PG::copy_after vs backlog

If you call copy_after(..., 0) on a log with a backlog, you get all the
backlog entries, but no backlog flag.  That's invalid.  You either need
the _complete_ backlog + the flag, or no backlog entries; getting only
some of them is useless information.

Make copy_after stop when it hits the tail.  Callers who need the backlog
are already checking for that and copying the whole log as appropriate.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix traverse snap vs forward check

This should compare against the forward attempt, not retry attempt.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rbd: fix progress redraw

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

cls_rgw: store more info for each entry, stats categories

rgw: account object removal

rgw: account clone_range dest object

doc: Content for Getting Started with cephfs and rbd.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rgw: class init_index

doc: Content for Getting Started with RADOS.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Structure for Getting Started.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>