rados.py: allow configuring rados through constructor options

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: make Image a context manager

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: make Rados and Ioctx objects context managers

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: make exceptions inherit from a base class

This way you can catch them all.
Also remove the unused NoData exceptions
- this was just useful for xattrs in librados.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

Use cpp_strerror or strerror_r when appropriate.

The only remaining uses of strerror are in single-threaded programs
that don't depend on libcommon.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: add partial bindings for librbd

This is all the basic operations, but nothing with callbacks yet.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rados.py: raise specific error when write is too long

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: init counter to 0 so errors don't say '-1% done'

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ReplicatedPG, tmap: return -ENOENT if the key to remove does not exist

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: use pre-existing helper methods for tmap operations

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: add debugging for failing to remove a header object

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: improve error messages

One of these was accidentally not negated.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

mon: fix it again

Signed-off-by: Sage Weil <sage@newdream.net>

mon: fix dup pool creation race

Fixes: #1574
Signed-off-by: Sage Weil <sage@newdream.net>

init-ceph: more c* -> ceph-* cleanup

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix warning more

Argh

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
Signed-off-by: Sage Weil <sage@newdream.net>

radosgw_admin: 'bucket list' or 'buckets list'

The plural looks weird to me.

Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: include init-radosgw in tarball

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: shut down threadpool

Don't crash when run as a non-daemon without apache.. e.g. 'radosgw' with
no args.

Fixes: #1572
Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix warning

Signed-off-by: Sage Weil <sage@newdream.net>

debian: include radosgw init script in package

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw: init script

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'stable'

filejournal: improve debug error messge

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd/mon: make max maps per MOSDMap message tunable

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: limit size of osdmap messages

Limit size of MOSDMap messages sent from OSDs.  Send many smaller
messages instead of a single large one.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: drop map_cache_keep_from

The purpose here was to avoid trimming cached maps prior to what we have
on disk.  However, now that we have the map_bl cache, this isn't needed:
anything after that epoch will come out of that cache.

Also, it was broken anyway--the value was never read.  So clean it out!

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: limit size of OSDMap cache

If we get way way behind on our maps, we may end up with a really large
OSDMap cache because we currently on trim old maps based on
oldest_last_clean, which may be way in the past.  Avoid eating up gobs of
RAM by putting a ceiling on the cache size.  It'll mean more disk IO in
those situations, but it also means that we'll only load up the old maps
that we actually need (not every single one).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: only require content-length to be set on non-acl puts

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: don't finish boot unless instance in map is really us

We were going BOOTING->ACTIVE as soon as we showed up in the map with the
same client_addr.  Also verify that we were up_from an epoch after when
we started or rebound, to avoid the case where we rebind to the same
ports for client_addr (but maybe not others) and get caught in a rebind
loop.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

init-ceph: don't umount btrfs by default

As an admin I find this very annoying, even though it isn't strictly
symmetrical with the mount behavior.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

- Fix atomic set.

rgw: only require content-length to be set on non-acl puts

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Revert "rgw: tolerate missing content-length header on canned ACL puts"

This reverts commit 05b06e807ef20c7fe1bd7c7abbeecd656d2280c0.

Don't be stupid when an empty content-length can only be on ACL puts!

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

Merge remote branch 'gh/stable'

man: c* -> ceph-* rename: Regenerate manpages.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Fix reStructuredText formatting warnings.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile: c* -> ceph-* rename: missed cdebugpack in CLEANFILES.

make distcheck said:

  ERROR: files left in build directory after distclean:
  ./src/ceph-debugpack
  make[1]: *** [distcleancheck] Error 1

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile: Add missing header test/unit.h

Used to fail "make distcheck".

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

.gitignore: distcheck temporary directory.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile: include run-cli-tests-maybe-unset-ccache in dist tarball.

Apparently "make distcheck" has been broken for 3 months;
I find that hard to believe, but that's what it looks like.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ceph-run: c* -> ceph-* rename: Use ceph-run as stderr prefix.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

ceph-debugpack: c* -> ceph-* rename: use "ceph-debugpack" as temp dir name too.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

udev: c* -> ceph-* rename: missed crbdnamer.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

debian: c* -> ceph-* rename: missed csyn, crbdnamer.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rpm: c* -> ceph-* rename: missed crbdnamer, cdebugpack.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Makefile: c* -> ceph-* rename: missed crun.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: v0.35 is out, remove rpm build workaround.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Add index into the sidebar.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Search page is always in sidebar, no need to have it on page.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Add a space after sudoers "NOPASSWD:", as that seems more common.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Add index entries, remove glossary as it's practically replaced by the index.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Leftovers of the c* -> ceph-* rename.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

rgw: tolerate missing content-length header on canned ACL puts

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

mon: remember source client address in routed requests

when we resend_routed_requests, the source client address is lost.
This may cause problems. For example, if we resend an mds beacon
(boot) to a new monitor leader, the new mdsmap will contain a new
mds entry without IP address.

To reproduce this bug:
1. deploy a cluster with 3 mons.
2. let active mds send beacon to mon0; standby mds send becaon to mon2
3. gdb attach to mon2 to make it unresponsive and make standby mds laggy.
4. gdb attach to mon0 to make it unresponsive and make active mds laggy.
5. detach mon2, then the standby mds will become active.
6. ceph mds dump -o - shows the active mds address is :/0

Signed-off-by: Henry C Chang <henry_c_chang@tcloudcomputing.com>
Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix race between handle_osd_ping and handle_osd_map

If handle_osd_map is in progress and handle_osd_ping doesn't have the
map lock, we can't call osdmap->get_inst() in send_still_alive().  Keep
the entity_inst_t in the failure_pending map so that we don't need to.

osd/OSDMap.h: In function 'entity_inst_t OSDMap::get_inst(int)', in thread '0x7fda6a46b710'
osd/OSDMap.h: 477: FAILED assert(is_up(osd))
 ceph version 0.24.1 (commit:e06fb657842379259826f3d9215101fc14575fbd)
 1: (OSD::send_still_alive(int)+0x1b9) [0x4dd1b9]
 2: (OSD::handle_osd_ping(MOSDPing*)+0x716) [0x4f6446]
 3: (OSD::heartbeat_dispatch(Message*)+0x36) [0x4f6666]
 4: (SimpleMessenger::dispatch_entry()+0x882) [0x46f002]
 5: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x465fac]
 6: (()+0x6a3a) [0x7fda7835aa3a]
 7: (clone()+0x6d) [0x7fda76f7777d]

Reported-by: Henry C Chang <henry_c_chang@tcloudcomputing.com>
Signed-off-by: Sage Weil <sage@newdream.net>

Makefile: drop useless all_sources thing

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rename source files c* -> ceph-*

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

misc: use strncpy to prevent buffer overflow

ceph-elaster: add parentheses around macros

test_store: parse args, don't link librados

Signed-off-by: Sage Weil <sage@newdream.net>

crushtool: add unit test for adding items to map

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crushtool: fix argument parsing

Reported by: Henry C Chang <henry.cy.chang@gmail.com>
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

crushtool: crush_finalize on any map modification

This recalculates max_devices and the parent maps.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Revert "crush: refuse to add item >= max_devices"

This reverts commit 58f98e7afc364d28a385adc7ef0fee2291d70c17.

max_devices only affects the size of the parent arrays, which aren't allocated
until crush_finalize().  No problems here.

Merge branch 'stable'

Merge branch 'wip-names'

man: rebuild manpages

Signed-off-by: Sage Weil <sage@newdream.net>

doc: more c* -> ceph-* renames

Signed-off-by: Sage Weil <sage@newdream.net>

rbd: set default pool when none is specified

096e3b6353e5035362cffdcbd2e4a4f5572aa2ba broke this by only using
set_pool_image_name for commands that accept the snapshot
parameter. This whole undocumented format parsing should be reworked
or removed at some point.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

libceph -> libcephfs

Signed-off-by: Sage Weil <sage@newdream.net>

c* -> ceph-*

Hopefully I didn't miss too much...

Signed-off-by: Sage Weil <sage@newdream.net>

mount.ceph: use ":/" instead of just ":" as ip/path delimiter

This is tolerant of colons in the mount subdir.

Signed-off-by: Sage Weil <sage@newdream.net>

addr_parsing: null terminate safe_cat result

Signed-off-by: Sage Weil <sage@newdream.net>

mount.ceph: fix use-after-free

Signed-off-by: Sage Weil <sage@newdream.net>

test/rados-api/test.cc: Add ability to set client id

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

test/test_stress_watch.cc: added simple watch stress test

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: set log.backlog to false on corrupt log

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: set log.backlog to false on corrupt log

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

vstart: debug filestore = 20

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: fix PG::copy_after vs backlog

If you call copy_after(..., 0) on a log with a backlog, you get all the
backlog entries, but no backlog flag.  That's invalid.  You either need
the _complete_ backlog + the flag, or no backlog entries; getting only
some of them is useless information.

Make copy_after stop when it hits the tail.  Callers who need the backlog
are already checking for that and copying the whole log as appropriate.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mds: fix traverse snap vs forward check

This should compare against the forward attempt, not retry attempt.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rbd: fix progress redraw

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

doc: Content for Getting Started with cephfs and rbd.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Content for Getting Started with RADOS.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Structure for Getting Started.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Separate key management from mounting.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Move config reference to top-level.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Split data placement into concrete operations and recommendations.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Reorganize ops/manage/

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Move recommendations under /rec.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Move ops/grow under ops/manage.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Split manage into multiple pages.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Split install into multiple pages.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

doc: Clean reStructuredText syntax.

Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>

Merge branch 'stable'

client: tear down dir when setting I_COMPLETE on empty

If the dir is supposed to be empty and we are setting I_COMPLETE, empty
it out and close it.  This ensures we don't return bad results on a
subsequent readdir().

_Maybe_ related to #1509.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rados: accept '-b' as an argument.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

v0.35

Makefile: include config_opts.h

Signed-off-by: Sage Weil <sage@newdream.net>