do_autogen: control optimization level

-O 2 -> -O2

Signed-off-by: Sage Weil <sage@newdream.net>

log: move create_entry() into Log interface

This will let us be smarter than putting it on the heap.

Signed-off-by: Sage Weil <sage@newdream.net>

bench_log: simple util to time how long it takes to log stuff

Signed-off-by: Sage Weil <sage@newdream.net>

test log performance with PreallocatedStreambuf

- faster than ostringstream in optimistic case
- same as ostreamstream + std::string assignment in worst case (use
  overflow string)

Signed-off-by: Sage Weil <sage@newdream.net>

common: add PrebufferedStreambuf

Simple streambuf that uses a preallocated buffer, and then spills over
into a std::string if necessary.

Signed-off-by: Sage Weil <sage@newdream.net>

assert: dump recent log entries on failed assertions

Signed-off-by: Sage Weil <sage@newdream.net>

log: flush on_exit

Signed-off-by: Sage Weil <sage@newdream.net>

log: new logging infrastructure

- explicitly defined subsystems, and ceph_subsys_FOO enums to go with them
- modular log system with Entry object
- separate gather level and log level
- drop lots of DoutStreambuf hackery

Signed-off-by: Sage Weil <sage@newdream.net>

config: use our assert

Signed-off-by: Sage Weil <sage@newdream.net>

msg: assert pipe->msgr == msgr

Fixes: #2216
Signed-off-by: Sage Weil <sage@newdream.net>

doc: include crush in toctree

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

doc: few notes on manipulating the crush map

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

doc/dev/peering.rst: fix typo

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

vstart: enable omap for xattrs

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge remote branch 'gh/wip-doc-peering'

rgw: don't #include fcgi from rgw_common.h

ceph-dencoder #includes rgw_common.h, and needs to build even when
--without-radosgw is specified and libfcgi isn't installed.

Reported-by: vandemar in #ceph
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rados_bench: generate_object_name now takes a buffer length

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

doc: dev/peering.rst edits from Greg

Signed-off-by: Sage Weil <sage@newdream.net>

doc: update dev/peering document

- fix discussion of last epoch started
- define terms for current and past intervals
- describe role of pg info
- remove mention of the backlog
- fix discussion of up_thru
- etc.

Signed-off-by: Sage Weil <sage@newdream.net>

msgr: fix tcp.cc linkage

Signed-off-by: Sage Weil <sage@newdream.net>

cephtool: don't prefix log items

This just makes it hard to read them.

Signed-off-by: Sage Weil <sage@newdream.net>

doc: update list of debian dists

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'stable'

Objecter: resend linger_ops on any change

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

TestRados: Add watch

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: keep pool placement info also in cacheable location

Mirror the pools placement info, so that we can cache it.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: remove unused definition

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Revert "Objecter: add op->resend_on_any_change"

This reverts commit c53194d75390dd6d5aa4a9a33f741cbd106e3338.

recalc_linger_op_target is used for linger_ops

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: return -EBUSY on delete for objects with watchers

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Objecter: add op->resend_on_any_change

lingers must be resent even if the primary does not change.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: osd_max_notify_timeout -> osd_default_notify_timeout

This setting should not override user specified timeout.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: replace bucket_id generation

bucket_id is now string: <global instance id>.<num> where
num is increasing monotonically within the current rgw
instance.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

v0.44

rgw: process default alt args before processing conf file

this fixes #2189

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: process default alt args before processing conf file

this fixes #2189

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: incrase socket backlog

20 is too small

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: fix internal cache api

This fixes issue #2190

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

OSD: do not hold obc lock in disconnect_session_watches

ObjectContext::lock is used only for implementing read_lock and
write_lock.  PG::lock is used to protect the ObjectContexts
belonging to a pg.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

ReplicatedPG: there should be no object_contexts during on_activate

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'next'

osd: fix object_info.size mismatch file due to truncate_seq on new object

If the first write that creates an object includes a truncate_seq and
truncate_size, we were taking the truncte patch and doing a truncate op
in our transaction prior to the write, and then setting the object_info
size appropriately.  However, if the object doesn't exist, the truncate
op fails even though the oi.size gets set.

Later, this turns up as a scrub error (see #2080).

Fix this by skipping the truncate if it is a new object.  Instead, we
should just initialize our truncate_{seq,size} metadata so that we're all
up to date for any later writes.

Alternatively, we could touch the object and then truncate it (up) to the
large size, but this is sort of a waste; data beyond a short object eof is
defined to be zeros, so all we would accomplish is making recovery work
harder by copying zeros around.

Fixes: #2080
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

.gitignore: xattr_bench

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

heartbeatmap: use utimes(2) instead of futimens(2)

For poor users with ancient glibc.  We don't much care about rename races
here anyway.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: remove special handline for head recovery from clone

This breaks because:

 - we don't have the head or current snapset
 - get_object_context() creates a new snapset, which is wrong

We probably can only do this if we are certain we can construct/modify
the old snapset and end up with the correct one.

Signed-off-by: Sage Weil <sage@newdream.net>

config: fix recursive locking of md_config_t::lock

Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

osd: explicitly create new object,snap contexts on push

We specifically want to use this during recovery to avoid loading the obc
or ssc for a previous version of the object and populating the watchers.
We know we won't have any existing obc here because it is missing (old or
dne).

For the snapset context, we provide it explicitly when we recover the head
or snapset object (which we always do first).  For clones, we re-use the
existing get_snapset_context(), which will either have the ssc open or
can load it from the head/snapset object.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: re-use create_object_context() in get_object_context()

Signed-off-by: Sage Weil <sage@newdream.net>

osd: ReplicatedPG::create_object_context()

New helper that creates a new object context.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: create_snapset_context()

Signed-off-by: Sage Weil <sage@newdream.net>

osd: ensure we don't clobber other *contexts when registering new ones

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip_omap_xattrs'

ReplicatedPG,FileStore: clone should copy xattrs as well

_make_clone (called from make_writeable) and _rollback_to included
attr reads from head or a clone.  In that case, an ondisk read
lock would be necessary.  Now, clone also handles xattrs, so the
attr read should not be necessary.

Signed-off-by: Samuel Just <sam.just@dreamhost.com>

RadosModel: test xattrs with omap

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

FileStore: add support for omap xattrs

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'wip-msgr4'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

ObjectMap: use Index object for locking rather than path object

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

tests/: Added xattr bench

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

config_opts.h: opts for omap_xattrs

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

test_object_map: update unit test for xattr

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

DBObjectMap: implement xattr interface

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ObjectMap: add interface for storing xattrs

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

DBObjectMap: add support for storing xattrs

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

configure: fix warnings

Finally!

Signed-off-by: Sage Weil <sage@newdream.net>

ReplicatedPG: populate_object_context during handle_pull_response

A cached objectcontext should always have its watchers populated.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

leveldb: .gitignore TAGS

Signed-off-by: Sage Weil <sage@newdream.net>

leveldb: un-revert

Accidentally reverted by c2af646b38995ba005140e748a21baba4263e53f.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: maybe clear DEGRADED on recovery completion

We set degraded if we don't have enough "active" replicas, which excludes
the backfill target.  We need to recheck that when we finish recovery and
the backfill target is now complete.

Fixes: #2160
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: remove extra layer of RGWAccess

Not needed, now that we got rid of RGWFS

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

msgr: switch all users over to abstract interface

This will let us transparently swap implementations out.

Signed-off-by: Sage Weil <sage@newdream.net>

msgr: introduce static Messenger::create() function

Create a new messenger, with whatever implementation is appropriate.

Signed-off-by: Sage Weil <sage@newdream.net>

msgr: promote more methods to abstract Messenger interface

This will be everything that people actually use.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: put_obj() uses bufferlist instead of extra alloc/copy

makes it cleaner.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: remove fs backend

was broken anyway

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

introduce CEPH_FEATURE_OMAP

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

leveldb: new .gitignore entry

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'next'

osd: rev cluster internal protocol

This covers:

- the push/pull changes in 0.43 (which we forgot to protect against; see
  #2132)
- the new omap stuff for 0.44

Maybe we could make this finer grained so that ceph-osd would fail only
when mismatched versions are talking _and_ there is actual omap data in
play, but it's not worth the effort at this point.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-fuse: make big_writes optional via 'fuse big writes'

Fixes: #2159
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: remove some more globals from librgw

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

ceph-dencoder: don't use rgw types if configured without rgw

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: fix identation

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Merge branch 'master' of github.com:ceph/ceph

Added documentation for building the ceph documentation.

rgw: get rid of references to g_ceph_context where required

trickling down ceph context.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

doc: update project status/stability blurb

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Mark Kampe <mark.kampe@dreamhost.com>

global: drop yellow warning on startup

Fixes: #2143
Signed-off-by: Sage Weil <sage@newdream.net>

qa: kclient/file_layout.sh: ...

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'librados-cleanup'

Reviewed-by: Sage Weil <sage.weil@dreamhost.com>

osd_types: use uint64_t for ObjectExtent offsets and lengths

This is just client in-memory state, and allows us to address objects >4GiB,
to match the existing librados/Objecter interfaces.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

OSDMap: make get_pools() const

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

ObjectCacher: remove unused and crufty atomic sync operations

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados: move methods that require an IoCtx to IoCtxImpl

RadosClient still does a few different things, but at least it
no longer does all the work of an IoCtx.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados: split into separate files and remove unnecessary headers

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: add more meaningful tests instances of encoded objects

this completes #2140

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

build-doc: use alternate virtualenv dir, if specified

The docs gitbuilder will use this to avoid rebuilding the virtualenv on
every build.

Signed-off-by: Sage Weil <sage@newdream.net>

test_idempotent: fix global_init call

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

qa: kclient/file_layout.sh poking

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: modify bucket instance for encoding test

This makes 'make check' happy, otherwise we need to create
a bucket name that starts with a period. This version is better.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

test: add test_filestore_idempotent2

Signed-off-by: Samuel Just <rexludorum@gmail.com>

FileStore: ignore ERANGE and ENOENT on replay

The source object may either not exist or be the wrong size
during replay if the destination object was deleted in a future
already-applied operation.  This should not impact correctness
of the replay.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>