omap: add sync method to ObjectMap

Signed-off-by: Samuel Just <rexludorum@gmail.com>

filestore: remove old post-idempotent transaction trigger_commit

The old strategy was to initiate a commit after any non-idempotent
transaction.  This only worked if the transaction was idempotent with
respect to itself, or could be replayed partially without problems,
and in reality that isn't the case.  For example:

 - clone A -> B
 - write to A
 - <sync>

If we crash before the sync, and replay the clone A->B, we corrupt B with
the new A data.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: guard collection_remove replay

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: guard replay of collection_add

- set guard on apply
- check guard on replay

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: guard replay of basic collection ops

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: guard collection_rename replay

- check guard on replay
- set guard on apply

Signed-off-by: Sage Weil <sage@newdream.net>

filestore: fix collection_rename error code

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: guard clone replay

- set guard xattr on clone, clone_range
- check before applying/replaying

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: implement _set_replay_guard, _check_replay_guard

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: maintain SequencerPosition during _do_transaction

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: fgetxattr helpers/wrappers

Also, do the getxattr using fgetxattr, to avoid duplicating code.  This is
slightly slower probably because we open a file handle, but if we care we
should really clean up the code to use lfn_open instead of lfn_find and
avoid the repeated path traversal too.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

add SequencerPosition type

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: pass trans_num into _do_transaction

This gives us the <op_seq, trans_num, op_num> triple to identify every
constituent operation.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

filestore: use proper type for readdir_r tmp

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'master' of github.com:ceph/ceph

RadosModel: fix omap_clear case in RemoveAttrsOp

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'wip-rgw-encode'

Conflicts:
src/rgw/rgw_cls_api.h

Reviewed-by: Sage Weil <sage@newdream.net>

rgw: fix rgw_cls_list_ret ctor

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ceph-object-corpus: added 0.43

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

test/encoding/import.sh: fix target directory

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

Makefile.am: update link dependencies for some unit tests

rgw: various encoding related fixes

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

osd: fix watch_lock vs map_lock ordering

watch_lock is inside map_lock (and pg->lock), which means we need to
drop it to take pg->lock here.  That means verifying in
handle_watch_timeout that we haven't raced with another thread canceling
the timeout event, which would be indicated by

 - the entity not appearing in unconnected_watchers
 - the entity having a different (presumably newer) expire time

Fixes: #2103
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

osd: update_heartbeat_peers as needed

Before, we were being very careful about updating the heartbeat peers if
new PGs were created or when certain types of messages were received.
However, the PG can change it's peers in lots of cases (e.g., when
recovery completes), but the OSD doesn't re-aggregate.

Instead, set a flag when each PG updates it's set, and check that flag in
the OSD code periodically or in likely places.  A call in tick() acts as
a catch-all.

The num_created counts can probably be cleaned out now...

Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: implement dump() for encoders

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: add stubs for dencoder test

still need to add some content to the dump methods

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Merge remote branch 'origin/wip-2139'

Conflicts:
src/cls_rgw.cc
src/rgw/rgw_rados.cc
src/rgw/rgw_rados.h

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

Merge branch 'master' of ssh://github.com/ceph/ceph

ceph: document the way files are laid out

This adds a document that I wrote about how Ceph client file data
is striped across Ceph objects to the repository.  It's a text
document.  Someone with better document preparation skills than I
should use the content below as a basis for something prettier if
that's appropriate.

[Made a few edits... -sage]

Signed-off-by: Alex Elder <elder@dreamhost.com>
Signed-off-by: Sage Weil <sage@newdream.net>

librados: fix unit test for omap_get_vals_by_key rename

Signed-off-by: Sage Weil <sage@newdream.net>

osd: format time nicely in ops_in_flight output

Signed-off-by: Sage Weil <sage@newdream.net>

librados: fix map -> std::map in header, string -> std::string

Signed-off-by: Sage Weil <sage@newdream.net>

doc: example of diagnosing radosgw hang

Signed-off-by: Sage Weil <sage@newdream.net>

doc: explain how unfound objects happen

Signed-off-by: Sage Weil <sage@newdream.net>

doc: make osd failure example include >3 osds

More realistic.

Signed-off-by: Sage Weil <sage@newdream.net>

testrados: fix omap_get_vals_by_keys call

Signed-off-by: Sage Weil <sage@newdream.net>

osd: add zero_to field to PG::OndiskLog; track zeroed region of pg log

Track which region of the log has been zeroed on disk.  This may be
different from tail if 'osd preserved trimmed log = false' in the config.

Only zero the portion of the log we need to.  This avoids rezeroing regions
or missing bits when 'osd preserved trimmed log' was off and is then turned
on.

Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

filestore: use FL_ALLOC_PUNCH_HOLE to zero, when available

First try the FL_ALLOC_PUNCH_HOLE fallocate() flag.  If we get EOPNOTSUPP,
fall back to writing zeros.

Check for fallocate(2) with configure.  Also, avoid this if we are not
Linux, since I'm not sure about the hard-coded FL_ALLOC_PUNCH_HOLE being
correct on other platforms.

Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

osd: fix op_wq vs pg->lock ordering

map_lock
 -> pg->lock
   -> op_wq

Fixes: #2153
Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'master' of ssh://skinny/home/yehudasa/ceph

Merge branch 'wip-rgw-new-atomic'

rgw: append the currect bucket marker when removing bucket

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Merge branch 'master' of ssh://skinny/home/yehudasa/ceph

Merge branch 'wip-rgw-omap'

cls_rgw: fix rgw_bucket_init_index

was failing to error in case header already existed

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: remove extra unused params from omap_get()

and also rename it to omap_get_all()

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: add cls_cxx_map_clear

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

leveldb: drop compaction unit test

Signed-off-by: Samuel Just <rexludorum@gmail.com>

ReplicatedPG,librados: add filter_prefix to omap_get_vals

Signed-off-by: Samuel Just <rexludorum@gmail.com>

rgw: use prefix filter for bucket listing

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

objclass, cls_rgw: add prefix to omap_get_vals()

librados: add higher level call for omap_get_keys() with prefix

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Merge remote-tracking branch 'origin/wip_prefix' into wip-rgw-omap

rgw: switch to new encoders/decoders

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

enconding: add legacy decoder for 32 bit versions

In these cases if the version is smaller than the compat ver
we skip 3 bytes.

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rbd: pass all mon addrs when mapping devices

Previously this repeated the address of the first monitor.

Fixes: #2152
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>
Reviewed-by: Sage Weil <sage@newdream.net>

msgr: remove declaration of undefined SimpleMessenger::write_pid_file

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
Reviewed-by: Sage Weil <sage.weil@dreamhost.com>

msgr: remove SimpleMessenger::get_ms_addr() in favor of Messenger::get_myaddr

And fix the comments on set_ip.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
Reviewed-by: Sage Weil <sage.weil@dreamhost.com>

objectstore: fix collection_move() encoding

This was broken in the original f43c3d958fe5c32ae647ffa715390ada51ae2650.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ReplicatedPG,librados: add filter_prefix to omap_get_vals

Signed-off-by: Samuel Just <rexludorum@gmail.com>

rgw: some minor cleanups

following a review

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

objclass: fix cls_cxx_map_write_header

Claiming the buffer instead of encoding it.

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

cls_rgw: fix debug message

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

Merge remote-tracking branch 'gh/wip-doc'

Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com>

osd: make degraded pgs count missing replicas as degraded objects

If a PG is smaller than it should be, make sure the missing replicas are
included in the degraded object count.  This makes the overall degraded
percentage consistently meaningful even for PGs that aren't mid-recovery
of mid-backfill.

Fixes: #2137
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>

mon: fix full osd detail

And use a helper to avoid dup code.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: assign severity to each health summary/detail item

These can be included in the detail dump in the future.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

doc: fix misc typos, bad phrasing

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objclass, cls_rgw: update to use omap

Signed-off-by: Yehuda Sadeh <yehuda.sadeh@dreamhost.com>

doc: 2 words about radosgw failures

- restarting the daemon.
- using the admin socket

Signed-off-by: Sage Weil <sage@newdream.net>

doc: talk about mon failures a bit

Signed-off-by: Sage Weil <sage@newdream.net>

doc: fix link

Signed-off-by: Sage Weil <sage@newdream.net>

doc: slow osd requests

Signed-off-by: Sage Weil <sage@newdream.net>

doc: diagnose full osd cluster

Signed-off-by: Sage Weil <sage@newdream.net>

mon: list nearfull/full osd detail

Signed-off-by: Sage Weil <sage@newdream.net>

doc: describe 'stuck' states we check for

Signed-off-by: Sage Weil <sage@newdream.net>

doc: document some osd failure recovery scenarios

- simple osd failure
- ceph health [detail]
- peering failure ('down') state
- unfound objects

Signed-off-by: Sage Weil <sage@newdream.net>

osd: list might_have_unfound locations in query result

Signed-off-by: Sage Weil <sage@newdream.net>

mon: include unfound count in health detail

Signed-off-by: Sage Weil <sage@newdream.net>

mon: refactor health, include optional detail

'ceph health' to get the usual summary, 'ceph health detail' to
additionally get a comprehensive list of problems found.

Eventually we can format this as yaml, json, whatever, too.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'wip-collmove'

rgw: switch to omap api

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

leveldb: remove flawed unit test for now

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

librados: rename omap_get_vals_by_key to omap_get_vals_by_keys

merge fail

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

librados: add high level omap calls

also rename get_vals_by_key to get_vals_by_keys

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: fix warning

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

rgw: read bucket through tmap_get

Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>

Merge branch 'wip_omap'

Reviewed-by: Sage Weil <sage.weil@dreamhost.com>

test_rados_api_aio: add omap

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd: testing for tmap auto upgrade

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: transparently upgrade TMAP

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

RadosModel: Add omap operations to RadosModel

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: Add omap ops to ReplicatedPG

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

librados: Added omap operations to librados

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osdc: Add omap operation stubs to Objecter::ObjectOperation

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: add omap_header to recovery

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

librados: add tmap_put to ObjectWriteOperation

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Merge branch 'wip-1796'

Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>

mds: respawn when blacklisted

If we are blacklisted by the OSD cluster, it's because we were too slow
and were replaced by another ceph-mds.  Respawn and re-register as a
standby.

If we get some other write error, shut down.

Fixes: #1796
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

journaler: add generic write error handler

Specify a generic callback for any write error the journaler encounters.
This is more helpful than passing up write errors to specific callers
because

 - there are several of them
 - journaler initiates writes on its own (like the head)

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>