PG: set_last_peering_reset in Reset constructor

If an osd in the prior set comes up, we can restart peering without a
new peering interval starting.  However, we still want to ignore
anything we previously requested from replicas.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

osd: kill unused on_osd_failure() hook

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

RadosModel.h: use default conf location

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

Revert "PG: call set_last_peering_reset in Started contructor"

Unfortunately, the Started constructor doesn't occur until map
activation.  We need to reset last_peering_reset exactly when the acting
set changes.

This reverts commit 6d123067ce1ba99522281d5c72623bd5ba3e0fc8.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

hadoop: Return NULL when the path does not exist.

Although unspecified in the declaration header, other file
systems return a single result when the path is a file.

This fixes tracker issue #1661

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

osdmap: fix g_ceph_context reference

Use cct.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

ReplicatedPG: check for peering restart before share_pg_info

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

mkcephfs: build initial osdmap from information in ceph.conf

Signed-off-by: Sage Weil <sage@newdream.net>

osdmaptool: build initial map from ceph.conf

This builds the intial osd and crush maps from what is in the ceph.conf,
taking advantage of host or rack tags that are present there.

If there are >1 hosts, separate replica across hosts.  If there are >2
racks, separate across racks.  Semi arbitrary, but should capture most
use cases.

Signed-off-by: Sage Weil <sage@newdream.net>

crush: make insert_item take float for weight

Signed-off-by: Sage Weil <sage@newdream.net>

ReplicatedPG: Clean up old snap links when recovering a clone

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

PG: Create new snap directories independently on replica

Previously, we shipped over the collection creation as part
of the transaction.  However, the snap directory on the
replica might or might not exist already due to recovery
progress.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: canonical resource should use unencoded url

Merge pull request #4 from vzctl/master

fix error: 'snprintf' was not declared in this scope

rgw: cleanup, remove unused user_id

Some access methods required user_id param, but that was never really used. At
this point we should just remove them.

mkcephfs: skip non-btrfs osds even with --mkbtrfs

This lets you do a mixed btrfs and non-btrfs file systems.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'stable'

debian: break redundant dependencies

They confuse APT it seems.

 ceph-common -> librbd1 -> librados2
 radosgw -> ceph-common -> librados2

Signed-off-by: Sage Weil <sage@newdream.net>

MOSDMap: do not leave {oldest,newest}_map uninitialized when decoding old messages

This leads to badness like

  osd_map(295..296 src has 74308224..0) v1

Signed-off-by: Sage Weil <sage@newdream.net>

include stdio in order to fix snprintf compilation error

Signed-off-by: Alexey Lapitsky <lex@realisticgroup.com>

ceph: fix snprintf warning

warning: tools/ceph.cc:146: format not a string literal and no format arguments

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

auth: return unknown if no supported auth is found

If NONE is supported, it will already be in the list of supported
protocols, so there's no need to default to it here. This prevents
clients that request the NONE protocol from authenticating when the
server only accepts CEPHX. Instead, they get -ENOTSUP from the
AuthMonitor.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: swift related adjustments

Merge branch 'master' of github.com:NewDreamNetwork/ceph

fixed graphic reference and headings

fixed image reference

fixed architecture document

add images for documentation

rgw: handle swift PUT with incorrect etag

rgw: handle swift PUT with incorrect etag

rgw: handle swift PUT with incorrect etag

rgw: handle swift PUT with incorrect etag

rgw: handle swift PUT with incorrect etag

rgw: handle swift PUT with incorrect etag

ceph: refactor for generic --admin-daemon <sock> <cmd> too

Signed-off-by: Sage Weil <sage@newdream.net>

ceph: --dump-perf-counters[-schema] sockpath

Quick and dirty way to dump perfcounters stats.  Not documenting this until
we decide this is where it should live.

Signed-off-by: Sage Weil <sage@newdream.net>

filejournal: journal_replay_from

Force journal replay from a point other than the op_seq recorded by the
fs.  This is useful if you want to skip bad entries in the journal (e.g.,
because they were non-idempotent and you know they were applied and the fs
operations were fully ordered).

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'stable'

rados: improve error message

Signed-off-by: Sage Weil <sage@newdream.net>

radosgw-admin: fix key create check

Also fixes warning

warning: rgw/rgw_admin.cc:812: suggest parentheses around ‘&&’ within ‘||’

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: guard checks for writes

fa722de6708d3e92037df6289cc29ece12c8ea66 moved these checks, and
accidentally removed the may_write() guard. This caused reading from
snapshots to fail.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rgw: handle swift PUT with incorrect etag

rgw: rgw-admin --skip-zero-entries

perfcounters: fix accessor name

FreakingCamelCaps

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

objecter: instrument with perfcounter

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

rgw: rgw-admin generate-key/access-key=false fix

rgw: rgw-admin can show log summation

osd: read_log: only list the collection once

After upgrading we may need to list the collection to recover the hash
value when upgrading an old collection.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: fix bucket suspension

rgw: fix uninitialized variable warnings

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

Merge branch 'master' of ssh://github.com/NewDreamNetwork/ceph

Conflicts:
src/rgw/rgw_rados.cc

hadoop: bring back Java changes.

These convert the Hadoop stuff to work on the branch-0.20 API.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: fix attr cache

fix osdmaptool clitests

Merge branch 'wip-pools'

mon: reencode routed messages

The message encoding may depend on the target features.  Clear the
payload so that the Message gets reencoded appropriately.

Signed-off-by: Sage Weil <sage@newdream.net>

MOSDMap: reencode full map embedded in Incremental, as needed

The Incremental may have a bufferlist containing a full map; reencode
that too if we are reencoding for old clients.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote-tracking branch 'gh/wip-rbd-tool'

mon: fix rare races with pool updates

Signed-off-by: Sage Weil <sage@newdream.net>

mon: parse 0 values properly

Signed-off-by: Sage Weil <sage@newdream.net>

Merge remote branch 'gh/wip-osd-queue'

osd: fix last_complete adjustment after recovering an object

After we recover each object, we try to raise the last_complete value
(and matching complete_to iterator).  If our log was purely a backlog, this
won't necessarily end up bringing last_complete all the way up to the
last_update value, and we'll fail an assert later.

If complete_to does reach the end of the log, then we fast-forward
last_complete to last_update.

The crash we were hitting was in finish_recovery(), and looked something
like

osd/PG.cc: In function 'void PG::finish_recovery(ObjectStore::Transaction&, std::list<Context*, std::allocator<Context*> >&)', in thread '0x7f4573df7700'
osd/PG.cc: 1800: FAILED assert(info.last_complete == info.last_update)
 ceph version 0.36-251-g6e29c28 (commit:6e29c2826066a7723ed05b60b8ac0433a04c3c13)
 1: (PG::finish_recovery(ObjectStore::Transaction&, std::list<Context*, std::allocator<Context*> >&)+0x8d) [0x6ff0ed]
 2: (PG::RecoveryState::Active::react(PG::RecoveryState::ActMap const&)+0x316) [0x729196]
 3: (boost::statechart::simple_state<PG::RecoveryState::Active, PG::RecoveryState::Primary, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x21b) [0x759c0b]
 4: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x8d) [0x7423dd]
 5: (PG::RecoveryState::handle_activate_map(PG::RecoveryCtx*)+0x183) [0x711f43]
 6: (OSD::activate_map(ObjectStore::Transaction&, std::list<Context*, std::allocator<Context*> >&)+0x674) [0x579884]
 7: (OSD::handle_osd_map(MOSDMap*)+0x2270) [0x57bd50]
 8: (OSD::_dispatch(Message*)+0x4d0) [0x596bb0]
 9: (OSD::ms_dispatch(Message*)+0x17b) [0x59803b]
 10: (SimpleMessenger::dispatch_entry()+0x9c2) [0x617562]
 11: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x4a3dec]
 12: (Thread::_entry_func(void*)+0x12) [0x611a92]
 13: (()+0x7971) [0x7f457f87b971]
 14: (clone()+0x6d) [0x7f457e10b92d]

Fixes: #1609
Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix generate_past_intervals maybe_went_rw on oldest interval

We stop working backwards when we hit last_epoch_clean, which means for the
oldest interval first_epoch may not be the _real_ first_epoch.  (We can't
continue working backward because we may have thrown out those maps
entirely.)

However, if the last_epoch_clean epoch is contained within that interval,
we know that the OSD did in fact go rw because it had to have completed
recovery (and thus peering) to set last_clean_epoch in the first place.

This fixes cases where two different nodes have slightly different
past intervals, generate different prior probe sets as a result, and
flip/flop on the acting set choice.  (It may have eventually resolved when
the wrongly excluded node's notify races and arrives in time to be
considered, but that's still clearly no good.)

This does leave the start epoch for that oldest interval incorrect.  That
doesn't currently matter except that it's confusing, but I'm not sure how
to mark it properly, or if it's worth the effort.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: MOSDPGNotify: print prettier

Signed-off-by: Sage Weil <sage@newdream.net>

osd: print useful debug info from choose_acting

Signed-off-by: Sage Weil <sage@newdream.net>

osd: make proc_replica_log missing dump include useful information

I needed to see have/need to debug a weird unfound issue turned up by
thrashing.

Signed-off-by: Sage Weil <sage@newdream.net>

osd: fix/simplify op discard checks

Use a helper to determine when we should discard an op due to the client
being disconnected.  Use this when the op is first received, (re)queued,
and dequeued.

Fix the check to keep ops that are replayed ACKs, as we should make every
effort to reapply those even when the client goes away.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: move queue checks into enqueue_op, kill _handle_ helpers

This simplifies things, and renames the checks to make it clear that we are
doing validation checks only, with no side-effects allowed.

Also move some checks into the parent handle_op() to further simplify the
(re)queue checks.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: move op cap check into helper

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: drop useless PG hooks

These no longer need to be exposed to the generic OSD code.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: drop ability to disable op queue entirely

This is pretty useless, and broken wrt requeueing anyway.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: handle missing/degraded in op thread

The _handle_op() method (and friends) are called when an op is initially
queued and when it is requeued.  In the requeue case we have to be more
careful because the caller may be in the middle of doing all sorts of
random stuff.  That means we need to limit ourselves to queueing or
discarding the op, and refrain from doing anything else with dangerous
side effects.

This fixes a crash like

osd/ReplicatedPG.cc: In function 'void ReplicatedPG::recover_primary_got(hobject_t, eversion_t)', in thread '7f21d0189700'
osd/ReplicatedPG.cc: 4109: FAILED assert(missing.num_missing() == 0)
 ceph version 0.37-105-gc2069eb (commit:c2069eb1e562ba7d753c9b5ce5c904f4f5ef6abe)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x76) [0x8ab95a]
 2: (ReplicatedPG::recover_primary_got(hobject_t, eversion_t)+0x62e) [0x767eea]
 3: (ReplicatedPG::sub_op_push(MOSDSubOp*)+0x2b79) [0x76abeb]
 4: (ReplicatedPG::do_sub_op(MOSDSubOp*)+0x1ab) [0x74761b]
 5: (OSD::dequeue_op(PG*)+0x47d) [0x820ac3]
 6: (OSD::OpWQ::_process(PG*)+0x27) [0x82cc8b]

due to an object being pushed to a replica before it is activated.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

osd: set reqid on push/pull ops

Not strictly necessary, but makes logs easier to follow.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: remove compatset cruft

The CompatSet is built on demand; it's no longer static.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

FileStore: ignore EEXIST on clones and collection creation !btrfs_snap

We need to ignore EEXIST on btrfs also when m_filestore_btrfs_snap is
disabled.

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

ReplicatedPG: fix snapshot directory handling in snap_trimmer

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

rgw: fix rgw_obj compare function

rgw: use a uint64_t instead of a size_t for storing the size

librados uses uint64_t so that 32-bit architectures aren't hobbled.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

workunits: test rbd python bindings

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd.py: update python bindings for new copy interface

It was changed to return 0 on success in d7f7a213546b599d2eec4c6617593d232b43a7d6

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librados: use stored snap context for all operations

Using an empty snap context led to the failure of
test_rbd.TestImage.test_rollback_with_resize, since clones weren't
created when deleting objects. This test now passes.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: resize if necessary before rolling back

This is a partial fix for test_rbd.TestImage.test_rollback_with_resize

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

test_rbd: add a test for rolling back after resizing

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: propagate error from snap_set

Previously rbd_snap_set always returned 0, even when the snapshot did
not exist.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

workunits: add rbd rollback and snapshot removal tests

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: remove unnecessary condition

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

clitests: add rbd usage and invalid snap usage tests

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

workunit: check that rbd info returns the right size for snapshots

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

librbd: show correct size for snapshots

header.size is the current size of the image.
ImageCtx::get_image_size() already does the right thing for
snapshots.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: let all commands use the pool/image@snapshot format

This way you aren't forced to use '-p' or '--snap' to specify a pool
or snapshot for some commands.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: specify which commands take --snap in usage

Maybe this will be less confusing.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

rbd: check command before opening the image

Now map/unmap won't use librbd, and commands that don't take --snap
will give an error when it's used.

Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>

test/osd: Add TestReadWrite

Signed-off-by: Samuel Just <samuel.just@dreamhost.com>

mon: allow adjustment of per-pool crash_replay_interval

Signed-off-by: Sage Weil <sage@newdream.net>

Merge branch 'rgw-dir-cleanup'

rgw: fix check_disk_state; add a strip_namespace function.

Use copies of the IoCtx rather than references so that
we can set locators without breaking stuff, and make use of the
on-disk locators which we just added.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: add locators to the directory objects, and functions handling them

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

rgw: rename translate_raw_obj to translate_raw_obj_to_obj_in_ns

And document it. Because the naming is so bad that neither I nor
the author noticed it wasn't doing what we wanted it to until I ran
a test and it failed.

Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>

librados: behave if shutdown is called twice

On failure, we shut ourselves down.  If the caller calls shutdown again,
don't crash.

Fixes: #1650
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>

mon: need to print pool id for output to be useful

Signed-off-by: Sage Weil <sage@newdream.net>