Merge pull request #117 from ceph/wip-ceph-disk

ceph-disk-* refactor

Merge branch 'next'

osd: reenable 'journal aio = true'

Now that #4079 is resolved.  Reverts 1cfc3ae0.

Signed-off-by: Sage Weil <sage@inktank.com>

os/FileJournal: fix aio self-throttling deadlock

This block of code tries to limit the number of aios in flight by waiting
for the amount of data to be written to grow relative to a function of the
number of aios.  Strictly speaking, the condition we are waiting for is a
function of both aio_num and the write queue, but we are only woken by
changes in aio_num, and were (in rare cases) waiting when aio_num == 0 and
there was no possibility of being woken.

Fix this by verifying that aio_num > 0, and restructuring the loop to
recheck that condition on each wakeup.

Fixes: #4079
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #137 from dalgaaf/wip-da-cleanup-includes

Cleanup some twice included header

Reviewed-by: Sage Weil <sage@inktank.com>

test/test_snap_mapper.cc: remove twice included <tr1/memory>

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/MDSMonitor.cc: remove twice included MonitorDBStore.h

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/LogMonitor.cc: remove twice included <sstream>

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/AuthMonitor.cc: remove twice included <sstream>

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

common/Formatter.h: remove twice included <list>

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge remote-tracking branch 'upstream/wip_osd_shutdown_notification'

Fixes: #1857
Fixes: #4267
Reviewed-by: Sage Weil <sage@inktank.com>

Makefile: add MOSDMarkMeDown

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: notify mon prior to shutdown

Signed-off-by: Samuel Just <sam.just@inktank.com>

Monitor: add MOSDMarkMeDown support

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDMonitor: factor out check_source helper

Signed-off-by: Samuel Just <sam.just@inktank.com>

messages: add MOSDMarkMeDown

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: clear OpHistory on shutdown

Signed-off-by: Samuel Just <sam.just@inktank.com>

OpRequest: use OpRequestRef for OpHistory

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore::stat: valgrind: don't read *st on error

Signed-off-by: Samuel Just <sam.just@inktank.com>

shared_cache: clear lru in destructor

Otherwise, the live references will attempt to extricate
themselves from a disolving SharedLRU instance as the
member destructors run.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ceph_osd: clear client_throttler prior to putting g_ceph_context

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: reorder OSD::shutdown

Reorder teardown:
- pgs
- queues/threadpools
- persist superblock
- filestore
- timers
- messengers

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: check for is_stopping after locking osd_lock or heartbeat_lock

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: lookup_lock_raw_pg is dead

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: rename timer to tick_timer

Only used for scheduling ticks - we should keep it
that way.

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: move backfill_request_timer cleanup to OSDService::shutdown

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG,OSD: move dequeueing into on_shutdown

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: clarify PG::deleting comment

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #135 from ceph/wip-4519

mon: AuthMonitor: delete auth_handler while increasing max_global_id

Reviewed-by: Sage Weil <sage@inktank.com>

mon: AuthMonitor: delete auth_handler while increasing max_global_id

By not deleting and setting NULL the session's auth_handler, we could
hit a scenario in which we'd end up dispatching a previously-wait-listed
auth message and we wouldn't start its auth session.

This only happened when increasing max_global_id via Paxos (in which case
we would wait-list the message) and would only be noticeable when running
with cephx disabled.

Fixes: #4519
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

FileJournal: quieter debugging on journal scanning

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 6740d512ac12263f7bee370bc14b1179f83af5be)

FileJournal: quieter debugging on journal scanning

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>

doc: Fixed some typos.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #128 from dalgaaf/wip-da-pylint

Fix some python issues

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #132 from ceph/wip-libcephfs-fix

libcephfs: fix ceph_get_osd_crush_location

Reviewed-by: Sage Weil <sage@inktank.com>

libcephfs: fix ceph_get_osd_crush_location

(a) Allow NULL buffer when length is zero to get the target buffer
length. (b) fix edge case where buffer length is the exact size
required.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #127 from alram/master

[doc] undisplayed notes, tips, and important sections

Reviewed-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #120 from ceph/wip-msgr-acks

msg/Pipe: fix seq handshake on reconnect

Reviewed-by: Greg Farnum <greg@inktank.com>

rados.py: remove unnecessary semicolon

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: fix bad indention

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: remove lenght modifier 'l' on format string

Remove length modifier 'l' since it's ignored for python.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: fix 'Rados' has no member 'name'

Lets store rados_id from __init__ and use this instead of 'name'.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: add missing space after ','

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-create-keys: rename log to LOG since it's a constants

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge branch 'next'

doc/release-notes: v0.59

Signed-off-by: Sage Weil <sage@inktank.com>

msg/Pipe: fix seq handshake on reconnect

We go to the trouble to exchange our seq numbers during the handshake, but
the bit that then avoids resending old messages was broken because we
already requeue_sent() before we get to this point.  Fix it by discarding
queued items (in the high prio slot) that we don't need to resend, and
adjust out_seq as needed.

Drop the optional arg to requeue_sent() now that it is unused.

Signed-off-by: Sage Weil <sage@inktank.com>

Fix tips in documentation

Use of .. tip: instead of .. tip:: resulting in content not
being displayed.

Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>

Fix important in documentation

Use of .. important: instead of .. important:: resulting in content
not being displayed.

Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>

Fix notes in documentation

Use of .. note: instead of .. note:: resulting in content not
being displayed

Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>

Merge pull request #126 from alram/master

Update Chef deployment documentation

Reviewed-by: Sage Weil <sage@inktank.com>

Update Chef deployment documentation

Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>

Merge pull request #124 from ceph/wip-4509

mon: DataHealthService: shutdown mon if failed to obtain disk stats

Reviewed-by: Sage Weil <sage@inktank.com>

mon: DataHealthService: log to derr instead if we're about to shutdown

Otherwise the message would -- or could -- be lost.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: DataHealthService: shutdown mon if failed to obtain disk stats

Being unable to run a ::statfs() may be a symptom of something bigger.

We want to cleanly shutdown the monitor ASAP if such thing happens.

Fixes: #4509
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

Merge pull request #123 from dalgaaf/wip-da-sca-misc-1

Some smaller misc fixes

Reviewed-by: Sage Weil <sage@inktank.com>

client/Client.cc: handle error if _lookup() fails

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

qa/workunits/direct_io/test_sync_io.c: add proper error handling

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_short_dio_read.c: add proper error handling

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/Locker.cc: prefer prefix ++operator for iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mount/mount.ceph.c: remove unused variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #122 from dalgaaf/wip-da-ceph-disk-1

More fixes for ceph-disk

Reviewed-by: Sage Weil <sage@inktank.com>

ceph-disk: cast output of subprocess.Popen() to str()

Cast output of subprocess.Popen() to str() to be able to use
str.split() and str.splitlines() without warnings from pylint.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #121 from ceph/wip-4448

mon: Monitor: clearer output on error during attempt to convert store

Reviewed-by: Sage Weil <sage@inktank.com>

mon: Monitor: clearer output on error during attempt to convert store

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

ceph-disk: fix adjust_symlink() replace 'journal' with 'target'

Replace 'journal' variable with 'target' since journal doesn't
exist in this function.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: fix adjust_symlink() replace 'canonical' with 'path'

Replace 'canonical' variable with 'path' since canonical doesn't
exist in this function.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: there is no os.path.lstat use os.lstat

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: remove unused variable key from prepare_journal_dev()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: fix except to catch OSError

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: add missing space after >> operator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

fix: Redefining name 'uuid' from outer scope (line 14)

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: remove unused variables from list_partitions()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: fix /dev/dm-[0-9] handling list_all_partitions()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

v0.59

Merge pull request #106 from ceph/wip-crush

crush: update weights properly for DAG (not tree) maps

Reviewed-by: caleb miles <caleb.miles@inktank.com>

Merge pull request #119 from dalgaaf/wip-da-ceph-disk

Cleanup and fixes for wip-ceph-disk

ceph-disk: rename local variable shadowing builtin

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: remove twice defined identical function unmount

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: remove twice defined function mount

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: remove double defined function get_conf

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: rename local variable shadowing builtin

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #118 from dalgaaf/wip-da-enum

QuorumService.h: use enum instead of static const int

Reviewed-by: Sage Weil <sage@inktank.com>

QuorumService.h: use enum instead of static const int

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Missed adding rados_types.hpp to package

Caused by 3bd48cbbadb7908dd833ccde75359f085828fc5c
feature 4207 implementation

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Gary Lowell <gary.lowell@inktank.com>
(cherry picked from commit e1e2d5d2176cc9debd436ba944e6ca65b3253c8a)

ceph-disk: install and package

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-disk: simplify command dispatch

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-disk: consolidate exceptions

Use a single exception type, and catch it at the top level.

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-disk: consolidate ceph-disk-* into a single binary

 ceph-disk prepare ...
 ceph-disk activate ...
 ceph-disk ...

This let's us share code (we were already duplicating a bunch of stuff!)
and to add new commands easily.

Signed-off-by: Sage Weil <sage@inktank.com>

PendingReleaseNotes: fix typo

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

librbd: optionally wait for a flush before enabling writeback

Older guests may not send flushes properly (i.e. never), so if this is
enabled, rbd_cache=true is safe for them transparently.

Disable by default, since it will unnecessarily slow down newer guest
boot, and prevent writeback caching for things that don't need to send
flushes, like the command line tool.

Refs: #3817
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Makefile: missing header

Signed-off-by: Sage Weil <sage@inktank.com>

mon: use enum instead of static const int

This way it compiles.

Signed-off-by: Sage Weil <sage@inktank.com>

mon/Paxos: set state to RECOVERING during restart

This ensures that the paxos state is not active when the PaxosService
restart() methods run right afterwards, and that EAGAIN waiters will get
requeued appropriately.

Signed-off-by: Sage Weil <sage@inktank.com>

Makefile.am: fix misspelt header name

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon/PaxosService: handle non-zero return values

If 7aec13f749035b9bef5e398c1ac3d56ceec8eb81 we started passing non-zero
return values to these completions; now we have to deal with them
accordingly.

RetryMessage behaves just like the Monitor variant.

Propose and Committed update state but otherwise ignore non-zero
return values.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge branch 'next'

ceph-disk-prepare: 'mkfs -t' instead of 'mkfs --type='

Older mkfs (el6) doesn't like --type=.

Fixes: #4495
Reported-by: Alexandre Maragone <alexandre.maragone@inktank.com>
Signed-off-by: Sage Weil <sage@inktank.com>

mds: Handle ENODATA returned from getxattr

The osds might return ENODATA if we request an
xattr that doesn't exist.  In this case, we're
requesting the 'parent' xattr so that we can
remove all the forwarding pointers, but the xattr
may not have been written (which only happens on
log segment trim), so we don't assert here.

Signed-off-by: Sam Lang <sam.lang@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

mon/PaxosService: fix proposal waiter handling

- Cancel the propsal waiters with EAGAIN on election, etc.
- Drop the wakeup helper and open-code the one caller.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>