CrushWrapper.cc: remove some std::string::c_str() calls

Passing the result of c_str() to a function that takes
std::string as argument is slow and redundant.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge remote-tracking branch 'gh/wip-crush'

The non-crush bits
Reviewed-by: Dan Mick <dan.mick@inktank.com>

doc/release-notes: extra note for v0.56.4

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: v0.56.4

Signed-off-by: Sage Weil <sage@inktank.com>

Improve test by getting cloneid from my_snaps vector

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Sam Just <sam.just@inktank.com>

Merge branch 'next'

client: don't set other if lookup fails in rename

On rename, only set the other inode if the
lookup for the destination succeeds, otherwise we hit
a segv in set_other_inode().

Fixes #4517.
Signed-off-by: Sam Lang <sam.lang@inktank.com>
Tested-by: Noah Watkins <jayhawk@cs.ucsc.edu>

Merge branch 'next'

test/libcephfs: Test rename error cases

Make sure that rename fails with the ENOENT
if the source path doesn't exist.

Signed-off-by: Sam Lang <sam.lang@inktank.com>

client: Fix rename returning ENOENT for dest

Introduced by fc80c1dc6ee315ae5e039986602ffadba46cb43b,
the client should _not_ fail if the lookup for the
destination path on rename returns ENOENT.

The previous code also did not check that the lookup
returned ENOENT or success.  We add the check and fail
if we get any other errors.

Fixes #4517.
Signed-off-by: Sam Lang <sam.lang@inktank.com>

preserve /var/lib/ceph on deb/rpm purge

We should clobber configuration and log data, but *not* user data.  Leave
/var/lib/ceph alone.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Gary Lowell <gary.lowell@inktank.com>

mon: factor out _get_pending_crush() helper

Signed-off-by: Sage Weil <sage@inktank.com>

mon, crush: add some tests to build a DAG via the cli

Signed-off-by: Sage Weil <sage@inktank.com>

crush, mon: unlink vs remove

Make an 'unlink' mode of remove that will remove a link to a bucket but
not remove the bucket itself.  This refactors remove_item[_under] and moves
some of the checks into common helpers where they are not duplicated.  Fix
callers to pass the extra arg.

Signed-off-by: Sage Weil <sage@inktank.com>

crush: fix remove_item on bucket removal

Remove the bucket if there are no references left.

Remove the name from the map even if it is a bucket (not sure why that
condition was there in the first place!).

Signed-off-by: Sage Weil <sage@inktank.com>

mon: add 'osd crush add-bucket <name> <type>'

This is (I think) the last missing piece to let you construct an entire
map via the CLI.  The add/set commands will construct intervening ancestor
nodes provide there is an existing ancestor to stick them under, but this
is needed to create the initial root node.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: allow removal of buckets via 'osd crush rm ...'

No reason to limit this to leaves.

Signed-off-by: Sage Weil <sage@inktank.com>

crush: change find_roots(); add find_takes()

The find_roots() was looking for nodes referenced by 'take', but those
aren't necessarily roots, which is what the callers actually want.

Rename to find_takes() and add a real find_roots().  Not very efficient,
but we don't care.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: add optional ancestor arg to 'ceph osd crush rm <item> [ancestor]'

Remove only instances of the item underneath a particular ancestor.

Signed-off-by: Sage Weil <sage@inktank.com>

crush: add remove_item_under()

Remove only instances of item nested beneath a particular ancestor.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: 'ceph osd crush link ...' to add a link to an existing bucket

Allow a second reference to an existing bucket to be added.  This lets
you create a DAG instead of a tree using the CLI.

Signed-off-by: Sage Weil <sage@inktank.com>

crush: prevent formation of a loop

If we are adding an item, ensure it cannot form a loop in the tree/map/
DAG.

Signed-off-by: Sage Weil <sage@inktank.com>

crush: add link_bucket()

Allow an existing bucket to get linked from a new position in the tree.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: 'ceph osd crush add ...' to add a second link to an item

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'upstream/wip_4435'

Fixes: #4435
Reviewed-by: David Zafman <david.zafman@inktank.com>

PG::GetMissing: need to check need_up_thru in MLogRec handler

Backport: bobtail
Fixes: #4534
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

PG,osd_types: improve check_new_interval debugging

Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

common/MemoryModel: remove logging to /tmp/memlog

This was a hack for dev purposes ages ago; remove it.  The predictable
filename is a security issue.

CVE-2013-1882

Reported-by: Michael Scherer <misc@zarb.org>
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

init-ceph: clean up temp ceph.conf filename on exit

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

init-ceph: push temp conf file to a unique location on remote host

The predictable file name is a security problem.

CVE-2013-1882

Reported-by: Michael Scherer <misc@zarb.org>
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

mkcephfs: make remote temp directory name unique

The predictable file name is a security problem.

CVE-2013-1882

Reported-by: Michael Scherer <misc@zarb.org>
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #130 from ceph/wip-fs-rename

test: add ceph_rename test

Reviewed-by: Sage Weil <sage@inktank.com>

doc: Added {id} argument to OSD lost.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

ceph-disk: re-add python 2.7 dependency comment

FIXME!

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #117 from ceph/wip-ceph-disk

ceph-disk-* refactor

Merge branch 'next'

osd: reenable 'journal aio = true'

Now that #4079 is resolved.  Reverts 1cfc3ae0.

Signed-off-by: Sage Weil <sage@inktank.com>

os/FileJournal: fix aio self-throttling deadlock

This block of code tries to limit the number of aios in flight by waiting
for the amount of data to be written to grow relative to a function of the
number of aios.  Strictly speaking, the condition we are waiting for is a
function of both aio_num and the write queue, but we are only woken by
changes in aio_num, and were (in rare cases) waiting when aio_num == 0 and
there was no possibility of being woken.

Fix this by verifying that aio_num > 0, and restructuring the loop to
recheck that condition on each wakeup.

Fixes: #4079
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #137 from dalgaaf/wip-da-cleanup-includes

Cleanup some twice included header

Reviewed-by: Sage Weil <sage@inktank.com>

test/test_snap_mapper.cc: remove twice included <tr1/memory>

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/MDSMonitor.cc: remove twice included MonitorDBStore.h

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/LogMonitor.cc: remove twice included <sstream>

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/AuthMonitor.cc: remove twice included <sstream>

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

common/Formatter.h: remove twice included <list>

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ReplicatedPG: add debug flag to skip full check at reservation

This will make it easier to test the check in do_scan.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: replica should post BackfillTooFull in do_scan if full

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: halt backfill on RemoteReservationRejected in Backilling

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: add helper for adding a timer event to retry backfill

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: add BackfillTooFull event for RepRecovering

Replica will use this to notify Primary to stop backfilling.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: add helper for rejecting backfill reservation

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: use OSDService::too_full_for_backfill in RepWaitBackfillReserved

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDService: add too_full_for_backfill

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge remote-tracking branch 'upstream/wip_osd_shutdown_notification'

Fixes: #1857
Fixes: #4267
Reviewed-by: Sage Weil <sage@inktank.com>

Makefile: add MOSDMarkMeDown

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: notify mon prior to shutdown

Signed-off-by: Samuel Just <sam.just@inktank.com>

Monitor: add MOSDMarkMeDown support

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDMonitor: factor out check_source helper

Signed-off-by: Samuel Just <sam.just@inktank.com>

messages: add MOSDMarkMeDown

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: clear OpHistory on shutdown

Signed-off-by: Samuel Just <sam.just@inktank.com>

OpRequest: use OpRequestRef for OpHistory

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore::stat: valgrind: don't read *st on error

Signed-off-by: Samuel Just <sam.just@inktank.com>

shared_cache: clear lru in destructor

Otherwise, the live references will attempt to extricate
themselves from a disolving SharedLRU instance as the
member destructors run.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ceph_osd: clear client_throttler prior to putting g_ceph_context

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: reorder OSD::shutdown

Reorder teardown:
- pgs
- queues/threadpools
- persist superblock
- filestore
- timers
- messengers

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: check for is_stopping after locking osd_lock or heartbeat_lock

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: lookup_lock_raw_pg is dead

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: rename timer to tick_timer

Only used for scheduling ticks - we should keep it
that way.

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: move backfill_request_timer cleanup to OSDService::shutdown

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG,OSD: move dequeueing into on_shutdown

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: clarify PG::deleting comment

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #135 from ceph/wip-4519

mon: AuthMonitor: delete auth_handler while increasing max_global_id

Reviewed-by: Sage Weil <sage@inktank.com>

mon: AuthMonitor: delete auth_handler while increasing max_global_id

By not deleting and setting NULL the session's auth_handler, we could
hit a scenario in which we'd end up dispatching a previously-wait-listed
auth message and we wouldn't start its auth session.

This only happened when increasing max_global_id via Paxos (in which case
we would wait-list the message) and would only be noticeable when running
with cephx disabled.

Fixes: #4519
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

FileJournal: quieter debugging on journal scanning

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 6740d512ac12263f7bee370bc14b1179f83af5be)

FileJournal: quieter debugging on journal scanning

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>

doc: Fixed some typos.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #128 from dalgaaf/wip-da-pylint

Fix some python issues

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #132 from ceph/wip-libcephfs-fix

libcephfs: fix ceph_get_osd_crush_location

Reviewed-by: Sage Weil <sage@inktank.com>

libcephfs: fix ceph_get_osd_crush_location

(a) Allow NULL buffer when length is zero to get the target buffer
length. (b) fix edge case where buffer length is the exact size
required.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

test: add ceph_rename test

Triggers bug in ticket #4517

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #127 from alram/master

[doc] undisplayed notes, tips, and important sections

Reviewed-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #120 from ceph/wip-msgr-acks

msg/Pipe: fix seq handshake on reconnect

Reviewed-by: Greg Farnum <greg@inktank.com>

rados.py: remove unnecessary semicolon

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: fix bad indention

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: remove lenght modifier 'l' on format string

Remove length modifier 'l' since it's ignored for python.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados.py: fix 'Rados' has no member 'name'

Lets store rados_id from __init__ and use this instead of 'name'.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

crush: allow an item to be inserted twice

Modify insert_item() to let you add an existing item to a new position
in the map.

Signed-off-by: Sage Weil <sage@inktank.com>

crush: use const string& args

Compiler will convert const char * to a std::string implicitly.

Signed-off-by: Sage Weil <sage@inktank.com>

rados.py: add missing space after ','

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-create-keys: rename log to LOG since it's a constants

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge branch 'next'

doc/release-notes: v0.59

Signed-off-by: Sage Weil <sage@inktank.com>

msg/Pipe: fix seq handshake on reconnect

We go to the trouble to exchange our seq numbers during the handshake, but
the bit that then avoids resending old messages was broken because we
already requeue_sent() before we get to this point.  Fix it by discarding
queued items (in the high prio slot) that we don't need to resend, and
adjust out_seq as needed.

Drop the optional arg to requeue_sent() now that it is unused.

Signed-off-by: Sage Weil <sage@inktank.com>

Fix tips in documentation

Use of .. tip: instead of .. tip:: resulting in content not
being displayed.

Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>

Fix important in documentation

Use of .. important: instead of .. important:: resulting in content
not being displayed.

Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>

Fix notes in documentation

Use of .. note: instead of .. note:: resulting in content not
being displayed

Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>

Merge pull request #126 from alram/master

Update Chef deployment documentation

Reviewed-by: Sage Weil <sage@inktank.com>

Update Chef deployment documentation

Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>

Merge pull request #124 from ceph/wip-4509

mon: DataHealthService: shutdown mon if failed to obtain disk stats

Reviewed-by: Sage Weil <sage@inktank.com>

mon: DataHealthService: log to derr instead if we're about to shutdown

Otherwise the message would -- or could -- be lost.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: DataHealthService: shutdown mon if failed to obtain disk stats

Being unable to run a ::statfs() may be a symptom of something bigger.

We want to cleanly shutdown the monitor ASAP if such thing happens.

Fixes: #4509
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>