Merge remote-tracking branch 'gh/next'

testing: fix hadoop-internal-test

Remove now superfluous directory changes
that are causing tests to fail.
This code should have been removed when we transitioned
from running tests with Ant to using Java to run the tests.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #149 from ceph/wip-4530

#4530

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #139 from ceph/wip-topo-java

Merging in Noah's branch for adding topology calls. This passes existing libcephfs, libcephfs-java and hadoop tests.

client: Don't signal requests already handled

The assertion failure reported in #4530 is triggered
by the following:

1. client sends request
2. mds sends unsafe reply
3. before request gets journaled, mds is killed
4. mds restarts
5. client receives session close (from close request before restart)
6. session close does kick_requests()
7. kick_requests tries to signal caller that doesn't exist.

This fix avoids signaling a caller if the unsafe reply
has been received and the make_request() function has completed.
We do this by setting the caller_cond to null once the caller
is woken up, and only signal the caller in kick_requests if
caller_cond is non-null.  This avoids trying to resend requests
listed in mds_request but that have already received unsafe replies.
The unsafe requests are handled by resend_unsafe_requests() code,
so skipping those requests is allowable.

Fixes #4530.
Signed-off-by: Sam Lang <sam.lang@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #144 from dalgaaf/wip-da-ceph-disk

Fix some issues in ceph-dsk

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #143 from ceph/wip-mds-health

improve mds health checks

Reviewed-by: Sam Lang <sam.lang@inktank.com>

ceph-disk:  udevadm settle before partprobe

After changing the partition table, allow the udev event to be
processed before calling partprobe.  This helps prevent partprobe
from getting a resource busy error on some platforms.

Signed-off-by: Gary Lowell <gary.lowell@inktank.com>

Merge pull request #147 from ceph/wip-4537

mds: CInode::build_backtrace() always incr iter

Reviewed-by: Sage Weil <sage@inktank.com>

mds: CInode::build_backtrace() always incr iter

Always increment the iterator when adding old pools
to the backtrace.  This fixes a bug on files where
the layout had been set to a different pool and then
back to the same pool, causing continuous looping in
the build_backtrace() function.

Fixes #4537.
Signed-off-by: Sam Lang <sam.lang@inktank.com>

java: fix test name typo

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #145 from dalgaaf/wip-da-c_str

CrushWrapper.cc: remove some std::string::c_str() calls

Reviewed-by: Sage Weil <sage@inktank.com>

client: Cleanup request signaling

Split up the conditionals handling unsafe reply
and signaling the caller to improve readability.
The overall behavior of the code remains the same.

Signed-off-by: Sam Lang <sam.lang@inktank.com>

client: Handle duplicate safe replies

If the mds sends a duplicate safe reply, the mds_requests
map won't contain a matching request id (tid).  Instead of
assert failing, we log a message that we saw a reply without
a matching request.

Also remove redundant mds_requests->erase(tid) line.

Signed-off-by: Sam Lang <sam.lang@inktank.com>

client: Always cleanup request after safe

The client MetaRequest should always be cleaned up
and removed from the mds_requests map once the client
gets a safe reply.  This patch avoids a leak where the
mds does not send back an unsafe reply and the request
is never cleaned up.

Signed-off-by: Sam Lang <sam.lang@inktank.com>

client: Remove got_safe from MetaRequest

Once a safe reply is received, we remove the
request from the mds_requests map, so checking that
it might be a duplicate won't succeed.  This patch
removes the got_safe checks in the reply handling code
and the got_safe field on the MetaRequest to avoid confusion.

Signed-off-by: Sam Lang <sam.lang@inktank.com>

CrushWrapper.cc: remove some std::string::c_str() calls

Passing the result of c_str() to a function that takes
std::string as argument is slow and redundant.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge remote-tracking branch 'gh/wip-crush'

The non-crush bits
Reviewed-by: Dan Mick <dan.mick@inktank.com>

doc/release-notes: extra note for v0.56.4

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: v0.56.4

Signed-off-by: Sage Weil <sage@inktank.com>

Improve test by getting cloneid from my_snaps vector

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Sam Just <sam.just@inktank.com>

rgw: bucket index ops on system buckets shouldn't do anything

Fixes: #4508
Backport: bobtail
On certain bucket index operations we didn't check whether
the bucket was a system bucket, which caused the operations
to fail. This triggered an error message on bucket removal
operations.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

ceph-disk: rename some local variabels in list_*partitions

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: fix naming of a local variable in find_cluster_by_uuid

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: rename some constants to upper case variable names

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: add some more docstrings

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: print subprocess.CalledProcessError on error

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph-disk: fix indention

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

java: pretty print Ceph extent

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

java: support ceph_get_osd_addr

Adds a few JNI utilities from the Android project (license: Apache) to
help with IP address conversions. These functions are also updated to
work in our environment (use Ceph exception utilities, edit header
paths).

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

java: support ceph_get_osd_crush_location

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

java: support ceph_get_file_extent_osds

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge branch 'next'

client: don't set other if lookup fails in rename

On rename, only set the other inode if the
lookup for the destination succeeds, otherwise we hit
a segv in set_other_inode().

Fixes #4517.
Signed-off-by: Sam Lang <sam.lang@inktank.com>
Tested-by: Noah Watkins <jayhawk@cs.ucsc.edu>

Merge branch 'next'

test/libcephfs: Test rename error cases

Make sure that rename fails with the ENOENT
if the source path doesn't exist.

Signed-off-by: Sam Lang <sam.lang@inktank.com>

client: Fix rename returning ENOENT for dest

Introduced by fc80c1dc6ee315ae5e039986602ffadba46cb43b,
the client should _not_ fail if the lookup for the
destination path on rename returns ENOENT.

The previous code also did not check that the lookup
returned ENOENT or success.  We add the check and fail
if we get any other errors.

Fixes #4517.
Signed-off-by: Sam Lang <sam.lang@inktank.com>

MDSMap: improve health check

Note if the cluster is degraded.  If so, indicate specifically which MDSs
are degraded and what state they are in.

Signed-off-by: Sage Weil <sage@inktank.com>

MDSMap:: constify a bunch of methods

Signed-off-by: Sage Weil <sage@inktank.com>

preserve /var/lib/ceph on deb/rpm purge

We should clobber configuration and log data, but *not* user data.  Leave
/var/lib/ceph alone.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Gary Lowell <gary.lowell@inktank.com>

mon: factor out _get_pending_crush() helper

Signed-off-by: Sage Weil <sage@inktank.com>

mon, crush: add some tests to build a DAG via the cli

Signed-off-by: Sage Weil <sage@inktank.com>

crush, mon: unlink vs remove

Make an 'unlink' mode of remove that will remove a link to a bucket but
not remove the bucket itself.  This refactors remove_item[_under] and moves
some of the checks into common helpers where they are not duplicated.  Fix
callers to pass the extra arg.

Signed-off-by: Sage Weil <sage@inktank.com>

crush: fix remove_item on bucket removal

Remove the bucket if there are no references left.

Remove the name from the map even if it is a bucket (not sure why that
condition was there in the first place!).

Signed-off-by: Sage Weil <sage@inktank.com>

mon: add 'osd crush add-bucket <name> <type>'

This is (I think) the last missing piece to let you construct an entire
map via the CLI.  The add/set commands will construct intervening ancestor
nodes provide there is an existing ancestor to stick them under, but this
is needed to create the initial root node.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: allow removal of buckets via 'osd crush rm ...'

No reason to limit this to leaves.

Signed-off-by: Sage Weil <sage@inktank.com>

crush: change find_roots(); add find_takes()

The find_roots() was looking for nodes referenced by 'take', but those
aren't necessarily roots, which is what the callers actually want.

Rename to find_takes() and add a real find_roots().  Not very efficient,
but we don't care.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: add optional ancestor arg to 'ceph osd crush rm <item> [ancestor]'

Remove only instances of the item underneath a particular ancestor.

Signed-off-by: Sage Weil <sage@inktank.com>

crush: add remove_item_under()

Remove only instances of item nested beneath a particular ancestor.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: 'ceph osd crush link ...' to add a link to an existing bucket

Allow a second reference to an existing bucket to be added.  This lets
you create a DAG instead of a tree using the CLI.

Signed-off-by: Sage Weil <sage@inktank.com>

crush: prevent formation of a loop

If we are adding an item, ensure it cannot form a loop in the tree/map/
DAG.

Signed-off-by: Sage Weil <sage@inktank.com>

crush: add link_bucket()

Allow an existing bucket to get linked from a new position in the tree.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: 'ceph osd crush add ...' to add a second link to an item

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'upstream/wip_4435'

Fixes: #4435
Reviewed-by: David Zafman <david.zafman@inktank.com>

PG::GetMissing: need to check need_up_thru in MLogRec handler

Backport: bobtail
Fixes: #4534
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

PG,osd_types: improve check_new_interval debugging

Backport: bobtail
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

common/MemoryModel: remove logging to /tmp/memlog

This was a hack for dev purposes ages ago; remove it.  The predictable
filename is a security issue.

CVE-2013-1882

Reported-by: Michael Scherer <misc@zarb.org>
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

init-ceph: clean up temp ceph.conf filename on exit

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

init-ceph: push temp conf file to a unique location on remote host

The predictable file name is a security problem.

CVE-2013-1882

Reported-by: Michael Scherer <misc@zarb.org>
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

mkcephfs: make remote temp directory name unique

The predictable file name is a security problem.

CVE-2013-1882

Reported-by: Michael Scherer <misc@zarb.org>
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #130 from ceph/wip-fs-rename

test: add ceph_rename test

Reviewed-by: Sage Weil <sage@inktank.com>

doc: Added {id} argument to OSD lost.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

ceph-disk: re-add python 2.7 dependency comment

FIXME!

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #117 from ceph/wip-ceph-disk

ceph-disk-* refactor

Merge branch 'next'

osd: reenable 'journal aio = true'

Now that #4079 is resolved.  Reverts 1cfc3ae0.

Signed-off-by: Sage Weil <sage@inktank.com>

os/FileJournal: fix aio self-throttling deadlock

This block of code tries to limit the number of aios in flight by waiting
for the amount of data to be written to grow relative to a function of the
number of aios.  Strictly speaking, the condition we are waiting for is a
function of both aio_num and the write queue, but we are only woken by
changes in aio_num, and were (in rare cases) waiting when aio_num == 0 and
there was no possibility of being woken.

Fix this by verifying that aio_num > 0, and restructuring the loop to
recheck that condition on each wakeup.

Fixes: #4079
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #137 from dalgaaf/wip-da-cleanup-includes

Cleanup some twice included header

Reviewed-by: Sage Weil <sage@inktank.com>

test/test_snap_mapper.cc: remove twice included <tr1/memory>

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/MDSMonitor.cc: remove twice included MonitorDBStore.h

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/LogMonitor.cc: remove twice included <sstream>

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/AuthMonitor.cc: remove twice included <sstream>

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

common/Formatter.h: remove twice included <list>

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ReplicatedPG: add debug flag to skip full check at reservation

This will make it easier to test the check in do_scan.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: replica should post BackfillTooFull in do_scan if full

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: halt backfill on RemoteReservationRejected in Backilling

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: add helper for adding a timer event to retry backfill

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: add BackfillTooFull event for RepRecovering

Replica will use this to notify Primary to stop backfilling.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: add helper for rejecting backfill reservation

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: use OSDService::too_full_for_backfill in RepWaitBackfillReserved

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDService: add too_full_for_backfill

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge remote-tracking branch 'upstream/wip_osd_shutdown_notification'

Fixes: #1857
Fixes: #4267
Reviewed-by: Sage Weil <sage@inktank.com>

Makefile: add MOSDMarkMeDown

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: notify mon prior to shutdown

Signed-off-by: Samuel Just <sam.just@inktank.com>

Monitor: add MOSDMarkMeDown support

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDMonitor: factor out check_source helper

Signed-off-by: Samuel Just <sam.just@inktank.com>

messages: add MOSDMarkMeDown

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: clear OpHistory on shutdown

Signed-off-by: Samuel Just <sam.just@inktank.com>

OpRequest: use OpRequestRef for OpHistory

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore::stat: valgrind: don't read *st on error

Signed-off-by: Samuel Just <sam.just@inktank.com>

shared_cache: clear lru in destructor

Otherwise, the live references will attempt to extricate
themselves from a disolving SharedLRU instance as the
member destructors run.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ceph_osd: clear client_throttler prior to putting g_ceph_context

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: reorder OSD::shutdown

Reorder teardown:
- pgs
- queues/threadpools
- persist superblock
- filestore
- timers
- messengers

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: check for is_stopping after locking osd_lock or heartbeat_lock

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: lookup_lock_raw_pg is dead

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: rename timer to tick_timer

Only used for scheduling ticks - we should keep it
that way.

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: move backfill_request_timer cleanup to OSDService::shutdown

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG,OSD: move dequeueing into on_shutdown

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: clarify PG::deleting comment

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #135 from ceph/wip-4519

mon: AuthMonitor: delete auth_handler while increasing max_global_id

Reviewed-by: Sage Weil <sage@inktank.com>