test: Wait for tier removal before next test starts

Fixes: #7860
Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge pull request #1535 from ceph/wip-7823

osd: trim copy-get backend read to object size

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd: trim copy-get backend read to object size

We are passing a big number to the backend to read and it is trimming it
to the stripe boundary, and then setting the cursor at a slightly smaller
offset bound by oi.size.  This is invalid, and will trigger an assert in
the _write_copy_chunk code:

     0> 2014-03-21 15:12:23.761509 7f8dd2324700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::_write_copy_chunk(ReplicatedPG::CopyOpRef, PGBackend::PGTransaction*)' thread 7f8dd2324700 time 2014-03-21 15:12:23.758866
osd/ReplicatedPG.cc: 5554: FAILED assert(cop->data.length() + cop->temp_cursor.data_offset == cop->cursor.data_offset)

To fix this, trim the buffer to the correct length in the completion
context.

Fixes: #7823
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1518 from dachary/wip-mailmap

mailmap updates

Reviewed-By: Christophe Courtaut <christophe.courtaut@gmail.com>

Merge remote-tracking branch 'gh/firefly'

osd: fix tests due to no default erasure-code ruleset

Signed-off-by: Sage Weil <sage@inktank.com>

Revert "osd: create the erasure-code ruleset in OSDMap::build_simple"

This reverts commit b85330eb1d62632a1c3f2025b1f309816d54bd5a.

We can't have this in the OSDMap by default or else all clients will require
the new features.

[Fixed up #include for strmap]

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1530 from onlyjob/fix-defaultweight

fix bug in 'defaultweight' calculation on OSD start.

Reviewed-by: Sage Weil <sage@inktank.com>

fix bug in 'defaultweight' calculation on OSD start.

  awk: cmd. line:1: { d=$2/1073741824 ; r = sprintf(\"%.2f\", d); print r }
  awk: cmd. line:1:                                 ^ backslash not last character on line
  awk: cmd. line:1: { d=$2/1073741824 ; r = sprintf(\"%.2f\", d); print r }
  awk: cmd. line:1:                                 ^ syntax error

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>

Merge branch 'wip-7676-firefly' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1533 from ceph/wip-7863-wusui

Make sure s3_utilities are found.

Make sure s3_utilities are found.

Add FindBin code to rgw/s3 tests to include s3_utilites.pm properly.

Fixes: 7863
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #1527 from ceph/wip-6465

be nitpicky about missing references

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1531 from onlyjob/debian

minor corrections from Debian

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1363 from ceph/wip-3863

Add ceph-client-debug inode lookup

Reviewed-by: Greg Farnum <greg@inktank.com>

minor corrections to package descriptions

minor init.d scripts lintianisation:

 * init.d-script-does-not-source-init-functions
 * init.d-script-does-not-implement-required-option
 * init.d-script-missing-lsb-description

rgw: don't modify rgw_bucket for data extra pool

This cleans up the interface, as we don't have this ugly implicit
structure modification when we deal with object that resides on the data
extra pool. Replcae lot's of calls to get_obj_bucket_and_oid_key() to a
cleaner get_obj_ref() that uses a struct to hold the updated fields, and
update the relevant callers.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: multipart meta object uses extra data pool

Fixes: #7676
We need the extra data pool since EC backends cannot handle the omap
operations that are needed for the multipart meta object.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: zone placement info includes extra data pool

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add data_extra pool to bucket

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1528 from ceph/wip-dz-watch-test

Reviewed-by: Samuel Just <sam.just@inktank.com>

mailmap: Inktank jenkins bot

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Huang Jun name normalization

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Tyler Brekke name normalization

Reviewed-by: Tyler Brekke <tyler.brekke@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Guang Yang affiliation

and name normalization

Reviewed-by: Guang Yang <yguang@yahoo-inc.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: James Ryan Cresawn affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Mohammad Salehe affiliation

Reviewed-by: Mohammad Salehe <salehe+dev@gmail.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Sharif Olorin affiliation

Reviewed-by: Sharif Olorin <sio@tesser.org>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Stephan Renatus affiliation

Reviewed-by: Stephan Renatus <s.renatus@x-ion.de>
Signed-off-by: Loic Dachary <loic@dachary.org>

qa: Add ceph_multi_stress_watch for rep and ec

Fixes: #7438
Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge pull request #1525 from ceph/wip-listxattr

client: don't include ceph.{file,dir}.layout vxattr in listxattr()

Reviwed-by: Sage Weil <sage@inktank.com>

ensure pybind is in the PYTHONPATH

Signed-off-by: Alfredo Deza <alfredo@deza.pe>

Merge pull request #1523 from ceph/wip-7827

msg: set recv stamp for local delivery

Reviewed-by: Greg Farnum <greg@inktank.com>

be nitpicky about missing references

Signed-off-by: Alfredo Deza <alfredo@deza.pe>

Merge pull request #1526 from ceph/wip-status-function-names

mon: Monitor: change 'status' function names for clarity's sake

Reviewed-by: Sage Weil <sage@inktank.com>

Revert "erasure-code: gf-complete detects SSE at runtime"

This reverts commit dbcf4471838fbf5dce2ee06fc73493f513dbf9f0.

Merge pull request #1524 from dachary/wip-sse

erasure-code: deactivate SSE optimizations

mon: Monitor: s/_mon_status()/get_mon_status()/

'Monitor::_mon_status()' provides status specific to the monitor being
poked.  This is information that does not necessarily relate with
cluster status, and can even be obtained when there's no quorum (via the
admin socket).

We change the function name to make the distinction between mon-specific
status and cluster-status, which is obtained via a different function.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Monitor: s/get_status()/get_cluster_status()/

Monitor::get_status() is actually used to obtain 'cluster status' (i.e.,
status about *all* the monitors, maps, IO, overall healthiness).

Just make that clear in the function name, avoiding confusion between
multiple 'status-related' functions in the monitor.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

client: don't include ceph.{file,dir}.layout vxattr in listxattr()

This avoids 'cp -a' modifying layout of new files/directories.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

erasure-code: deactivate SSE optimizations

Workaround until the bug is diagnosed and fixed

http://tracker.ceph.com/issues/7826 refs #7826

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1521 from ceph/wip-7810

mds: reset connection priv after connection's session is removed

Reviewed-by: Sage Weil <sage@inktank.com>

msg: set recv stamp for local delivery

Fixes: #7827
Signed-off-by: Sage Weil <sage@inktank.com>

mailmap: Allen Samuels affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Warren Usui is with Inktank

Reviewed-by: Warren Usui <warren.usui@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mds: reset connection priv after connection's session is removed

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge tag 'v0.78'

v0.78

doc/release-notes: 0.78 final notes

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1515 from ceph/wip-rgw-less-alloc

Wip rgw less alloc

Reviewed-by: Sage Weil <sage@inktank.com>

doc/release-notes: final 0.78

Signed-off-by: Sage Weil <sage@inktank.com>

rgw: get rid of a memory allocation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: remove memory allocation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

0.78

Merge pull request #1512 from dachary/wip-jerasure-sse

 erasure-code: gf-complete detects SSE at runtime

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1513 from ceph/wip-7438

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1510 from ceph/wip-pq

osd: some PrioritizedQueue cleanups; dump locking fix; debug pqueue state at high debug level

Reviewed-by: Samuel Just <sam.just@inktank.com>

Revert "ReplicatedPG: disallow trans which atomically create and remove an object"

This reverts commit 9ee108428176c93fe48ae5ae95b160eadabca9b6.

This appears to break rgw in its current state; test and fix soon.

doc/release-notes: 0.78 notes

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/wip-multimds-fixes'

Reviewed-by: Sage Weil <sage@inktank.com>

erasure-code: gf-complete detects SSE at runtime

And the gf-complete code has been updated to rely on the
gf_internal_t::sse flag instead of #if conditionals.

Signed-off-by: Loic Dachary <loic@dachary.org>

autotools: AX_SSE detects the compiler SSE capabilities

For each SSE feature supported by the compiler

* add the corresponding -msse* flag
* define HAVE_SSE*

Remove AX_EXT because it decides based on the CPU capabilities of the
machine compiling the binary which may or may not be the one running
them.

Signed-off-by: Loic Dachary <loic@dachary.org>

test: Add erasure coding to stress watch test

Fixes: #7438
Signed-off-by: David Zafman <david.zafman@inktank.com>

test: Reorg multi_stress_watch to prepare for ec option

If old options are specified maintain backward compatibility

Signed-off-by: David Zafman <david.zafman@inktank.com>

test: Fix ceph_filestore_dump.sh test for new EC pool creation

Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge pull request #1500 from ceph/wip-tid_t

Change tid_t to ceph_tid_t to avoid conflicts with a popular type

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1465 from ceph/wip-s3pm-wusui

Remove code duplication from s3 tests.

Merge pull request #1511 from dachary/wip-exclamation

tests: use ^ instead of ! as invalid char

Reviewed-by: Sage Weil <sage@inktank.com>

tests: use ^ instead of ! as invalid char

Because !!! is incorrectly interpreted as an error by gitbuilder.

Signed-off-by: Loic Dachary <loic@dachary.org>

doc/release-notes: stop confusing sphinx

ERROR: /srv/autobuild-ceph/gitbuilder.git/build/doc/release-notes.rst:594: Unknown target name: "debug".

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1503 from dachary/wip-jerasure-submodules

 erasure-code: add gf-complete / jerasure submodules

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1505 from ceph/wip-7728

Wip 7728

Reviewed-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/firefly'

Merge pull request #1509 from guangyy/wip-folder-merge

Make it possible for FileStore to be in a state which never do merge and keep it backward compatible

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1506 from ceph/wip-7755

PG::find_best_info: fix log_tail component

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1507 from ceph/wip-7733

PG::start_peering_interval: always send_notify if !primary

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1508 from ceph/wip-7777

ReplicatedPG::do_op: delay if snapdir is unreadable

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1312 from cernceph/master

rados.py: add aio_remove

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1504 from ceph/wip-7780

objecter: waive OSDMAP_FULL check for MDS

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1476 from ceph/wip-scrub-lock

osd: do not queue scrub for non-primary

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1502 from ceph/wip-flush-journal-asok

wip-flush-journal-asok

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

objecter: waive OSDMAP_FULL check for MDS

The MDS expects to be able to perform writes to OSDs even
if the full ratio has been reached, in order to journal
file deletions to free space.

Fixes: #7780
Signed-off-by: John Spray <john.spray@inktank.com>

qa: test_alloc_hint: flush journal before prodding the FS

OSDs that for some reason get behind on processing their op queue break
expect_alloc_hint_eq(), as it pokes the FS and not the journal.  Fix it
by flushing the journal before proceeding with anything else.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

osd: add flush_journal admin socket command

Add flush_journal admin socket command to be able to flush journal to
the permanent store for online osds.  (For offline osds we already have
ceph-osd --flush-journal.)

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>

mds: fix typo in MDCache::handle_cache_rejoin_strong

It should invent dirfrags according to the dirfragtree

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

erasure-code: add gf-complete / jerasure submodules

Instead of copying the files in the ceph repository, which is less
convenient.

When building the headers are ignored, even though they do
not exist. When creating the tarbal with make dist, it fails because
they cannot be found. I misread src/gf_int.h to be include/gf_int.h and
wrongfully thought the submodules were to blame. This is why they were
removed shortly after being added.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: remove copy of gf-complete / jerasure

Signed-off-by: Loic Dachary <loic@dachary.org>

mds: fix NULL pointer dereference in Server::handle_client_rename

Inode's parent dentry can be NULL, which causes segment fault in
Locker::acquire_locks

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: avoid infinite loop in MDLog::submit_entry()

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge pull request #1496 from ceph/wip-7708

mds/Locker: fix null deref on cap import

Merge pull request #1499 from ceph/wip-no-version

Add NO_VERSION to avoid rebuilding ceph_ver.h and relinking

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1498 from ceph/wip-limit-libkeyutils

Makefiles: remove libkeyutils from every binary except two

Reviewed-by: Sage Weil <sage@inktank.com>

mds: fix potential invalid pointer dereference

pin importing inode until subtree importing finishes

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: rdlock dirfragtree lock when renaming directory

This make sure inode importer and inode exporter have consistent
view of dirfrags.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: don't mark scatter locks dirty when dirfrag is dirty

The journal reply code has check that decides which scatter locks
should be marked as dirty. So don't unconditionally mark scatter
locks dirty when dirfrag is dirty

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: trim other MDS' mdsdir during recovery

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: remove wrong assertion

when handling resolve message, it's possible that bound dirfrags
are missing.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

fragtree_t: fix get_leaves_under()

If fragtree is (*^1) and the caller wants leaves under frag 00*.
get_leaves_under() should return empty list.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: improve success rate of subtree exporting

When exporting a subtree, the migrator acquires the required locks,
then freezes the subtree and releases the locks. After subtree is
frozen, it try acquiring the same locks again.

This patch make scatter locks keep in their old states if inode has
exporting dirfrag. It improves the chance that migrator acquires all
required locks when subtree is frozen.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: cleanup CInode::decode_import()

We have bumped protocol version several times, no need to maintain
compatibility for ancient message.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>