msgr: add KEEPALIVE2 feature

This is similar to KEEPALIVE, except a timestamp is also exchanged.  It is
sent with the KEEPALIVE, and then returned with the ACK.  The last
received stamp is stored in the Connection so that it can be queried for
liveness.  Since all of the users of keepalive are already regularly
triggering a keepalive, they can check the liveness at the same time.

See #7888.

Signed-off-by: Sage Weil <sage@inktank.com>

Pipe: rename keepalive->send_keepalive

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge branch 'wip-7875'

Reviewed-by: Samuel Just <sam.just@inktank.com>

mon/OSDMonitor: require OSD_CACHEPOOL feature before using tiering features

The OSDs need to support this feature before we allow users to turn it
on.  This is similar to what the erasure pool support does.

Signed-off-by: Sage Weil <sage@inktank.com>

mon/OSDMonitor: prevent setting hit_set unless all OSDs support it

We are using OSD_CACHEPOOL as a proxy for the support for the tiering
OSDMap infrastructure.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1546 from ceph/wip-fix-pools

fix pool ops test

qa/workunits/mon/pool_ops.sh: fix test

The pool create command doesn't take k/v pairs any more.

Signed-off-by: Sage Weil <sage@inktank.com>

qa/workunits/mon/pool_ops.sh: use expect_false

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1545 from ceph/wip-7849-b

ceph-conf: do not log

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

ceph-conf: no admin_socket

We don't need to worry about pidfile because that is done by the fork
functions, which ceph-conf doesn't call.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1522 from themgt/patch-1

document adding dev key for custom Apache/FCGI install

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

jerasure: fix up .gitignore

Signed-off-by: Sage Weil <sage@inktank.com>

ceph-conf: do not log

If you are querying the conf for an osd and it has a log configured, we
should not generate any log activity.

This isn't super pretty, but it is much less intrusive that wiring a 'do
not log' flag down into CephContext and a zillion other places.

Fixes: #7849
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1542 from onlyjob/debian

logrotate: do not rotate empty logs (2nd logrotate file)

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1544 from ceph/wip-7876

rgw: use s->content_length instead of s->length

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1534 from dachary/wip-sse-fix

erasure code sse optimized jerasure plugin

Reviewed-by: Sage Weil <sage@inktank.com>

rgw: use s->content_length instead of s->length

Fixes: #7876
Need to use the actual content length, not the pointer to the string.
This was probably working because there's correlation to when
content_length > 0 to whether s->length is not null.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1540 from ceph/wip-7860

test: Wait for tier removal before next test starts

Reviewed-by: Sage Weil <sage@inktank.com>

logrotate: do not rotate empty logs (2nd logrotate file)

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>

Merge pull request #1541 from onlyjob/debian

logrotate improvement: do not rotate empty logs

Reviewed-by: Sage Weil <sage@inktank.com>

erasure-code: test encode/decode of SSE optimized jerasure plugins

If the machine running make check has the required CPU features
available, load the SSE optimized plugin and check that it can encode /
decode a simple payload. If the CPU features are not available, only
test the generic plugin and display an informative message about the
tests that were skipped.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: test jerasure SSE optimized plugins selection

Test the selection of the plugin depending on the CPU features. The
prefix of the plugin is "jerasure" by default (jerasure_generic,
jerasure_sse3, jerasure_sse4) and can be modified with the
"jerasure-name" parameter. A test plugin is created for each
variant (test_jerasure_generic, test_jerasure_sse3, test_jerasure_sse4).
The flags set by ceph_probe are modified by the test to check if the
expected plugin suffix is appended.

Signed-off-by: Loic Dachary <loic@dachary.org>

osd: increase osd verbosity during functional tests

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: SSE optimized jerasure plugins

The jerasure plugin is compiled with three sets of flags:

* jerasure_generic with no SSE optimization
* jerasure_sse3 with SSE2, SSE3 and SSSE3 optimizations
* jerasure_sse4 with SSE2, SSE3, SSSE3, SSE41, SSE42 and PCLMUL optimizations

The jerasure plugin loads the appropriate plugin depending on the CPU
features detected at runtime.

http://tracker.ceph.com/issues/7826 fixes #7826

Signed-off-by: Loic Dachary <loic@dachary.org>

arch: add SSE3, SSSE3, SSSE41 and PCLMUL intel features

And add a note about valgrind forcing a fake cpuid.

Signed-off-by: Loic Dachary <loic@dachary.org>

autotools: intel cpu features detection

Rename SIMD to INTEL for clarity.

Instead of agregating all flags in INTEL_FLAGS, create individual flags
for each feature (INTEL_SSE2_FLAGS etc.) for finer control in the
makefiles.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: gf-complete / jerasure modules updates

To avoid confusion, the jerasure v1 branch that contains commits pending
review upstream is named v2-ceph and the gf-complete v2 branch is named
v2-ceph.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: allow loading a plugin from factory()

The Mutex scope is restricted to only protect the load() method and not
the factory() method. This allows a plugin to load another plugin from
within the factory() method.

Reviewed-By: Christophe Courtaut <christophe.courtaut@gmail.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge remote-tracking branch 'gh/firefly'

logrotate improvement: do not rotate empty logs

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>

osd: do not make pg_pool_t incompat when hit_sets are enabled

If we enable HitSet tracking, the OSD needs to know this, but clients do
not care.  Setting the compat version is too heavyweight as it locks out
older kernels (*any* currents, currently) who are unaffected by the new
fields.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1537 from ceph/wip-7871

RadosModel: allow --no-omap to be specified seperately from --ec-pool

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1536 from ceph/wip-7870

ReplicatedPG::do_osd_ops: only return ENOTSUP on OMAP write ops

Reviewed-by: Sage Weil <sage@inktank.com>

test: Wait for tier removal before next test starts

Fixes: #7860
Signed-off-by: David Zafman <david.zafman@inktank.com>

rgw: configurable chunk size

Fixes: #7589
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1535 from ceph/wip-7823

osd: trim copy-get backend read to object size

Reviewed-by: Samuel Just <sam.just@inktank.com>

RadosModel: allow --no-omap to be specified seperately from --ec-pool

Fixes: #7871
Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::do_osd_ops: only return ENOTSUP on OMAP write ops

This way, redirected omap reads will work correctly by returning
empty maps and buffers.

Fixes: #7870
Signed-off-by: Samuel Just <sam.just@inktank.com>

osd: trim copy-get backend read to object size

We are passing a big number to the backend to read and it is trimming it
to the stripe boundary, and then setting the cursor at a slightly smaller
offset bound by oi.size.  This is invalid, and will trigger an assert in
the _write_copy_chunk code:

     0> 2014-03-21 15:12:23.761509 7f8dd2324700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::_write_copy_chunk(ReplicatedPG::CopyOpRef, PGBackend::PGTransaction*)' thread 7f8dd2324700 time 2014-03-21 15:12:23.758866
osd/ReplicatedPG.cc: 5554: FAILED assert(cop->data.length() + cop->temp_cursor.data_offset == cop->cursor.data_offset)

To fix this, trim the buffer to the correct length in the completion
context.

Fixes: #7823
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1518 from dachary/wip-mailmap

mailmap updates

Reviewed-By: Christophe Courtaut <christophe.courtaut@gmail.com>

Merge remote-tracking branch 'gh/firefly'

osd: fix tests due to no default erasure-code ruleset

Signed-off-by: Sage Weil <sage@inktank.com>

Revert "osd: create the erasure-code ruleset in OSDMap::build_simple"

This reverts commit b85330eb1d62632a1c3f2025b1f309816d54bd5a.

We can't have this in the OSDMap by default or else all clients will require
the new features.

[Fixed up #include for strmap]

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1530 from onlyjob/fix-defaultweight

fix bug in 'defaultweight' calculation on OSD start.

Reviewed-by: Sage Weil <sage@inktank.com>

fix bug in 'defaultweight' calculation on OSD start.

  awk: cmd. line:1: { d=$2/1073741824 ; r = sprintf(\"%.2f\", d); print r }
  awk: cmd. line:1:                                 ^ backslash not last character on line
  awk: cmd. line:1: { d=$2/1073741824 ; r = sprintf(\"%.2f\", d); print r }
  awk: cmd. line:1:                                 ^ syntax error

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>

Merge branch 'wip-7676-firefly' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1533 from ceph/wip-7863-wusui

Make sure s3_utilities are found.

Make sure s3_utilities are found.

Add FindBin code to rgw/s3 tests to include s3_utilites.pm properly.

Fixes: 7863
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #1527 from ceph/wip-6465

be nitpicky about missing references

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1531 from onlyjob/debian

minor corrections from Debian

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1363 from ceph/wip-3863

Add ceph-client-debug inode lookup

Reviewed-by: Greg Farnum <greg@inktank.com>

minor corrections to package descriptions

minor init.d scripts lintianisation:

 * init.d-script-does-not-source-init-functions
 * init.d-script-does-not-implement-required-option
 * init.d-script-missing-lsb-description

rgw: don't modify rgw_bucket for data extra pool

This cleans up the interface, as we don't have this ugly implicit
structure modification when we deal with object that resides on the data
extra pool. Replcae lot's of calls to get_obj_bucket_and_oid_key() to a
cleaner get_obj_ref() that uses a struct to hold the updated fields, and
update the relevant callers.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: multipart meta object uses extra data pool

Fixes: #7676
We need the extra data pool since EC backends cannot handle the omap
operations that are needed for the multipart meta object.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: zone placement info includes extra data pool

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: add data_extra pool to bucket

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1528 from ceph/wip-dz-watch-test

Reviewed-by: Samuel Just <sam.just@inktank.com>

mailmap: Inktank jenkins bot

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Huang Jun name normalization

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Tyler Brekke name normalization

Reviewed-by: Tyler Brekke <tyler.brekke@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Guang Yang affiliation

and name normalization

Reviewed-by: Guang Yang <yguang@yahoo-inc.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: James Ryan Cresawn affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Mohammad Salehe affiliation

Reviewed-by: Mohammad Salehe <salehe+dev@gmail.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Sharif Olorin affiliation

Reviewed-by: Sharif Olorin <sio@tesser.org>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Stephan Renatus affiliation

Reviewed-by: Stephan Renatus <s.renatus@x-ion.de>
Signed-off-by: Loic Dachary <loic@dachary.org>

qa: Add ceph_multi_stress_watch for rep and ec

Fixes: #7438
Signed-off-by: David Zafman <david.zafman@inktank.com>

Merge pull request #1525 from ceph/wip-listxattr

client: don't include ceph.{file,dir}.layout vxattr in listxattr()

Reviwed-by: Sage Weil <sage@inktank.com>

ensure pybind is in the PYTHONPATH

Signed-off-by: Alfredo Deza <alfredo@deza.pe>

Merge pull request #1523 from ceph/wip-7827

msg: set recv stamp for local delivery

Reviewed-by: Greg Farnum <greg@inktank.com>

be nitpicky about missing references

Signed-off-by: Alfredo Deza <alfredo@deza.pe>

Merge pull request #1526 from ceph/wip-status-function-names

mon: Monitor: change 'status' function names for clarity's sake

Reviewed-by: Sage Weil <sage@inktank.com>

Revert "erasure-code: gf-complete detects SSE at runtime"

This reverts commit dbcf4471838fbf5dce2ee06fc73493f513dbf9f0.

Merge pull request #1524 from dachary/wip-sse

erasure-code: deactivate SSE optimizations

mon: Monitor: s/_mon_status()/get_mon_status()/

'Monitor::_mon_status()' provides status specific to the monitor being
poked.  This is information that does not necessarily relate with
cluster status, and can even be obtained when there's no quorum (via the
admin socket).

We change the function name to make the distinction between mon-specific
status and cluster-status, which is obtained via a different function.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon: Monitor: s/get_status()/get_cluster_status()/

Monitor::get_status() is actually used to obtain 'cluster status' (i.e.,
status about *all* the monitors, maps, IO, overall healthiness).

Just make that clear in the function name, avoiding confusion between
multiple 'status-related' functions in the monitor.

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

client: don't include ceph.{file,dir}.layout vxattr in listxattr()

This avoids 'cp -a' modifying layout of new files/directories.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

erasure-code: deactivate SSE optimizations

Workaround until the bug is diagnosed and fixed

http://tracker.ceph.com/issues/7826 refs #7826

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1521 from ceph/wip-7810

mds: reset connection priv after connection's session is removed

Reviewed-by: Sage Weil <sage@inktank.com>

msg: set recv stamp for local delivery

Fixes: #7827
Signed-off-by: Sage Weil <sage@inktank.com>

mailmap: Allen Samuels affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Warren Usui is with Inktank

Reviewed-by: Warren Usui <warren.usui@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

document adding dev key for custom Apache/FCGI install

following the docs will only get you the release key and the install of custom Apache/FCGI fails w/ key error

mds: reset connection priv after connection's session is removed

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

Merge tag 'v0.78'

v0.78

doc/release-notes: 0.78 final notes

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1515 from ceph/wip-rgw-less-alloc

Wip rgw less alloc

Reviewed-by: Sage Weil <sage@inktank.com>

doc/release-notes: final 0.78

Signed-off-by: Sage Weil <sage@inktank.com>

rgw: get rid of a memory allocation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: remove memory allocation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

0.78

Merge pull request #1512 from dachary/wip-jerasure-sse

 erasure-code: gf-complete detects SSE at runtime

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1513 from ceph/wip-7438

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #1510 from ceph/wip-pq

osd: some PrioritizedQueue cleanups; dump locking fix; debug pqueue state at high debug level

Reviewed-by: Samuel Just <sam.just@inktank.com>

Revert "ReplicatedPG: disallow trans which atomically create and remove an object"

This reverts commit 9ee108428176c93fe48ae5ae95b160eadabca9b6.

This appears to break rgw in its current state; test and fix soon.

doc/release-notes: 0.78 notes

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'gh/wip-multimds-fixes'

Reviewed-by: Sage Weil <sage@inktank.com>

erasure-code: gf-complete detects SSE at runtime

And the gf-complete code has been updated to rely on the
gf_internal_t::sse flag instead of #if conditionals.

Signed-off-by: Loic Dachary <loic@dachary.org>

autotools: AX_SSE detects the compiler SSE capabilities

For each SSE feature supported by the compiler

* add the corresponding -msse* flag
* define HAVE_SSE*

Remove AX_EXT because it decides based on the CPU capabilities of the
machine compiling the binary which may or may not be the one running
them.

Signed-off-by: Loic Dachary <loic@dachary.org>

test: Add erasure coding to stress watch test

Fixes: #7438
Signed-off-by: David Zafman <david.zafman@inktank.com>