Merge pull request #3049 from dachary/wip-10211-erasure-code-buffer-alignement

erasure-code: enforce chunk size alignment

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/release-notes: v0.89

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3024 from ceph/wip-librados-symbols

librados: Only public API symbols from the shared library

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3028 from majianpeng/fix4

Add perf reset command.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3074 from ceph/wip-10229

osdc/Filer: use finisher to execute C_Probe and C_PurgeRange

osdc/Filer: use finisher to execute C_Probe and C_PurgeRange

Currently contexts C_Probe/C_PurgeRange are executed while holding
OSDSession::completion_lock. C_Probe and C_PurgeRange may call
Objecter::stat() and Objecter::remove() respectively, which acquire
Objecter::rwlock. This can cause deadlock because there is intermediate
dependency between Objecter::rwlock and OSDSession::completion_lock:

 Objecter::rwlock -> OSDSession::lock -> OSDSession::completion_lock

The fix is exexcute C_Probe/C_PurgeRange in finisher thread.

Fixes: #10229
Signed-off-by: Yan, Zheng <zyan@redhat.com>

mon: fix signed/unsigned comparison warning

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

0.89

Merge pull request #3065 from ceph/wip-cephtool-exec

qa: fix cephtool tests not getting run

Reviewed-by: Loic Dachary <ldachary@redhat.com>

qa: fix teardown in cephtool's test_mon_mds

This was missing from 17b5fc9a but we didn't notice
because the test wasn't being run by the gitbuilders.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3044 from osynge/wip=suse-systemd

better systemd support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3067 from dachary/wip-cephtool

fix cephtools running only part of the tests + avoid killing wrong daemons

Reviewed-by: Sage Weil <sage@redhat.com>

common/perf_counters: Remove the duplicated code.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

test/perf_counters: Replace perfcounters_dump to perf dump.

The func of command  perfcounters_dump and 'perf dump' are the same .
But from the print 'ceph --admin-daemon help', it only print 'perf
dump'. So replace.
In order to keep consistent, still keep perfcounters_dump in code for
old user.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

test/perf_counters: Add test case for cmd 'perf reset'

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/perf_counters: Add 'perf reset all|perfcountername' for 'ceph
--admin-daemon'

Using this command, we can reset all perfcounters or reset on specified
perfcounter.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3068 from ktdreyer/doc-os-recommendations-cleanup

doc: os recommendations cleanup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: rm Fedora minor versions in os recommendations

Fedora doesn't have point releases, so it doesn't make sense to talk
about "Fedora 18.0" or "Fedora 19.0" - it's just "Fedora 18" or "Fedora
19".

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #3061 from ktdreyer/rpm-ceph-disk-man-page

packaging: package ceph-disk(8)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

packaging: package ceph-disk(8)

The ceph-disk man page was added in
a450cab2b8148cb8a9b043d629feccf89e5aabac, but this was not added to the
RPM or DEB packaging. Add it here.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc: rm old releases in os recommendations page

Similar to the change in commit
4cdc9c793627eec13c65543c09c604ec27a4dd8a, remove the references to old
Ceph releases.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc: clarify "B" flag in os recommendations page

We don't exactly do continuous builds on all the platforms marked with
"B", but we have published binary RPMs for them. Adjust the "B"
footnote definition to reflect this.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

qa: cephtool/test.sh use regular strings to list tests

Using the array notation to list test is error prone and more
complicated to write.

It also fixes a bug : only the first test of each series (MON, OSD, MDS)
was run and the others were ignored.

Signed-off-by: Loic Dachary <loic@dachary.org>

tests: vstart_wrapper.sh must call vstart.sh with a list of daemons

The list of daemons is provided in the $CEPH_START environment variable
but vstart_wrapper.sh incorrectly uses the $VSTART_ARGS empty variable.

As a consequence vstart.sh calls init-ceph stop which kills daemons that
do not belong to the test.

http://tracker.ceph.com/issues/10215 Fixes: #10215

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: enforce chunk size alignment

Let say the ErasureCode::encode function is given a 4096 bytes
bufferlist made of a 1249 bytes bufferptr followed by a 2847 bytes
bufferptr, both properly starting on SIMD_ALIGN address. As a result the
second 2048 had to be reallocated when bufferlist::substr_of gets the
second 2048 buffer, the address starts at 799 bytes after the beginning
of the 2847 buffer ptr and is not SIMD_ALIGN'ed.

The ErasureCode::encode must enforce a size alignment based on the chunk
size in addition to the memory alignment required by SIMD operations,
using the bufferlist::rebuild_aligned_size_and_memory function instead of
bufferlist::rebuild_aligned.

http://tracker.ceph.com/issues/10211 Fixes: #10211

Signed-off-by: Loic Dachary <loic@dachary.org>

common: allow size alignment that is not a power of two

Do not assume the alignment is a power of two in the is_n_align_sized()
predicate. When used in the context of erasure code it is common
for chunks to not be powers of two.

Signed-off-by: Loic Dachary <loic@dachary.org>

common: add bufferlist::rebuild_aligned_size_and_memory

The function bufferlist::rebuild_aligned checks memory and size
alignment with the same variable. It is however useful to separate
memory alignment constraints from size alignment constraints. For
instance rebuild_aligned could be called to allocate an erasure coded
buffer where each 2048 bytes chunk needs to start on a memory address
aligned on 32 bytes.

Signed-off-by: Loic Dachary <loic@dachary.org>

doc: Reformatting on rbd replay doc.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added rbd-replay-many and restructured index.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Fixed index syntax.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3051 from ceph/wip-doc-rbd-replay

doc: Document RBD Replay

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3060 from ktdreyer/doc-os-recommendations-centos-7

doc: add CentOS 7 to recommended OSes

Reviewed-by: John Wilkins <jowilkin@redhat.com>

doc: add CentOS 7 to recommended OSes

CentOS 7 was missing from the list of recommended OSes for Firefly. Add
it to the list.

Reported-By: Frank Even <lists+ceph.com@elitists.org>
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #2993 from theanalyst/s3/keystone-date-fixes

rgw: check for timestamp for s3 keystone auth

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3058 from nilamdyuti/wip-doc-ceph-disk

doc: Adds man page for ceph disk in TOC.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Adds man page for ceph disk in TOC.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

Merge pull request #3056 from ceph/wip-9997-1

client: invalidate kernel dentries one by one

Reviewed-by: John Spray <john.spray@redhat.com>

client: invalidate kernel dentries one by one

Our trick to trim the whole kernel dentry tree does not work for 3.18+ kernel.
The fix is trim kernel dentries one by one.

Fixes: #9997
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3052 from ceph/wip-flush-journal-fix

mds: fix race of trimming log segments

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #2994 from dachary/wip-10175-compile-dependencies

script replacement for deps.{rpm,deb}.txt

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Thorsten Behrens <tbehrens@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@redhat.com>

mds: fix race of trimming log segments

trimming old log segments can start new log segments, make sure
MDLog::trim_all() not expire newly started log segments.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3041 from xiaoxichen/cleanup-noop

Cleanup noop in message

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Document RBD Replay

Signed-off-by: Adam Crume <adamcrume@gmail.com>

Merge pull request #3008 from ceph/wip-10135

mon: OSDMonitor: allow adding tiers to FS pools

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3050 from dachary/wip-10213-const

msg: do not const methods that return a mutable pointer

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

msg: do not const methods that return a mutable pointer

Add remove misleading const on post_rx_buffer argument.

http://tracker.ceph.com/issues/10213 Fixes: #10213

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #2825 from ceph/wip-9921

SimpleMessenger: Pipe: do not block on takeover while holding global locks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3046 from yuyuyu101/fix-kinetic

KnieticStore: add necessary header file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2922 from ceph/wip-da-SCA-20141111

Coverity fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2937 from simon3z/master

common: do not unlock rwlock on destruction

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2945 from ceph/wip-9321

mon: PaxosService: can be readable even if proposing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2788 from ceph/wip-warn-max-pg

mon: warn when there are too many pgs per osd

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2735 from ceph/wip-pg-stat

mon: make 'pg stat -f json' not pg dump; make JSON dump of floats floats and not strings

Reviewed-by: Samuel Just <sjust@redhat.com>

documentation: replace deps.{deb,rpm}.txt with scripts

install-deps.sh installs the dependencies based on the content of
debian/control (if apt-get is found) or ceph.spec.in (if yum is
found). The manually maintained list of dependencies for deb based
operating systems (deps.deb.txt) and rpm based operating
systems (deps.rpm.txt) are deprecated.

http://tracker.ceph.com/issues/10175 Fixes: #10175

Signed-off-by: Loic Dachary <ldachary@redhat.com>

KnieticStore: add necessary header file

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

better systemd support

Added ceph.target
Made ceph-mds, ceph-mon, ceph-osd services part of ceph.target
Made ceph-mds, ceph-mon, ceph-osd services require partitions to be available.
Added ceph init script with sysV like behaviour.

Provided by Tim Serong tserong@suse.com and Owen Synge osynge@suse.com

Signed-off-by: Owen Synge <osynge@suse.com>

Cleanup noop in message

The noop loop are dead, so clean it up

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Merge pull request #2855 from ceph/wip-9881

#9881 MDS flush journal asok command

Reviewed-by: Yan, Zheng <zyan@redhat.com>

mds: implement "flush journal" asok

Start a new segment and flush everything older
to the backing store.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: in segment expiry, use end instead of start

Previously we were always leaving the last log
segment un-expired from Journaler's POV, because
we were updating expire_pos to the *start* of the
expired journal segment instead of the *end*.

This was harmless, but cleaning it up makes it
easier to see what's going on during journal flushes
in testing.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: fix MDLog I/O callback without finisher

This could deadlock if another caller was waiting
on a write_head callback while holding mds_lock.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3037 from stiopaa1/cleanup

added some consts, some cleanup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

added consts, cleanup
Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #3019 from theanalyst/doc/typo/ceph-auth

doc: minor fix in ceph-authtool man page

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3023 from ktdreyer/rpm-rbd-replay-prep-conditional

rpm: unconditionally package rbd-replay-prep(8)

Reviewed-by: Adam Crume <adamcrume@gmail.com>

doc: minor fix in ceph-authtool man page

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #3032 from dachary/wip-osd-log-cosmetics

osd: log message content cosmetics

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@redhat.com>

osd: log message content cosmetics

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2769 from ceph/wip-9557

mds: verify backtrace when fetching dirfrag

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #3027 from majianpeng/fix1

Add "make check" dep python-virtualenv in deps.rpm.txt

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Add "make check" dep python-virtualenv in deps.rpm.txt

In fedora20, When 'make  check' and met those error messages:
/src/test/run-cli-tests: virtualenv not installed, skipping python-using
tests.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3025 from majianpeng/fix1

Add isa-erasure yasm dep for deps.rpm.txt

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Add isa-erasure yasm dep for deps.rpm.txt

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rpm: unconditionally package rbd-replay-prep(8)

The build system will conditionally build and install the
rbd-replay-prep utility based on the presence of lttng.

Commit 230c5b8e7f58e2e8be32c3f31c6abcc5bdabc5ca adjusted the RPM
packaging so that we only package and ship the rbd-replay-prep utility
and its man page when we're on a platform that has lttng.

The problem is that this is only half-correct. The build system
unconditionally installs the man page for rbd-replay-prep, regardless of
whether lttng was present or absent, and (by extension,) without regard
for whether we've built and installed the rbd-replay-prep binary or not.

We have two options: either the build system should be adjusted to
conditionally skip installing rbd-replay-prep(8), or else we should just
adjust the RPM packaging to always ship the man page. This commit does
the latter.

Backport: firefly
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

librados: Only public API symbols from the shared library

The librados shared library was previously exporting all
symbols.  librados public API methods are now explicitly
exported and all other symbols are hidden.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3022 from leseb/fix-opts

Minor typos and trailing spaces

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Minor typos and trailing spaces

Fix typos and trailing spaces.

Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

Merge pull request #3016 from yuyuyu101/fix-kinetic-support

KineticStore: Fix compile error and add _test_init

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

KineticStore: Fix compile error and add _test_init

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2995 from nilamdyuti/wip-doc-ceph-disk

doc: Adds man page for ceph-disk utility.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3021 from tchaikov/wip-fix-typos

doc: fix some typos in ObjectStore.h

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: fix some typos in ObjectStore.h

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

doc: Adds the updated man page for ceph-disk utility.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Updates the man page for ceph-disk utility with some changes.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

Merge pull request #3011 from stiopaa1/someConsts

added some consts

Reviewed-by: Loic Dachary <ldachary@redhat.com>

added some consts
Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

mon: OSDMonitor: allow adding tiers to FS pools

This was an overly-strict check.  In fact it is perfectly
fine to set an overlay on a pool that is already in use
as a filesystem data or metadata pool.

Fixes: #10135
Signed-off-by: John Spray <john.spray@redhat.com>

doc: Adds man page for ceph-disk utility.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Removes ceph-deploy usage instances from ceph-disk man page.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

Merge pull request #3003 from dachary/wip-10185-neon

arch: fix neon feature detection

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@redhat.com>

arch: fix neon feature detection

The fread function returns the number of elements read, not the number
of bytes.

Introduced by 1cef8339f7a2a503327c20c014e696c91d36281e , after giant.

Add tests for protection against similar regressions.

http://tracker.ceph.com/issues/10185 Fixes: #10185

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3002 from thesues/fix-init

Remove pidfile and asok after stopping ceph

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Remove pidfile and asok after stopping ceph

Signed-off-by: Dongmao Zhang <deanraccoon@gmail.com>

Merge pull request #2996 from ceph/wip-10151

mon: fix MDS health status from peons

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #2872 from dachary/wip-10018-primary-erasure-code-hinfo

osd: erasure code deep scrub must not abort if hinfo is missing

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #2984 from stiopaa1/includeFix

remove unneeded include file

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Updates man page for ceph-disk utility.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

remove unneeded include file

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #2998 from theanalyst/doc/rgw-s3-bucket-loc

doc: rgw document s3 bucket location features.

Reviewed-by: John Wilkins <jowilkin@redhat.com>