erasure-code: add sanity check to guard against k=1

Add a call to ErasureCode::sanity_check_k for the isa and jerasure
plugins, with associated tests.

http://tracker.ceph.com/issues/10358 Fixes: #10358

Signed-off-by: Loic Dachary <ldachary@redhat.com>

erasure-code: implement ErasureCode::sanity_check_k

Implement the ErasureCode::sanity_check_k helper for plugins
that need to verify k >= 2.

http://tracker.ceph.com/issues/10358 Fixes: #10358

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4834 from liewegas/wip-11461

fix build on arm w/ old kernels

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

Merge remote-tracking branch 'origin/wip-client-fsync'

Conflicts:
src/client/Client.cc

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4831 from ceph/wip-11807

qa: fix multiple_rsync.sh to avoid using /usr/ directly

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4743 from ceph/wip-client-oldest-tid

Wip client oldest tid

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4723 from ceph/wip-snap-misc

Wip snap misc

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4702 from tchaikov/wip-fix-11590

mon: always reply mdsbeacon

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4602 from ceph/wip-11504

mon: refine check_remove_tier checks

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4781 from ceph/wip-librbd-perf-counters

librbd: re-add missing discard perf counters

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4870 from joaquimrocha/extra_doc_info_for_debug_option

build: Add info about tests to --with-debug option

Reviewed-by: Loic Dachary <ldachary@redhat.com>

build: Add info about tests to --with-debug option

The configure's --with-debug option builds also some of the tests
so this should be mentioned in its doc string in order to let users
know how to build all the tests.

Signed-off-by: Joaquim Rocha <joaquim.rocha@cern.ch>

Merge pull request #4866 from ddiss/fix_test_ceph_disk_dmcrypt_uuid_only

tests: fix test_activate_dmcrypt uuid usage

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: fix test_activate_dmcrypt uuid usage

4601e10800a63cf0e03108e1da0bf11c19c33e26 introduced a regression in that
an empty (uninitialised) OSD uuid is passed to test_pool_read_write for
IO. As a result, the "rados put" request times out causing test failure.

This change ensures that a correct OSD uuid is passed to
test_pool_read_write.

Signed-off-by: David Disseldorp <ddiss@suse.de>

Merge pull request #4528 from ceph/wip-librbd-helgrind

librbd: correct issues discovered via lockdep / helgrind

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4857 from jdurgin/wip-rbd-features-constants

update rbd features constants for python usage

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: flush pending ops while not holding lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: fix possible deadlock in librbd ImageWatcher tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: enable lockdep for librbd unit tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: owner_lock should be held during flush request

Flush might result in the cache writing out dirty objects, which
would require that the owner_lock be held.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc: ObjectCacher flusher might needs additional locks

librbd requires the ObjectCacher flusher thread to acquire
an additional lock in order to maintain lock ordering
constraints.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix recursive locking issues

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify state machine handling of exclusive lock

It is expected that all IO is flushed and all async ops are cancelled
prior to releasing the exclusive lock.  Therefore, replace handling of
lost exclusive locks in state machines with an assertion.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: AsyncObjectThrottle should always hold owner_lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: execute flush completion outside of cache_lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add AsyncRequest task enqueue helper method

In order to support the invariant that all state machine
callbacks occur without holding locks, transitions that
don't always involve a librados call should queue their
callback.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: disable lockdep on AioCompletion

It is only used by clients and it causes a large slowdown
in performance due to the rate at which the lock is constructed/
destructed for each IO request.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: AioCompletion shouldn't hold its lock during callback

The callback routine most likely will attempt to retrieve the result
code, which will result in a recursive lock attempt.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: complete cache read in a new thread context

The ObjectCacher complete the read callback while still holding
the cache lock.  This introduces lock ordering issues which are
resolved by queuing the completion to execute in a clean (unlocked)
context.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: give locks unique names to prevent false lockdep failures

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

log: fix helgrind warnings regarding possible data race

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: fix helgrind warnings

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: add support for flushing watches

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: lockdep now support unregistering once destructed

librbd use of an image hierarchy resulted in lock names being
re-used and incorrectly analyzed.  librbd now uses unique lock
names per instance, but to prevent an unbounded growth of
tracked locks, we now remove lock tracking once a lock is
destructed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: add valgrind.h convenience wrapper

Conditionally support helgrind annotations if valgrind support is
enabled during the build.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add work queue for op completions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

WorkQueue: ContextWQ can now accept a return code

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/rbd.py: update with new features constants

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

librbd: describe the purpose of features constants

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

librbd: add a constant for single-client-only features

This way it's easy to disable them if you want to access an image from
multiple clients.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4717 from athanatos/wip-8635

Wip 8635 -- Move scrub, snap_trim into a unified queue

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4718 from athanatos/wip-sync-omap

FileStore: sync object_map when we syncfs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4839 from ceph/wip-fix-arm-linking-issue

common/Cycles.cc: skip initialization if rdtsc is not implemented

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4852 from ceph/wip-rgw-vstart

make vstart + rgw more useful

vstart.sh: set rgw dns name = localhost

So we can test buckets in the URL by adding entries in /etc/hosts

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4858 from SUSE/wip_ceph.spec.in_remove_extra_else_statement

ceph.spec.in: remove extra %else statement

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4859 from ceph/wip-fix-i386-test-11870

test: fix the plugin_exists()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test: fix the plugin_exists()

in 8e61a99, we try to init the plugin when user set a profile using
"ceph osd erasure-code-profile set <profile-name> <profile>", so
if user passes in an unknown plugin, ceph will reject it right away,
instead of loading the plugin when "ceph osd crush rule create-erasure".

Fixes: #11870
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4856 from ceph/wip-install-deps-11869

install-deps.sh: create virtualenv only if necessary

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph.spec.in: remove extra %else statement

Remove extra conditional.

Signed-off-by: Owen Synge <osynge@suse.com>

install-deps.sh: s/get_pip_and_wheel/populate_wheelhouse/

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: create virtualenv only if necessary

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: do not create "wheelhouse" unless it's ready

* only create "wheelhouse" directory for tox when it is populated
  with all packages in *requiresments.txt. before than, the wheels
  are put in "wheelhouse-wip"

Fixes: #11869
Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Cycles.cc: skip initialization if rdtsc is not implemented

The Cycles initialization gets stuck in infinite loop if rdtsc is not
implemented. This patch fixes the issue by quitting the initialization
if rtdsc fails.

The patch was cherry-picked from ubuntu patch by James Page, see

https://bugzilla.redhat.com/show_bug.cgi?id=1222286

for more details on the patch.

Signed-off-by: James Page <james.page@ubuntu.com>

install-deps.sh: refactor pip install a little bit

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4855 from tchaikov/wip-man-page-fix

man/create-create-keys: minor fixes

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

man/create-create-keys: minor fixes

to address the comments on https://github.com/ceph/ceph/pull/4846

Fixes: #10725
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4853 from ceph/wip-rpm-rm-endif

ceph.spec.in: fix rpm build errors

Reviewed-by: Kefu Chai <kchai@redhat.com>

Revert "ceph.spec.in:ownership of dirs extension"

This reverts commit 6f495974a8255ec4bbd4c7c85f1ed21b3f281f2b.

This needs more discussion, since "make install" does not currently
create /usr/lib/ceph (and maybe /usr/share/ceph?)

This fixes the rpmbuild failure,

  error: File not found:
  /srv/autobuild-ceph/gitbuilder.git/build/rpmbuild/BUILDROOT/ceph-9.0.0-1225.ga16afdd.el6.x86_64/usr/lib/ceph

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec.in: rm stray %endif

Commit d8abde3338b0c7df373b762e35099ad5123866bf introduced a stray
%endif. Remove it.

This fixes the build error:

  "error: /srv/autobuild-ceph/gitbuilder.git/build/ceph.spec:140: Got a
  %endif with no %if"

Reported-by: Greg Farnum <gfarnum@redhat.com>
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

vstart.sh: sudo radosgw for low port numbers

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: show s3 creds at end

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4793 from SUSE/wip_ceph_spec_directories_not_owned_by_a_package

ceph.spec.in:ownership of dirs extension

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4821 from SUSE/wip_libcephfs_jni_postun

ceph.spec.in libcephfs_jni1 has no %post and %postun

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4725 from SUSE/wip_suse_forward_conditionals

Add rpm conditionals

Merge pull request #4808 from liewegas/wip-make-dist

make_dist_tarball.sh

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4850 from ktdreyer/wip-readme-simplify-deps

README: simplify build deps section

README: simplify build deps section

The install-deps.sh step is the same for Debian and RPM-based systems.
Simplify the two README sections into a single section.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4828 from liewegas/wip-11798

make upstart less aggressive with respawn (11798)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4844 from ceph/wip-11860

rgw: Do not enclose the Bucket header in quotes

Reviewed-by: Sage Weil <sage@redhat.com>

upstart: limit respawn to 3 in 30 mins (instead of 5 in 30s)

It may take tens of seconds to restart each time, so 5 in 30s does not stop
the crash on startup respawn loop in many cases.  In particular, we'd like
to catch the case where the internal heartbeats fail.

This should be enough for all but the most sluggish of OSDs and capture
many cases of failure shortly after startup.

Fixes: #11798
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Do not enclose the Bucket header in quotes

Fixes: #11860
Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #4836 from ceph/wip-11857

admin/build-doc: fix dependency checks

Reviewed-by: Nathan Cutler <ncutler@suse.cz>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4818 from dachary/wip-11272-ceph-disk-zap

ceph-disk: always check zap is applied on a full device

Reviewed-by: Travis Rhoden <trhoden@redhat.com>

Merge pull request #4842 from liewegas/wip-install-deps

make install-deps clean up

Reviewed-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: clean up after ourselves

Signed-off-by: Sage Weil <sage@redhat.com>

Add rpm conditionals : libs_compat

Extracted conditionals from SUSE rpm spec file to forward port.
Original work done by Thorsten Behrens <tbehrens@suse.com>

Signed-off-by: Owen Synge <osynge@suse.com>

Add rpm conditionals : tcmalloc

Extracted conditionals from SUSE rpm spec file to forward port.
Original work done by Thorsten Behrens <tbehrens@suse.com>

Signed-off-by: Owen Synge <osynge@suse.com>

Add rpm conditionals : tests

Extracted conditionals from SUSE rpm spec file to forward port.
Original work done by Thorsten Behrens <tbehrens@suse.com>

Signed-off-by: Owen Synge <osynge@suse.com>

Add rpm conditionals : cephfs_java

Extracted conditionals from SUSE rpm spec file to forward port.
Original work done by Thorsten Behrens <tbehrens@suse.com>

Signed-off-by: Owen Synge <osynge@suse.com>

admin/build-doc: fix dependency checks

Fixes: #11857
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4712 from dachary/wip-11663-verify-erasure-code-profile

normalize and verify the erasure code profile

Reviewed-by: Andreas Peters <andreas.joachim.peters@cern.ch>

Merge pull request #4590 from ceph/wip-da-SCA-20150427

SCA and Coverity fixes

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4823 from liewegas/wip-ceph-post-file

ceph-post-file: behave when sftp doesn't take -i

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4816 from dachary/wip-11624-cephtools-test

tests: cephtool/test.sh wait_for_clean after PG (re)mapping

Reviewed-by: David Zafman <dzafman@redhat.com>

arm: fix build on older kernels

that don't define HWCAP_CRC32

Fixes: #11461
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4827 from ceph/wip-librbd-aio-submission

librbd: new config option for legacy blocking aio behavior

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: fix multiple_rsync.sh to avoid using /usr/ directly

While we're at it, take only /usr/lib instead of all of /usr
to keep the overall file count more modest.

Fixes: #11807
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4701 from jdurgin/wip-nova-rbd-cache

doc: add rbd cache settings to nova.conf

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

tests: verify librbd blocking aio code path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add rbd_non_blocking_aio option to image metadata

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4643 from tchaikov/wip-fix-11569

fix "pg ls" with states of "recovering" and/or "repair"

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4760 from ketor/fix_mutex_memory_leak

Mutex: fix memory leak in lockdep _register

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4802 from isserrano/master

doc: Fixed example sintax in Absolute Sizing

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: s/osd pool hot-storage/osd pool set hot-storage/

Signed-off-by: Ismael Serrano <ismael.serrano@gmail.com>

Put new variable initialization in its own block

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

librbd: new rbd_non_blocking_aio config option

Setting this option to false reverts librbd to legacy behavior
where AIO operations could potentially block.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Makefile: install ceph-post-file keys with mode 600

Otherwise ssh (may) prompt for a password.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-post-file: improve check for a source install

Signed-off-by: Joseph McDonald <joseph.mcdonald@alcatel-lucent.com>

ceph-post-file: behave when sftp doesn't take -i

Fixes: #11836
Signed-off-by: Joseph McDonald <joseph.mcdonald@alcatel-lucent.com>