qa: use "sudo cp" in multiple_rsync.sh

VirtualBox has some files with weird
permissions in its /usr/lib, which was
tripping up this usually-safe operation
when run as an unprivileged user.

Fixes: #11959
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4921 from dachary/wip-11952-test-ceph-disk

tests: ceph-disk tests need to install pip > 6.1

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4863 from ceph/wip-9964-nosharding

rados import/export (minus pgls sharding)

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

test: Add --dry-run and --no-overwrite testing of rados import

Signed-off-by: David Zafman <dzafman@redhat.com>

tools/rados: dry-run/no-overwrite args for import

These were supported by the old objectstoretool
hook into RadosImport: wire them up here too.

Signed-off-by: John Spray <john.spray@redhat.com>

tools: better error message in objectstoretool

...for the case where someone exports a pool
with "rados export" and then tries to import
it into a PG.

Signed-off-by: John Spray <john.spray@redhat.com>

test: Switch ceph-objectstore-tool import-rados to rados import

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: remove objectstoretool's rados-import

Same functionality now exposed as "rados import".

This removes objectstoretool's librados dependency.

Signed-off-by: John Spray <john.spray@redhat.com>

tools: remove old rados_sync code

This is the code that used to sit behind the
old import/export commands that have been
disabled for some time.

Replaced by the new import/export commands.

Signed-off-by: John Spray <john.spray@redhat.com>

tools: implement rados import/export

New PoolDump (export) class to dump pool contents
in format compatible with ceph-objectstore-tool,
and wire up RadosImport class (import).

Signed-off-by: John Spray <john.spray@redhat.com>

tools: refactor RadosImport to pass ioctx

Callers from rados.cc will already have an ioctx,
whereas objectstoretool passes in a pool name.

Signed-off-by: John Spray <john.spray@redhat.com>

tools: remove RadosDump::debug in favour of dout

This attribute was a hangover from objectstore-tool

Signed-off-by: John Spray <john.spray@redhat.com>

tools: add support for pool exports in RadosImport

Signed-off-by: John Spray <john.spray@redhat.com>

tools: clean up errors in ceph-objectstore-tool

We use negative error codes everywhere else, should
use them here too.  Also add a couple of strategically
placed prints.

Signed-off-by: John Spray <john.spray@redhat.com>

tools: refactor objectstore tool

...to expose the definitions of the object dump
format (RadosDump) for use in rados import/export.

The serialized->RADOS code is now RadosImport, and
the serialized->ObjectStore code is ObjectStoreTool.

This is a step toward #9964, which should use the
same on serialization format as the existing
objectstore-tool dumps.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4922 from dachary/wip-releases

doc: add v9.0.1 to the release timeline

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add v9.0.1 to the release timeline

Signed-off-by: Loic Dachary <ldachary@redhat.com>

doc/release-notes: v0.94.2

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: v9.0.1

Signed-off-by: Sage Weil <sage@redhat.com>

tests: ceph-disk tests need to install pip > 6.1

Otherwise it will not be able to use the wheelhouse.

http://tracker.ceph.com/issues/11952 Fixes: #11952

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #4915 from ceph/wip-cot-list

tools: For ec pools list objects in all shards if the pgid doesn't sp…

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #4910 from dachary/wip-11932-non-regression-typo

tests: ceph_erasure_code_non_regression s/stipe/stripe/

Merge pull request #4873 from liewegas/wip-rgw-civetweb-port

rgw: strip off port from HTTP_HOST when determining bucket from URL

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4838 from ceph/wip-10950

#10950: Fix confusing errors using "mds rm"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4835 from ceph/wip-11746

tools: chunk reads in Dumper

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4825 from tchaikov/wip-drop-ignored-mdsbeacon

mon: drop ignored mdsbeacon

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

tests: erasure-code non regression tests must skip isa on i386

When the isa plugin is not present, the isa plugin payloads are not
tested. When that happens, remember that it was intentional by storing
the path that would have been tested so that the inventory is complete.

http://tracker.ceph.com/issues/11949 Fixes: #11949

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #4707 from stiopaa1/IosFwdNew

Removed unnecessary inclusion of iostream in several files

Reviewed-by: Kefu Chai <kchai@redhat.com>

tools: For ec pools list objects in all shards if the pgid doesn't specify

Signed-off-by: David Zafman <dzafman@redhat.com>

rgw: remove trailing :port from host for purposes of subdomain matching

Some clients (ahem, CrossFTP) include the :port in the HTTP_HOST header.
Strip it out.

Switch req_info field to a std::string and avoid copying it in preprocess.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4775 from yuyuyu101/wip-async-crc-decouple

AsyncMessenger: Make send/receive logic independent

Reviewed-by: Sage Weil <sage@redhat.com>

tests: ceph_erasure_code_non_regression s/stipe/stripe/

Synchronize withe the ceph-erasure-code-corpus submodule in which all
file names were modified to fix the typo.

http://tracker.ceph.com/issues/11932 Fixes: #11932

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4815 from dachary/wip-9720-erasure-code-corpus-variants

tests: sync ceph-erasure-code-corpus to verify jerasure variants

Reviewed-by: Andreas Peters <andreas.joachim.peters@cern.ch>

Merge pull request #4908 from dachary/wip-11931-parallel-tests

tests: automake 1.11 needs parallel-tests

Reviewed-by: Nathan Cutler <ncutler@suse.com>

tests: automake 1.11 needs parallel-tests

Prior to automake 1.13 the default behavior was serial-tests meaning
tests from make check were run one after the other and their output sent
to stdout/stderr. From automake 1.13 up the default became
parallel-tests which logs the output of each individual test in a
separate .log file and allows them to run in parallel.

http://lists.gnu.org/archive/html/automake/2012-12/msg00038.html

Enable parallel-tests so that tests can always run in parallel, even
with automake 1.11 which is the version found in CentOS 6.5 and Ubuntu
12.04 precise.

A nice side effect of always having the logs in separate files is that
tests do not need to worry about being too verbose because their output
will only be read for diagnostic purposes.

http://tracker.ceph.com/issues/11931 Fixes: #11931
http://tracker.ceph.com/issues/11906 Fixes: #11906

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Removed unnecessary inclusion of iostream

In several files the iostream wasn't being used, so it got removed.
In other files the iostream inclusion was replaced by including iosfwd
(for forward declarations), which is much smaller header than iostream,
  so in theory should reduce compilation time.
To make this work some of the functions must have been moved from .h to .cc file.
3 functions also needed to have inline removed - this shouldn't affect
performance in any way: two of them are
probably too long to have been inlined anyway and the third one is for
error reporting, so probably won't be called too often.

test/Makefile-client.am: added linker libs

This was required to avoid linker error when linking
src/test/cls_rbd/test_cls_rbd.cc file. Makefile was specyfing
libcommon.a as a part of a linker command even though this wasn't
required and wasn't being linked against. When inline functions from
buffer.h were moved to buffer.cc(and inline was removed) the
libcommon.a library became necessary. This wouldn't link without also
including additional libraries(CRYPTO_LIBS and EXTRA_LIBS)

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #4879 from SUSE/wip-master-specfile-cleanup

ceph.spec.in: remove duplicate BuildRequires

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4784 from ceph/wip-10794

ceph-objectstore-tool improvements

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #4903 from theanalyst/doc/arch-watch-notify

doc: architecture minor fixes in watch notify

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: architecture minor fixes in watch notify

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

ceph.spec.in: remove duplicate BuildRequires

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #4880 from SUSE/wip-master-specfile-cleanup2

ceph.spec.in: move specific BuildRequires to where they belong

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4893 from dachary/wip-11272-ceph-disk-false-negative

tests: ceph-disk.sh test zap gitbuilder false negative

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4888 from dachary/wip-11905-test-isa

tests: skip isa tests if the plugin is not available

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: ceph-disk.sh test zap gitbuilder false negative

When looking for an error message and testing ceph disk zap, use grep -q
to not display the error message that will be mistaken for a real error
by the gitbuilder parser.

http://tracker.ceph.com/issues/11272 Refs: #11272

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: skip isa tests if the plugin is not available

http://tracker.ceph.com/issues/11905 Fixes: #11905

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: use erasure_code_plugin_exists from ceph-helpers.sh

Instead of the local plugin_exists helper that does the same.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: implement erasure_code_plugin_exists in ceph-helpers.sh

Return 0 if the erasure code *plugin* is available, 1 otherwise.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4713 from dachary/wip-10358-erasure-code-k-1-m-1

ceph osd erasure-code-profile set myprofile k=1 should fail

Reviewed-by: Andreas Peters <andreas.joachim.peters@cern.ch>

tests: sync ceph-erasure-code-corpus to verify jerasure variants

http://tracker.ceph.com/issues/9720 Refs: #9720

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: add --{show_,}path to ceph_erasure_code_non_regression

The directory in which the payload is stored is created from the plugin
parameters. The --show-path shows the directory and exits. This
directory can then be used with --path to override the path created from
the plugin parameters. This is useful to verifying that the jerasure
variant optimized with AVX, SSE4 etc. instructions can be used on the
same payload and show no difference at all. In this case the directory
used by the default jerasure variant is used for each variant and the
parameter that sets the variant to use ( --parameters jerasure-variant )
must not be taken into account to figure out the location of the
payload.

http://tracker.ceph.com/issues/9720 Refs: #9720

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4811 from dachary/wip-11346-isa

ceph-erasure-code-corpus: isa: add k=10 m=4

Reviewed-by: Yuan Zhou <yuan.zhou@intel.com>

erasure-code: add sanity check to guard against k=1

Add a call to ErasureCode::sanity_check_k for the isa and jerasure
plugins, with associated tests.

http://tracker.ceph.com/issues/10358 Fixes: #10358

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4882 from SUSE/wip-submitting-patches-clarification

SubmittingPatches: clarify how Reviewed-by lines are added

Reviewed-by: Sage Weil <sage@redhat.com>

SubmittingPatches: clarify how Reviewed-by lines are added

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #4878 from dachary/wip-11901-spec

ceph.spec.in: add missing BuildRequires from SUSE block

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

ceph.spec.in: move specific BuildRequires to where they belong

Move distro-specific BuildRequires out of "common" section and
into the appropriate %if statement in the "specific" section.
Also remove a duplicated "Requires: gdisk".

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

install-deps.sh: detect yum-builddep errors

yum-builddep does not exit on error when an error happens: grep the
output for the error: string instead.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph.spec.in: add missing BuildRequires from SUSE block

The %endif removed by f94f23297c05019db754ec53b4ceaea7706152c9 is restored.
The %else removed by 75e87a20dac05441b97ce99dea76fff8e3a863ac is restored.
The stray %endif added by d8abde3338b0c7df373b762e35099ad5123866bf is removed.

May 29th, in d8abde3338b0c7df373b762e35099ad5123866bf Owen added a stray
%endif after

   BuildRequires: gperftools-devel

around line 133. June 3rd, in f94f23297c05019db754ec53b4ceaea7706152c9
Ken correctly attributed the error

     "error: /srv/autobuild-ceph/gitbuilder.git/build/ceph.spec:140: Got a
      %endif with no %if"

to a stray %endif but did not remove the one causing problem and in
doing so created another problem. June 4th, in
75e87a20dac05441b97ce99dea76fff8e3a863ac Owen incorrectly fixed
this new problem by removing the %else that is near

   BuildRequires: gperftools-devel

around line 116, instead of reverting
f94f23297c05019db754ec53b4ceaea7706152c9. As a consequence the
ceph.spec.in became syntactically correct but implemented an if/else
logic different from what was intended originally and a number of
BuildRequires became exclusive to SUSE and were not installed for CentOS
7 etc.

http://tracker.ceph.com/issues/11901 Fixes: #11901

Signed-off-by: Loic Dachary <ldachary@redhat.com>

erasure-code: implement ErasureCode::sanity_check_k

Implement the ErasureCode::sanity_check_k helper for plugins
that need to verify k >= 2.

http://tracker.ceph.com/issues/10358 Fixes: #10358

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4834 from liewegas/wip-11461

fix build on arm w/ old kernels

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

Merge remote-tracking branch 'origin/wip-client-fsync'

Conflicts:
src/client/Client.cc

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4831 from ceph/wip-11807

qa: fix multiple_rsync.sh to avoid using /usr/ directly

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4743 from ceph/wip-client-oldest-tid

Wip client oldest tid

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4723 from ceph/wip-snap-misc

Wip snap misc

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4702 from tchaikov/wip-fix-11590

mon: always reply mdsbeacon

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4602 from ceph/wip-11504

mon: refine check_remove_tier checks

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

9.0.1

Merge pull request #4781 from ceph/wip-librbd-perf-counters

librbd: re-add missing discard perf counters

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4870 from joaquimrocha/extra_doc_info_for_debug_option

build: Add info about tests to --with-debug option

Reviewed-by: Loic Dachary <ldachary@redhat.com>

build: Add info about tests to --with-debug option

The configure's --with-debug option builds also some of the tests
so this should be mentioned in its doc string in order to let users
know how to build all the tests.

Signed-off-by: Joaquim Rocha <joaquim.rocha@cern.ch>

Merge pull request #4866 from ddiss/fix_test_ceph_disk_dmcrypt_uuid_only

tests: fix test_activate_dmcrypt uuid usage

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: fix test_activate_dmcrypt uuid usage

4601e10800a63cf0e03108e1da0bf11c19c33e26 introduced a regression in that
an empty (uninitialised) OSD uuid is passed to test_pool_read_write for
IO. As a result, the "rados put" request times out causing test failure.

This change ensures that a correct OSD uuid is passed to
test_pool_read_write.

Signed-off-by: David Disseldorp <ddiss@suse.de>

Merge pull request #4528 from ceph/wip-librbd-helgrind

librbd: correct issues discovered via lockdep / helgrind

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4857 from jdurgin/wip-rbd-features-constants

update rbd features constants for python usage

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: flush pending ops while not holding lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: fix possible deadlock in librbd ImageWatcher tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: enable lockdep for librbd unit tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: owner_lock should be held during flush request

Flush might result in the cache writing out dirty objects, which
would require that the owner_lock be held.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc: ObjectCacher flusher might needs additional locks

librbd requires the ObjectCacher flusher thread to acquire
an additional lock in order to maintain lock ordering
constraints.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix recursive locking issues

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify state machine handling of exclusive lock

It is expected that all IO is flushed and all async ops are cancelled
prior to releasing the exclusive lock.  Therefore, replace handling of
lost exclusive locks in state machines with an assertion.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: AsyncObjectThrottle should always hold owner_lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: execute flush completion outside of cache_lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add AsyncRequest task enqueue helper method

In order to support the invariant that all state machine
callbacks occur without holding locks, transitions that
don't always involve a librados call should queue their
callback.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: disable lockdep on AioCompletion

It is only used by clients and it causes a large slowdown
in performance due to the rate at which the lock is constructed/
destructed for each IO request.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: AioCompletion shouldn't hold its lock during callback

The callback routine most likely will attempt to retrieve the result
code, which will result in a recursive lock attempt.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: complete cache read in a new thread context

The ObjectCacher complete the read callback while still holding
the cache lock.  This introduces lock ordering issues which are
resolved by queuing the completion to execute in a clean (unlocked)
context.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: give locks unique names to prevent false lockdep failures

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

log: fix helgrind warnings regarding possible data race

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: fix helgrind warnings

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: add support for flushing watches

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: lockdep now support unregistering once destructed

librbd use of an image hierarchy resulted in lock names being
re-used and incorrectly analyzed.  librbd now uses unique lock
names per instance, but to prevent an unbounded growth of
tracked locks, we now remove lock tracking once a lock is
destructed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: add valgrind.h convenience wrapper

Conditionally support helgrind annotations if valgrind support is
enabled during the build.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add work queue for op completions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

WorkQueue: ContextWQ can now accept a return code

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/rbd.py: update with new features constants

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

librbd: describe the purpose of features constants

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

librbd: add a constant for single-client-only features

This way it's easy to disable them if you want to access an image from
multiple clients.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>