ReplicatedPG: remove pushing/pulling

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: handle down pull sources

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: split recover_missing out of prepare_pull

Also, move prepare_pull to PGBackend.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: shuffle over submit_push_*/build_push_op

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: move over send_pushes/send_pulls/prep_push_op_blank

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: move over _failed_push

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: move handle_*

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: shuffle do_pull/do_push_reply

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: move _do_push,_do_pull_response,do_push

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: convert sub_op_pull

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: make log_subop_stats static

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: shuffle sub_op_push_reply

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG/Backend: shuffle sub_op_push over to ReplicatedBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: allow dout for ReplicatedBackend methods to work in ReplicatedPG.cc

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: extract PGBackend::Listener recovery callbacks

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG,ReplicatedPG: handle do_request in ReplicatedPG,PGBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: pass on dump_recovery_info to PGBackend

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD,ReplicatedPG: let PGBackend handle the temp collection

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: add PGBackend interfaces and stubs

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: Allow get_object_context caller to provide attributes

This will be used by PGBackend implementers to get the SnapSet
and ObjectInfo for newly recovered objects on the primary.

get_object_context may be called on a missing object for a lost_revert

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: send_push is unused

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore: ping tphandle between each operation within a transaction

Signed-off-by: Samuel Just <sam.just@inktank.com>

WorkQueue: add a workqueue which simply runs queued GenContexts

Signed-off-by: Samuel Just <sam.just@inktank.com>

Context: add GenContext templated on the callback argument

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #640 from ceph/wip-mon-pgmap

fix mon crash when osdmap and pgmap aren't yet synced

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

mon/PGMonitor: fix segfault when osdmap and pgmap stats are out of sync

The leader mon is responsible for propagating changes in the osdmap
into the pgmap (e.g., when a pool or osd goes away).  If that hasn't
happened yet, the pg_map map will be out of sync with the current set
of pools.  We shouldn't crash.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #637 from dachary/master

mailmap: add Matthew Roy, Matthew Wodrich

Merge pull request #639 from ceph/wip-gtest

Makefile: don't use srcdir in TESTS

Reviewed-by: Roald J. van Loon <roaldvanloon@gmail.com>

Makefile: don't use srcdir in TESTS

src/Makefile-env.am:31: error: using '$(srcdir)' in TESTS is currently broken: '$(srcdir)/unittest_bufferlist.sh'
src/Makefile.am:1:   'src/Makefile-env.am' included from here
src/Makefile-env.am:31: error: using '$(srcdir)' in TESTS is currently broken: '$(srcdir)/test/encoding/check-generated.sh'
src/Makefile.am:1:   'src/Makefile-env.am' included from here

on my (raring) laptop.

Signed-off-by: Sage Weil <sage@inktank.com>

mailmap: add Matthew Roy, Matthew Wodrich

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #615 from dachary/wip-jerasure-copy

ErasureCode: get rid of extra copy when encoding

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #634 from dachary/master

git: consolidate mails in commit logs

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #636 from dachary/wip-get_command_description

autoconf: remove get_command_descriptions  dependency to gtest

Reviewed-by: Sage Weil <sage@inktank.com>

ErasureCode: get rid of extra copy when encoding

The substr_of bufferlist method is used to reduce the copies to the
strict minimum. It is both able to split bufferptr when they are larger
than a chunk and to concatenate them when they are smaller.

Signed-off-by: Loic Dachary <loic@dachary.org>

autoconf: remove get_command_descriptions  dependency to gtest

It is not required and a leftover of the time when it was a unit test
which was a mistake to begin with. It has been reported to create
dependencies problems in some cases, when gest is not available at the
time get_command_descriptions is compiled.

Signed-off-by: Loic Dachary <loic@dachary.org>

git: consolidate mails in commit logs

Map duplicate mails / names in commit logs to a single canonical names.
Useful to report to management with git shortlog -nes

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge remote-tracking branch 'gh/next'

Conflicts:
src/osd/ReplicatedPG.cc

Merge pull request #633 from ceph/wip-repop-leaks

fix a copule of leaks of RepGather structs

Reviewed-by: David Zafman <david.zafman@inktank.com>

doc: Removed service ops language and added reference to operations.

fixes: #6374

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

mon/OSDMonitor: fix 'ceph osd crush reweight ...'

The adjust method returns a count of adjusted items.

Add a test.

Fixes: #6382
Backport: dumpling
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #477 from jpfrancois/patch-1

Fix for Swift key command

osd/ReplicatedPG: fix leak of RepGather on watch timeout

new_repop() gives us a ref; we need to release it, just as we do in
execute_ctx().

Backport: dumpling
Signed-off-by: Sage Weil <sage@inktank.com>

osd/ReplicatedPG: fix leak of RepGather on large-object COPY_FROM

new_repop() gives us a ref; we need to release it, just as we do in
execute_ctx().

Signed-off-by: Sage Weil <sage@inktank.com>

osd: change warn_interval_multiplier to uint32_t

to prevent overflow in OpTracker::check_ops_in_flight when
multiplying warn_interval_multiplier *= 2

Backport: cuttlefish, dumpling

http://tracker.ceph.com/issues/6370 fixes #6370

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 1bce1f009bffd3e28025a08775fec189907a81db)

Merge pull request #630 from dachary/master

osd: change warn_interval_multiplier to uint32_t

Reviewed-by: Sage Weil <sage@inktank.com>

arch/intel: fix old comment

Signed-off-by: Sage Weil <sage@inktank.com>

arch/intel: use intel probe instructions for x86_64 only

Not LP64, which includes ppc64 and clearly does not build.

Fixes: #6283
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

osd: change warn_interval_multiplier to uint32_t

to prevent overflow in OpTracker::check_ops_in_flight when
multiplying warn_interval_multiplier *= 2

Backport: cuttlefish, dumpling

http://tracker.ceph.com/issues/6370 fixes #6370

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #628 from dachary/wip-erasure-fix-thread-test

ErasureCode: fix plugin loading threaded test

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #627 from dachary/master

rgw: fix rgw test to reflect usage change

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

ErasureCode: fix plugin loading threaded test

The TEST_F(ErasureCodePluginRegistryTest, factory_mutex) was bugous and
only succeeded by chance. The sleep was on the factory constructor which
was never called. An erasure code plugin that hangs forever on load is
created instead.

The sleep_forever.detach is replaced by pthread_cancel to interrupt the
thread that hangs forever. If not, gtest will try to join the thread and
never exit.

Signed-off-by: Loic Dachary <loic@dachary.org>

rgw: fix rgw test to reflect usage change

otherwise src/test/cli/radosgw-admin/help.t fails when running make
check when run after a configure --with-radosgw

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #588 from dachary/wip-6274

mon: unit tests to protect against some MonCommands.h typos

osd: revert 'osd max xattr size' limit

Set it to 0 (unlimited) for now.

Backport: dumpling

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #566 from ceph/wip-purge-stray

Fixes for purging stray

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #591 from ceph/wip-misc

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge branch 'master' of github.com:ceph/ceph

Merge pull request #625 from ceph/wip-warn-pg

mon: warn when pg_num is too low or appears out of whack wrt the cluster size

Reviewed-by: Greg Farnum <greg@inktank.com>

mds: be more careful about decoding LogEvents

We need to wrap the full decode section or we can abort the process
if there's an issue (which we may want to just skip by).

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #619 from dachary/wip-erasure-pad

ErasureCode: optimize padding calculation

Reviewed-by: Sage Weil <sage@inktank.com>

ErasureCode: optimize padding calculation

Andreas-Joachim Peters pointed out that the ErasureCodeJerasure method
pad_in_length is very inefficient for no reason. It is replaced by the
alignment method and the calculation is inlined in the encode method.

Signed-off-by: Loic Dachary <loic@dachary.org>

vstart: set 'mon pg min per osd'

We need to tweak this since we create relatively few pgs per osd.

Signed-off-by: Sage Weil <sage@inktank.com>

mon/PGMonitor: health warn if pool has relatively high objects/pg

If there is a pool that has a high objects/pg relative to the rest of the
cluster, warn, as this suggests this particular pool may have too few
PGs.

Signed-off-by: Sage Weil <sage@inktank.com>

mon/PGMonitor: health warn if pg_num != pgp_num

Users need to adjust pg_num *and* pgp_num for split but may forget to do
both.

Signed-off-by: Sage Weil <sage@inktank.com>

mon: warn if pg to osd ratio is too low

If there are not enough PGs relative to the number of in osds, warn.

Signed-off-by: Sage Weil <sage@inktank.com>

ErasureCode: coding style conformance

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph_argparse: unit tests for validate_command config-key

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

ceph_argparse: unit tests for validate_command osd

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

ceph_argparse: unit tests for validate_command mon

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

ceph_argparse: unit tests for validate_command mds

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

ceph_argparse: unit tests for misc validate_command

Contrary to all other classes, this series of command ( Monitor ) does
not have a common prefix.

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

ceph_argparse: unit tests for validate_command auth

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

ceph_argparse: unit tests for validate_command pg

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

pybind: ceph_argparse unit tests foundations

The general idea is to have a series of commands, in the same order as
they show in mon/MonCommands.h, as if they were input to the ceph
client. For each command a valid combination is verified. And at least
one validation error is checked to produce a validation error. For
instance:

    ['pg', 'stat']

is a valid command and the validate_command function is expected to
return a value that is not None or {}. The command

    ['pg', 'stat', 'toomany' ]

is also given to validate_command to check that an error occurs when
an extra argument is given.

The TestArparse class implements a few methods to reduce the verbosity
of the tests. It does not provide many methods : only those that
significantly reduce the verbosity have been implemented. The drawback
of writing too many convenience methods is that they are more difficult
to read and maintain.

The signature dictionary is made a global variable so that
it is only extracted once for all classes. It is immutable.

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

pybind: catch EntityAddress missing /

If the / is missing in an EntityAddress, an ArgumentValid exception must
be raised so that it can be caught in the same way other argument
validation exceptions are.

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mon: validate mon dump epoch as a positive integer

All other epochs are validated in the same way

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

pybind: unit tests for ceph_argparse::parse_json_funcsigs

Run parse_json_funcsigs against the builtin commands found
in mon/MonCommands.h. Although it does not provide for a full
validation, it will detect some degenerate cases.

It is expected to raise if a space is missing at the end of an option
specification ( see https://github.com/ceph/ceph/pull/585 ) and this
case is tested. New tests of the same kind can be added whenever an
undetected error is found. Ideally a validation function is implemented
and it would be updated ( with an associated test ) when a new
pathological case is found.

The json string given to parse_json_funcsigs is obtained from
the support program get_command_descriptions.

The python-nose dependencies are added to the build requirements in
debian/control and ceph.spec.in because make check should always be run
at built time.

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

.gitignore gtags(1) generated files

Signed-off-by: Loic Dachary <loic@dachary.org>

mon: get_command_descriptions support program

The get_command_descriptions function is not designed to be tested in
C++ because all the validation happens in pybind/ceph_argparse.py. The
get_command_descriptions program is designed to be used by python unit
tests as a mean to get a JSON dump of the content of mon/MonCommands.h

      get_command_descriptions --all

      {"cmd000":{"sig":["pg","stat"],"help": ... "avail":"cli,rest"}}

It also provides a way to reproduce and keep track of past errors
( typos etc. ) to ensure the python validation keeps catching it.

      get_command_descriptions --pull585

Add /get_command_descriptions to .gitignore so that

      git ls-files --exclude-standard --others

does not see it which is required for
https://github.com/ceph/autobuild-ceph/blob/f018d220f2622a9fc8c86a31e1fa13263790c399/build-ceph.sh#L73

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mon: convenience function to convert commands to json

The get_command_descriptions is added to Monitor.h and contains
the code previously inlined in Monitor::handle_command to implement
the get_command_descriptions command. It is intended for tests to
convert command descriptions into json, including error cases.

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

autotools: set noinst_PROGRAMS

to be used by unit test support programs that do not need to be installed

Signed-off-by: Loic Dachary <loic@dachary.org>

autotools: group test scripts in check_SCRIPTS

The check_SCRIPTS is added in Makefile-env.am to list the tests that do
not require compilation. The scripts listed in check-local and in the
TESTS variable use check_SCRIPTS instead.

The PYTHONPATH environment variable is added to Makefile-env.am and
includes the pybind directory so that python unit tests can load the
libraries from sources.

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Roald J. van Loon <roaldvanloon@gmail.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #626 from ceph/wip-usage

common/config: include --cluster in default usage message

Merge pull request #599 from ceph/wip-6323

mon: OSDMonitor: fix #6322 and #6323

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #617 from dachary/wip-lfnindex

os: fix TestLFNIndex test warnings

Reviewed-by: Sage Weil <sage@inktank.com>

common/config: include --cluster in default usage message

Clean it up a bit too.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #624 from ceph/automake-fix-common

make: build common/secret only on linux

Reviewed-by: Sage Weil <sage@inktank.com>

make: build common/secret only on linux

libkeyutils is only available on linux. before the automake refactoring
secret.c was compiled into rbd and ceph_mount targets which are linux
only targets. secret.c was moved to libcommon during the refactoring,
but the conditional compilation was lost.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #618 from dachary/wip-jerasure-minimum-to-decode

ErasureCode: minimum_to_decode unit tests and optimization

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #605 from dachary/wip-erasure-code-plugin-test

erasure code plugin test

Merge pull request #603 from dachary/wip-erasure-code-example

erasure code example cleanup

Merge pull request #607 from ceph/prctl-getname-test

code_env: use feature test for PR_GET_NAME support

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #616 from ceph/automake-flags

automake: fixup am_common_flags to avoid re-definitions

Reviewed-by: Roald J. van Loon <roaldvanloon@gmail.com>

Merge pull request #614 from ceph/wip-rados-model

Prevent copy_from against yourself, and fix issues with the testing framework around it

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge remote-tracking branch 'gh/next'

doc: When bootstrapping mon set the correct caps for client.admin

Otherwise client.admin will be privilege-less and the admin can't
access the cluster.

ErasureCode: minimum_to_decode unit tests and optimization

The minimum_to_decode function simply returns the available chunks if no
recovery is necessary.

Add unit tests covering all minimum_to_decode situations:

* trying to read nothing
* read a chunk if none are available
* reading a subset of the available chunks
* read a missing chunk if there is less than k chunks available.
* reading when a chunk must be recovered

Signed-off-by: Loic Dachary <loic@dachary.org>

os: fix TestLFNIndex test warnings

Do not ignore system return value. Wrap emacs compile line to 80
characters.

Signed-off-by: Loic Dachary <loic@dachary.org>

mds: remove dirfrags when purging stray directory

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: avoid fetching backtrace when purging stray

we save old data pools in both inode_backtrace_t::old_pools and
inode_t::old_pools. We have the inode in the cache when purging
stray, so no need to fetch backtrace to find old data pools.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>