ceph-ansible-pipeline: Fix stable-3.0 phase conditional steps

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1058 from ceph/evelu-cache

ceph-pull-requests: Improving ccache efficiency

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-pull-requests: Improving ccache efficiency

As of today, ccache is installed by run-make-check but the statistics
reports almost no cache it (<1%) and even worse, this is slowing down
the whole process. The needed time to setup the cache isn't compensated
by the reuse of the cache as every build is rebuilding the cache. This
is leading to a 10% loss of the compilation time.

So the question, is why does two consecutives builds doesn't reuse the
cache. Its usually a matter of variables introduces on each builds like
a sha1 or a __DATE__ or __TIME__ in a header.

As per
https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal, several
tools from the tool chain (like compilers or make tools) are able to use
theses variables to enforce a particular timestamp at build time.

This commit also add a "-D ENABLE_GIT_VERSION=OFF" to avoid having the
commit id in the ceph_ver.h header. Not having the git versioning inside
the CI job is not a by deal as the binaries will be destroyed after the
build procedure.

These changes avoid introducing small variations in the source code
which breaks the build cache (ccache). The expected result is to get a
80 or 90% cache hit between two very close builds.

Signed-off-by: Erwan Velu <erwan@redhat.com>

Merge pull request #1042 from a2batic/test1

mgr/dashboard: Adds unit test coverage report

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

mgr/dashboard: Adds unit test coverage report
Enable plugin 'Cobertura' to generate code coverage report, for every PR for ceph-dashboard frontend unit test.
Cobertura jenkins plugin: https://wiki.jenkins.io/display/JENKINS/Cobertura+Plugin

Fixes: http://tracker.ceph.com/issues/24377
Signed-off-by: Kanika Murarka <kmurarka@redhat.com>

Merge pull request #1057 from ceph/wip-arm-name

ceph-pull-requests-arm64: Normalize name to appease JJB

ceph-pull-requests-arm64: Normalize name to appease JJB

Not sure why '!!python/unicode' was there originally but the current
version of JJB doesn't like it.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1055 from ceph/wip-ansible-nostable

ceph-ansible-pipeline: Skip *lvm* scenarios if merging into stable-3.0

ceph-ansible-pipeline: Skip *lvm* scenarios if merging into stable-3.0

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1054 from ceph/wip-ansible-condition

ceph-ansible-pipeline: Only run next phase on SUCCESS

ceph-ansible-pipeline: Only run next phase on SUCCESS

I think I misunderstood the jjb syntax before.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1053 from ceph/group-vars-fix

ceph-ansible-pr-syntax-check: fix grep error

ceph-ansible-pr-syntax-check: fix grep error

If grep fails the script exits because of the set -e so let's use ||
true to workaround that issue.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1052 from ceph/group-vars

ceph-ansible-pr-syntax-check: add group_vars check

ceph-ansible-pr-syntax-check: add group_vars check

This commit does 2 things:

* split the structure of the script to use functions
* add a function to test if group_vars weren't generated

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1050 from ceph/guits_update_ceph-ansible_nightlies

ceph-ansible: update nightlies definition

ceph-ansible: update nightlies definition

- remove testing of stable-2.2 against ceph jewel
- add testing of stable-3.1 against luminous
- update testing of stable-3.0 against luminous:
  - remove lvm scenarios

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1051 from ceph/guits-change_osd_bs_project_ceph-ansible

ceph-ansible: move bluestore_osds_container in ceph-ansible-prs-auto

ceph-ansible: move bluestore_osds_container in ceph-ansible-prs-auto

we are seeing this scenario failing because of the race condition,
moving it to a project which requires smithi nodes should help.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1049 from ceph/rm-ceph-disk-master

ceph-volume-nightly: remove master from simple, ceph-disk no longer available

ceph-volume-nightly: remove master from simple, ceph-disk no longer available

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1048 from ceph/ceph-deploy-distros

ceph-deploy-build: remove centos6

ceph-deploy-build: remove centos6

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1047 from ceph/wip-allow-all

ceph-ansible-*: permit-all to prevent github comment flood

ceph-ansible-*: permit-all to prevent github comment flood

See https://github.com/jenkinsci/ghprb-plugin/issues/608

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1046 from ceph/guits-revert_80f02250028c

Revert "ceph-ansible-prs: run 'auto' scenarios on smithi or ovh"

Revert "ceph-ansible-prs: run 'auto' scenarios on smithi or ovh"

This reverts commit 80f02250028ce5daae6eed3717787732d06302d1.

Merge pull request #1045 from ceph/wip-ansible-pipeline3

ceph-ansible-pipeline: Implement additional phases

ceph-ansible-pipeline: Implement additional phases

1) Docs and yaml linter (always)
2) Check each infrastructure playbook and run if modified
3) centos7 and docker clusters
4) All other scenarios if basic cluster tests pass

Phases 3 and 4 will NOT be run if infrastructure-playbooks were
modified.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1044 from ceph/wip-ansible-pipeline

Implement ceph-ansible pipeline

Let ceph-ansible-pipeline trigger ceph-ansible-* jobs

Signed-off-by: David Galloway <dgallowa@redhat.com>

Add ceph-ansible master multijob

This job will trigger docs, syntax check, and if they pass, then the
actual ceph-ansible jobs will run.

The ceph-ansible-prs-auto project has to move to this job config in order for
the pipeline to start the appropriate jobs for PRs.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1043 from ceph/guits-run_ceph-ansible_prs_ovh-smithi

ceph-ansible-prs: run 'auto' scenarios on smithi or ovh

ceph-ansible-prs: run 'auto' scenarios on smithi or ovh

This is slowing down CI testing because we can't run more than 1 or 2
PRs at a time. Since I'm not sure if it's still worth to limit these
scenarios to be run only on smithi node, let's try to temporary remove
this limitation.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1038 from ceph/wip-no-docs

ceph-ansible-prs: Don't run auto job if only docs were modified

ceph-ansible-prs: Don't run auto job if only docs were modified

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1041 from tchaikov/wip-build_utils.sh-cleanup

build_utils.sh: do not try to check non-existent file's size

Reviewed-by: Alfredo Deza <adeza@redhat.com>

build_utils.sh: do not try to check non-existent file's size

it's distracting to read the red alert in jenkins' log like

du: cannot access '/srv/debian-base/xenial.tgz': No such file or
directory

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1040 from ceph/ceph-ansible-prs-dev

build_utils: when running dev-* scenarios set release to nautilus

ceph-build-pull-requests: pin urllib3 to 1.22

We need to do this because jenkins-job-builder installs requests, and if
we don't pin here a version of urllib3 is installed that is incompatible
with the version of requests that jenkins-job-builder requires which is
urllib3<1.23,>=1.21.1

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

build_utils: when running dev-* scenarios set release to nautilus

When running dev-* tests for ceph-ansible PRs we were setting
CEPH_STABLE_RELEASE to 'dev' which is not an actual release and our
functional tests break because of that.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #1039 from ceph/ceph-ansible-prs-dev

ceph-ansible-prs: adds the dev-ansible2.4-centos7_cluster scenario

ceph-ansible-prs: adds the dev-ansible2.4-centos7_cluster scenario

This will allow us to test dev installs with a large cluster and many
node types.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #1037 from ceph/guits-followup_1036

ceph-ansible: followup on #1036

ceph-ansible: followup on #1036

these tasks are producing some useless output showing error on debian
based distribution. Let's add a condition to avoid this.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1036 from ceph/guits-debug_yum_failures

ceph-ansible: add some tasks for debugging purpose

ceph-ansible: add some tasks for debugging purpose

related to ceph/ceph-ansible/pull/2665 and ceph/ceph-ansible/pull/2652

We are seeing too many failures related to yum.
These additional tasks will help us to have more information when a yum
failure occurs.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1035 from tchaikov/wip-aarch64-repo

build_utils.sh: add ubuntu-ports repo for arm64 build

build_utils.sh: use arch-neutral path names

Signed-off-by: Kefu Chai <kchai@redhat.com>

build_utils.sh: add ubuntu-ports repo for arm64 build

as it offers updated binutils which does not exist in the default repo.
and the updated binutils is depended by gcc-7. this change address
following failure like:

The following packages have unmet dependencies:
 gcc-7 : Depends: binutils (>= 2.26.1) but 2.26-8ubuntu2 is to be
installed

when installing g++-7.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1034 from ceph/fix_collect_logs

ceph-ansible: fix teardown issue

ceph-ansible: fix teardown issue

At the moment, the teardown isn't fully executed because of `collect-logs.yml`
playbook.

it is expected the following task can fail according to the context of the main
failure:

```
TASK [find ceph logs] **********************************************************
task path: /home/jenkins-build/build/workspace/ceph-ansible-scenario/collect-logs.yml:4
fatal: [ceph-nfs0]: FAILED! => {
    "changed": true,
    "cmd": [
        "find",
        "/var/log/ceph",
        "-name",
        "test*.log"
    ],
    "delta": "0:00:00.003363",
    "end": "2018-05-23 16:45:22.845782",
    "rc": 1,
    "start": "2018-05-23 16:45:22.842419"
}

STDERR:

find: ‘/var/log/ceph’: No such file or directory

MSG:

non-zero return code
```

therefore, `teardown_vagrant_tests()` doesn't return `0` and causes the
teardown to stop because of `set -e`

```
PLAY RECAP *********************************************************************
ceph-client0               : ok=1    changed=0    unreachable=0    failed=1
ceph-client1               : ok=1    changed=0    unreachable=0    failed=1
ceph-iscsi-gw0             : ok=1    changed=0    unreachable=0    failed=1
ceph-mds0                  : ok=1    changed=0    unreachable=0    failed=1
ceph-mgr0                  : ok=1    changed=0    unreachable=0    failed=1
ceph-mon0                  : ok=3    changed=2    unreachable=0    failed=0
ceph-mon1                  : ok=3    changed=2    unreachable=0    failed=0
ceph-mon2                  : ok=3    changed=2    unreachable=0    failed=0
ceph-nfs0                  : ok=1    changed=0    unreachable=0    failed=1
ceph-osd0                  : ok=1    changed=0    unreachable=0    failed=1
ceph-rbd-mirror0           : ok=1    changed=0    unreachable=0    failed=1
ceph-rgw0                  : ok=1    changed=0    unreachable=0    failed=1

Build step 'Execute Scripts' marked build as failure
Archiving artifacts
Finished: FAILURE
```

adding a `failed_when: false` on this task should avoid this bug.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1033 from ceph/remove_fetch_dir_teardown

ceph-ansible: some teardown fixes

ceph-ansible: vagrant clean old cache

Somehow, it's possible sometimes there is some old cache left after a
failure. `vagrant global-status --prune` ensures old caches that were left are
removed.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-ansible: remove fetch directory in teardown

Remove the fetch directory in teardown for ceph-ansible to avoid
keeping leftover from previous jobs.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1032 from ceph/cv-disable-bionic

ceph-volume-ansible-prs: remove bionic

ceph-volume-ansible-prs: remove bionic

ceph-volume doesn't have a 'bionic' factor in tox, so there aren't any
functional scenarios to support this. At the moment, there aren't any
ceph packages to test against either. Disabling this for now.

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1031 from ceph/wip-pip-nobinary

build_utils.sh: Don't update pip after installing it

build_utils.sh: Don't update pip after installing

Attempting to update pip was resulting in job failures because the wheel
wasn't present in the local pip cache.

Since we pinned pip to 10.0.0 in
https://github.com/ceph/ceph-build/pull/1007, we don't need to update
pip immediately after.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1030 from joscollin/wip-fix-signedoff-guide

ceph-pr-commits/build: Fix Sign your work url

ceph-pr-commits/build: Fix Sign your work url

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #1029 from ceph/wip-rtslib-fb-pyudev

rtslib-fb: reduce pyudev version requirements

rtslib-fb: reduce pyudev version requirements

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1027 from ceph/cv-disable-bionic

ceph-volume-nightly: remove bionic

ceph-volume-nightly: remove bionic

ceph-volume doesn't have a 'bionic' factor in tox, so there aren't any
functional scenarios to support this. At the moment, there aren't any
ceph packages to test against either. Disabling this for now.

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1028 from tchaikov/bionic-gcc-7

build_utils.sh: help bionic to point its gcc back to 7

build_utils.sh: help bionic to point its gcc back to 7

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1026 from tchaikov/bionic-gcc-7

build_utils.sh: help bionic to point its gcc back to 7.3

build_utils.sh: help bionic to point its gcc back to 7.3

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1025 from ceph/wip-bionic-builddebs

setup_deb: Forgot bionic

setup_deb: Forgot bionic

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1024 from ceph/wip-bionic-build

build_utils.sh: Add Bionic

build_utils.sh: Add Bionic

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1023 from ceph/branch-regex-fix

ceph-build: use BRANCH instead of non-existing GIT_BRANCH

ceph-build: use BRANCH instead of non-existing GIT_BRANCH

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1022 from ceph/wip-bionic

Add support for Ubuntu Bionic globally

Add support for Ubuntu Bionic globally

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1019 from tchaikov/wip-do-not-build-mimi-on-trusty

ceph-build: do not build mimic on old distros

ceph-build: do not build mimic on old distros

mimic and nautilus require C++17 to build, but following distro do
not ship new enough GCC compiler:

- ubuntu trusty
- debian jessie
- debian stretch

so we won't build mimic or newer ceph on these distros.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1021 from ceph/disable_purge

ceph-ansible-prs: disable purge_bluestore_osds_container

ceph-ansible-prs: disable purge_bluestore_osds_container

This scenario is failing for every PRs because of a race condition.
A PR has been opened https://github.com/ceph/ceph/pull/21415 but until
it doesn't get merged we better to disable this scenario.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1020 from ceph/wip-bionic

sepia-fog-images: Add support for Ubuntu 18.04

sepia-fog-images: Add support for Ubuntu 18.04

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1018 from ceph/wip-timeout-arm

ceph-pull-requests-arm64: Timeout after 3h

ceph-pull-requests-arm64: Timeout after 3h

Builds were hanging indefinitely during unit tests

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1017 from ceph/centos-container

ceph-ansible-nightly: always test on centos

ceph-ansible-nightly: always test on centos

The job was still configured to test on Ubuntu and we want CentOS.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1016 from ceph/script-prune

cleanup: remove all top-level scripts

readme: update the deprecation with script locations

Signed-off-by: Alfredo Deza <adeza@redhat.com>

scripts: remove all top-level scripts

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1015 from ceph/evelu-ceph-container-arm64

ceph-container-build-push-imgs-arm64: Fixing scripts

ceph-container-build-push-imgs-arm64: Fixing scripts

As ceph-container needs a particular build script for arm64, let's fix
the script name.

Merge pull request #1014 from ceph/arm

ceph-container-build-push-imgs-arm64: fix node label

ceph-container-build-push-imgs-arm64: fix node label

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1013 from ceph/arm64-follwup

ceph-container-build-push-imgs-arm64: update node label

ceph-container-build-push-imgs-arm64: update node label

The arm64 nodes have a 'arm64 xenial' label.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1012 from ceph/ceph-container-prs

ceph-container-prs: only auto start a subsection of tests

ceph-container-prs: only auto start a subsection of tests

All tests can be triggered manually by github comments, but we only
want certain ones to start automatically when the PR is created.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #1011 from ceph/pip-easy-install

scripts: upgrade pip with easy_install before actual pinning

scripts: upgrade pip with easy_install before actual pinning

Signed-off-by: Alfredo Deza <adeza@redhat.com>