add Jans email for nightly notifications

Merge pull request #1345 from ceph/rm-xenial

ceph-dev-*: remove xenial for master and octopus

ceph-dev-trigger: add conditional step for master and octopus

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-dev-new-trigger: remove xenial for master and development branches

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1329 from ceph/ceph_container_nightly_devel_build

Add ceph container nightly devel build job

Merge branch 'master' into ceph_container_nightly_devel_build

Merge pull request #1342 from ceph/ceph-volume-nautilus

ceph-volume-test: adds on-evaluation-failure to the simple build step

ceph-volume-test: adds on-evaluation-failure to the simple build step

This makes the job skip the simple tests if the condition fails instead
of failing the run.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #1339 from ceph/ceph_ansible_nightly_registry_pagesize

ceph-ansible-nightly: update tags pagination

Merge branch 'master' into ceph_ansible_nightly_registry_pagesize

Merge pull request #1341 from ceph/ceph-volume-nautilus

ceph-volume-test: only run simple tests for luminous and mimic

ceph-volume-test: only run simple tests for luminous and mimic

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #1340 from ceph/ceph-volume-nautilus

ceph-volume: fix logic when deciding to run simple tests

ceph-volume: fix logic when deciding to run simple tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-ansible-nightly: update tags pagination

Set the page size to 100 instead of 10 (default).
Also we should add an extra page when the remainder of the modulo
operation between the tag count and page size isn't zero.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1337 from ceph/guits-install_jq

jjb: install jq package

jjb: actually fail if a command doesn't return 0

this sets `-euxo pipefail` to avoid this situation :

```
++ sort
++ grep -v jenkins-job-builder
++ jq -r '.jobs[].name'
/tmp/jenkins7521600802673204021.sh: line 1043: jq: command not found
++ curl -s https://2.jenkins.ceph.com/api/json
[PostBuildScript] - Executing post build scripts.
[jenkins-job-builder] $ /bin/sh -xe /tmp/jenkins7309655147676749318.sh
+ rm /home/jenkins-build/.jenkins_jobs.2.jenkins.ceph.com.ini
Finished: SUCCESS
```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1338 from ceph/guits-containers_releases

ceph-ansible: make it possible to run nighly job on custom branch

ceph-ansible: make it possible to run nighly job on custom branch

When running manually a nightly containerized job against a custom branch of
ceph-ansible, it fails because `CEPH_DOCKER_IMAGE_TAG` is not set.

Typical error:
```
+ case "$CEPH_ANSIBLE_BRANCH" in
+ CEPH_DOCKER_IMAGE_TAG=
+ '[' -n '' ']'
Build step 'Execute shell' marked build as failure
```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1336 from ceph/guits-remove_oldstable_jobs

ceph-ansible: remove old stable jobs

ceph-ansible: remove old stable jobs

This commit removes the (very) old stable jobs.
These jobs can't be launched, we have to use the new matrix.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1335 from ceph/guits-ubuntu_def

ceph-ansible: remove unused ubuntu jobs

ceph-ansible: remove unused ubuntu jobs

This commit removes some ubuntu job definition which are unused.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1333 from ceph/wip-ok-dan

ceph-dev-*: Clarify job descriptions

ceph-dev-*: Clarify job descriptions

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1332 from ceph/ceph-volume-nautilus

ceph-volume: only test simple on mimic and luminous branches

ceph-volume: only test simple on mimic and luminous branches

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #1331 from ceph/ceph-volume-nautilus

ceph-volume: add nautilus to nightly testing

ceph-volume: add nautilus to nightly testing

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #1330 from tchaikov/wip-40646

ceph-*build/build/setup_rpm: downgrade to gcc-c++-8.2.1

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph-*build/build/setup_rpm: downgrade to gcc-c++-8.2.1

Fixes: http://tracker.ceph.com/issues/40646
Signed-off-by: Kefu Chai <kchai@redhat.com>

Add ceph container nightly devel build job

The ceph container images are only built on github event and also based
on stable ceph content.
We can also have nightly builds with ceph developement content from
shaman/chacra.

See https://github.com/ceph/ceph-container/pull/1411

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1327 from rishabh-d-dave/add-shrink_rgw-to-CI

ceph-ansible: add shrink_rgw scenario test to CI

Merge branch 'master' into add-shrink_rgw-to-CI

Merge pull request #1328 from tchaikov/wip-gcc-9-is-nice-but

scripts/build_utils.sh: only use GCC-9 for building crimson

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

scripts/build_utils.sh: only use GCC-9 for building crimson

using GCC-9 for building mimic and other stable releases has following
consequences:

* it brings runtime dependencies of libstdc++ runtime shipped along with
  GCC-9
* GCC-9 is more picky, so some 3rd party libraries fail to build due to
  -Werror.

let's continue using GCC-8 on bionic when building non-crimson flavors

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-ansible: add shrink_rgw scenario test to CI

Add shrink_rgw scenario to CI so that it can be tested against master
and nautilus.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1326 from rishabh-d-dave/remove-shrink_mds-from-oldstable

ceph-ansible: remove shrink_mds from oldstable

Merge branch 'master' into remove-shrink_mds-from-oldstable

Merge pull request #1325 from rishabh-d-dave/add-shrink_rbdmirror-to-CI

ceph-ansible: add shrink_rbdmirror to CI

ceph-ansible: remove shrink_mds from oldstable

We don't need to run shrink_mds against "oldstable" releases.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

ceph-ansible: add shrink_rbdmirror to CI

Add shrink_rbdmirror scenario to CI so that it can be tested against
master and nautilus.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1324 from rishabh-d-dave/add-shrink_mgr-to-CI

ceph-ansible: add shrink_mgr scenario to CI

ceph-ansible: add shrink_mgr scenario to CI

Add shrink_mgr scenario to CI so that it can be tested against master
and nautilus.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1323 from ceph/guits-use_start_tox

ceph-ansible: build job name properly

ceph-ansible: build job name properly

Since tox.ini file has been split, this is needed to be able to run the
individual scenario testing. Those parameters are used to build the job
name passed to start_tox() function.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1322 from ceph/guits-use_start_tox

ceph-ansible: use start_tox function in ansible-scenario job

ceph-ansible: use start_tox function in ansible-scenario job

otherwise it always tries to run scenario using tox.ini only.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1320 from rishabh-d-dave/add-shrink_mds-to-CI

add shrink_mds scenario to CI

add shrink_mds scenario to CI

Add shrink_mds scenario to CI so that it can be tested against master.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1319 from tchaikov/wip-1315

ceph-dev-new-setup: missing pieces in PR#1315

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

scripts/build_utils.sh: add missing closing quote

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-setup: pass "--with seastar" to rpmbuild for crimson flavor build

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: dyna link against libgcc when building crimson

as seastar requires dynamic linkage of libgcc, see the comment for more
details. see also

https://github.com/scylladb/seastar/blob/master/src/core/exception_hacks.cc

without this change, we will have errors like:

/usr/lib/gcc/x86_64-linux-gnu/9/libgcc_eh.a(unwind-dw2.o): In function
`_Unwind_RaiseException':
(.text+0x24b0): multiple definition of `_Unwind_RaiseException'
../../../lib/libseastar.a(exception_hacks.cc.o):./obj-x86_64-linux-gnu/src/seastar/./src/seastar/include/seastar/core/future.hh:738:
first defined here
collect2: error: ld returned 1 exit status
src/test/crimson/CMakeFiles/unittest_seastar_lru.dir/build.make:140:
recipe for target 'bin/unittest_seastar_lru' failed

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: install gnupg

so that `apt-key add -` can import the pubkey

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-setup: move "git diff" before install-deps.sh

as install-deps.sh now changes debian/control if we are building the
crimson flavor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1315 from tchaikov/wip-crimson-flavor

scripts,ceph-dev-new*: add "crimson" flavor

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

scripts/build_utils.sh: update ubuntu-toolchain-r mirrors

the removed two do not have GCC-9, but the newly added mirros do.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: install GCC-9 on bionic

GCC-8 is good enough for build Ceph. but we are trying to focus on
1 year later's mainstream toolchain, and only the PPA of
"ubuntu-toolchain-r" offers GCC-9 for bionic, so let's just use GCC-9
for building all flavors on bionic. simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: add a param for "setup_pbuilder_for_ppa()"

allowing it to install other GCC than GCC-7

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new*: build "crimson" flavor for octopus and master

* add "crimson" as another flavor
* build "default", "notcmalloc" flavors for branches whose name
  match jewel, kraken, luminous, mimic, nautilus
* build "default", "notcmalloc" and "crimson" flavors for branches whose name
  does not match any of jewel, kraken, luminous, mimic, nautilus.
  and build "crimson" on bionic only, because the "ubuntu-toolchain-r"
  PPA offers GCC-9, which is the latest GCC release at the time of
  writing. we want to target the mainstream hardware and toolchain
  available at least 1 year later. by then, GCC-9 will be available
  in most distros. and hopefully, it will be included by the
  devtoolset-9.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-setup: add "crimson" flavor

currently, we have two flavors for ceph builds

1. "default", which is, well, the default build.
2. "notcmalloc" -- "no tcmalloc", originally, it was only for applease
   valgrind. but now, it is more like a Debug build, which is less
   performant, but it is compiled with some facilities enabled to help
   debugging.

as crimson is not yet production ready, a new flavor is added in this
change -- "crimson". this flavor is built only for testing crimson
in sepia nodes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-setup: set editor variables/options

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1318 from ceph/guits-modify_collect_log

ceph-ansible: improve collect ceph config file and logs

ceph-ansible: improve collect ceph config file and logs

let's collect these files in one task instead of two tasks.
By the way, it will avoid undesired failures regarding ceph config files
which don't exist (it first tried ceph.conf > test.conf >
mycluster.conf)

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1317 from ceph/ceph_ansible_dashboard_stable_4.0

ceph-ansible: add dashboard scenario to stable-4.0

ceph-ansible: add dashboard scenario to stable-4.0

This commit adds the dashboard scenario to nautilus deployment.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1316 from ceph/guits-add_ovh_nodes

ceph-ansible: use more slave in CI

ceph-ansible: use more slave in CI

we stopped to use OVH nodes because of an old ceph-disk race condition.
Since ceph-disk isn't present anymore as of nautilus, let's use ovh
nodes again.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1306 from ceph/wip-submodules-trigger

ceph-pr-submodules: add trigger phrase

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-pr-submodules: add trigger phrase

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1314 from ceph/wip-sepiafog

sepia-fog-images: Inject IPMI and FOG credentials

sepia-fog-images: Inject IPMI and FOG credentials

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1312 from ceph/wip-cb-shaman

Give a few more jobs SHAMAN_API_KEY

Add SHAMAN_API_KEY to a few missing jobs

Signed-off-by: David Galloway <dgallowa@redhat.com>

Ugh, this made it in by mistake

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1311 from ceph/wip-fixjjb

jenkins-job-builder: Inject JOB_BUILDER_PASS

jenkins-job-builder: Inject JOB_BUILDER_PASS

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1309 from callithea/wip-ceph-dashboard-pull-requests-fix-repo-path

ceph-dashboard-pull-requests: Fix repo path

Reviewed-by: Laura Paduano <lpaduano@suse.com>

ceph-dashboard-pull-requests: Fix repo path

Signed-off-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #1307 from tchaikov/wip-ceph-dashboard-pull-requests-only-on-amd64

ceph-dashboard-pull-requests: only select x86_64 nodes

Reviewed-by: Laura Paduano <lpaduano@suse.com>

ceph-dashboard-pull-requests: only select x86_64 nodes

as google chrome is not available on arm64

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dashboard-pull-requests: only install google chrome on amd64

and remove the repo settings after done with installation

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1308 from ceph/wip-tserlin-nfs-ganesha-disable-google-chrome-repo

nfs-ganesha: disable google-chrome yum repo

nfs-ganesha: disable google-chrome yum repo

Merge pull request #1305 from ceph/ceph_ansible_disable_container_dashboard

ceph-ansible: Disable dashboard container job

ceph-ansible: Disable dashboard container job

The ceph-ansible dashboard job with containerized deployment requires
some changes not merged yet upstream.
This job will be reenable when all the changes are merged.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1303 from ceph/guits-ca_podman

ceph-ansible: test podman on centos instead rhel

ceph-ansible: split project for ubuntu jobs

so we don't create jobs that are never triggered in the CI.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-ansible: fix ooo_collocation (stable-3.1)

fix a typo, this job should be tested in a containerized environment
only.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-ansible: test podman on centos instead rhel

podman is available on atomic os, from upstream perspective it's better
to test against this image instead a rhel8 which was a beta image outdated.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1302 from ceph/dsavineau-ceph-container-lint-selinux

ceph-container-lint: Add selinux flag to volume

ceph-container-lint: Add selinux flag to volume

Without the selinux flag on the container volume, the container
command will fail:

: openBinaryFile: permission denied (Permission denied)

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1300 from ceph/dsavineau-ansible-pr-syntax

ceph-ansible: remove useless command for syntax

ceph-ansible: remove useless command for syntax

Because we're now able to execute the infrastructure playbooks from
the main ceph-ansible directory, we don't need to define the ansible
library path (because already define in ansible.cfg).
The roles and group_vars directories are already present in the main
directory so we don't need to copy them anymore.
This commit also exclude both ansible-lint rules: 204 and 205

- [204] Lines should be no longer than 160 chars
- [205] Use ".yml" or ".yaml" playbook extension

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1299 from ceph/guits-update-pr-syntax

ceph-ansible: give a common inventory for syntax-check

ceph-ansible: give a common inventory for syntax-check

let's provide a typical 'all_daemons' inventory host file for the syntax
check process. Otherwise we can get failure because the site.yml file
contains `hosts:` which try to resolve some `{{ groups }}` variables.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1298 from ceph/guits-add_ca-dashboard

ceph-ansible: add new scenario 'dashboard'

ceph-ansible: add new scenario 'dashboard'

This commit adds a new scenario to test the dashboard deployment.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1297 from ceph/wip-ceph-dev-new-trigger-shaman

ceph-dev-new-trigger: Supply SHAMAN_API_KEY