ceph-dev-build: add osc build scripts

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

validate_rpm: reuse rpm validation from another project

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

osc: add osc related scripts:
- validate_osc
- setup_osc
- build_osc
Supposed for adding them later to ceph-dev-new-build or ceph-dev-build

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1361 from ceph/wip-release-from-version

build_utils: add release from version

build_utils: use virtualenv -p python2.7 instead virtualenv

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

build_utils: fix get_rpm_dist openSUSE detection

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

build_utils: add release from version function

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1359 from ceph/wip-leap15-slave

ansible/slave: add opensuse leap15 osc support

ansible/slave: add opensuse leap15 osc support

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1360 from ceph/wip-tserlin-nfs-ganesha-disable-xenial

nfs-ganesha: don't build on xenial

nfs-ganesha: don't build on xenial

Ceph no longer builds its master branch on Xenial. See:
https://github.com/ceph/ceph-build/pull/1345

So we can no longer build the latest nfs-ganesha V2.9-dev on Xenial.

Merge pull request #1357 from tchaikov/wip-ceph-pr-docs/python3

ceph-pr-docs: install python3 packages

Merge pull request #1355 from ceph/fix-validate

ceph-dev-build: fix deb and rpm validate

ceph-pr-docs: install python3 packages

see also https://github.com/ceph/ceph/pull/29528

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-build: fix deb and rpm validate

validate_deb - not redhat does not mean it is deb distro,
there can be other packaging systems
validate_rpm - we do not want to proceed for zypper based,
we only want to proceed for redhat like repo

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1346 from ceph/nfs_ganesha_disable_lizardfs

nfs-ganesha: Disable LizardFS FSAL

Merge pull request #1350 from ceph/guits-remove-dashboard

ceph-ansible: remove dashboard scenarios

ceph-ansible: remove dashboard scenarios

since dashboard is enabled by default, we test it in all_daemons
scenario.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1349 from tchaikov/wip-grep-cleanup

ceph-dev-new-trigger: only use positive match for filter

Merge pull request #1354 from tchaikov/wip-pin-jjb

ceph-build-pull-requests: ping jjb to 2.10.1

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-build-pull-requests: ping jjb to 2.10.1

jjb 3.0 brings some incompatible behavior:

* it tries to expand the strings like ${FOO} using the parameters,
  so if we continue using include-raw, jjb test will fail like

```
Failure formatting job
....
ERROR:root:Failure formatting job '{
with itself
Traceback (most recent call last):
...
  File
"/tmp/venv.Pa7CUfDCFK/local/lib/python2.7/site-packages/jenkins_jobs/formatter.py",
line 43, in deep_format
    raise JenkinsJobsException(desc)
jenkins_jobs.errors.JenkinsJobsException:
...
...
TEMPVENV=$(mktemp -td venv.XXXXXXXXXX)
```
* also, it fails to pass down some parameters to the build job, for
  instance, the `WORKSPACE` is not passed anymore.

both these change break the "test" job, as we are using `include-raw`
everywhere, and we also use `${WORKSAPCE}` in some yml files.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1352 from tchaikov/wip-include-raw

s/!include-raw/!include-raw:/

s/!include-raw/!include-raw:/

the former is deprecated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-trigger: only use positive match for filter

there is no need to put
```
grep -v '\(jewel\|kraken\|luminous\)' | grep '\(mimic\|nautilus\)'
```

as we are not likely to have both "jewel" and "mimic" in a branch name.

this change is a follow-up of the discussion when reviewing #1345

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1347 from ceph/jan--f-add-notifications

add Jans email for nightly notifications

Merge branch 'master' into jan--f-add-notifications

Merge pull request #1348 from ceph/master-fix

ceph-dev-trigger: add missing escape for grep on master

ceph-dev-trigger: add missing escape for grep on master

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add Jans email for nightly notifications

Merge pull request #1345 from ceph/rm-xenial

ceph-dev-*: remove xenial for master and octopus

ceph-dev-trigger: add conditional step for master and octopus

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-dev-new-trigger: remove xenial for master and development branches

Signed-off-by: Alfredo Deza <adeza@redhat.com>

nfs-ganesha: Disable LizardFS FSAL

LizardFS FSAL has been added in [1] and it's enabled by default. This
is currently failing in the CI.

CMake Error at CMakeLists.txt:595 (message):
  STRICT_PACKAGE: Cannot find LizardFS client lib.  Disabling lizardfs fsal

This patch configures the USE_FSAL_LIZARDFS option to OFF.

[1] https://github.com/nfs-ganesha/nfs-ganesha/commit/884ec97

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1329 from ceph/ceph_container_nightly_devel_build

Add ceph container nightly devel build job

Merge branch 'master' into ceph_container_nightly_devel_build

Merge pull request #1342 from ceph/ceph-volume-nautilus

ceph-volume-test: adds on-evaluation-failure to the simple build step

ceph-volume-test: adds on-evaluation-failure to the simple build step

This makes the job skip the simple tests if the condition fails instead
of failing the run.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #1339 from ceph/ceph_ansible_nightly_registry_pagesize

ceph-ansible-nightly: update tags pagination

Merge branch 'master' into ceph_ansible_nightly_registry_pagesize

Merge pull request #1341 from ceph/ceph-volume-nautilus

ceph-volume-test: only run simple tests for luminous and mimic

ceph-volume-test: only run simple tests for luminous and mimic

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #1340 from ceph/ceph-volume-nautilus

ceph-volume: fix logic when deciding to run simple tests

ceph-volume: fix logic when deciding to run simple tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-ansible-nightly: update tags pagination

Set the page size to 100 instead of 10 (default).
Also we should add an extra page when the remainder of the modulo
operation between the tag count and page size isn't zero.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1337 from ceph/guits-install_jq

jjb: install jq package

jjb: actually fail if a command doesn't return 0

this sets `-euxo pipefail` to avoid this situation :

```
++ sort
++ grep -v jenkins-job-builder
++ jq -r '.jobs[].name'
/tmp/jenkins7521600802673204021.sh: line 1043: jq: command not found
++ curl -s https://2.jenkins.ceph.com/api/json
[PostBuildScript] - Executing post build scripts.
[jenkins-job-builder] $ /bin/sh -xe /tmp/jenkins7309655147676749318.sh
+ rm /home/jenkins-build/.jenkins_jobs.2.jenkins.ceph.com.ini
Finished: SUCCESS
```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1338 from ceph/guits-containers_releases

ceph-ansible: make it possible to run nighly job on custom branch

ceph-ansible: make it possible to run nighly job on custom branch

When running manually a nightly containerized job against a custom branch of
ceph-ansible, it fails because `CEPH_DOCKER_IMAGE_TAG` is not set.

Typical error:
```
+ case "$CEPH_ANSIBLE_BRANCH" in
+ CEPH_DOCKER_IMAGE_TAG=
+ '[' -n '' ']'
Build step 'Execute shell' marked build as failure
```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1336 from ceph/guits-remove_oldstable_jobs

ceph-ansible: remove old stable jobs

ceph-ansible: remove old stable jobs

This commit removes the (very) old stable jobs.
These jobs can't be launched, we have to use the new matrix.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1335 from ceph/guits-ubuntu_def

ceph-ansible: remove unused ubuntu jobs

ceph-ansible: remove unused ubuntu jobs

This commit removes some ubuntu job definition which are unused.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1333 from ceph/wip-ok-dan

ceph-dev-*: Clarify job descriptions

ceph-dev-*: Clarify job descriptions

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1332 from ceph/ceph-volume-nautilus

ceph-volume: only test simple on mimic and luminous branches

ceph-volume: only test simple on mimic and luminous branches

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #1331 from ceph/ceph-volume-nautilus

ceph-volume: add nautilus to nightly testing

ceph-volume: add nautilus to nightly testing

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #1330 from tchaikov/wip-40646

ceph-*build/build/setup_rpm: downgrade to gcc-c++-8.2.1

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph-*build/build/setup_rpm: downgrade to gcc-c++-8.2.1

Fixes: http://tracker.ceph.com/issues/40646
Signed-off-by: Kefu Chai <kchai@redhat.com>

Add ceph container nightly devel build job

The ceph container images are only built on github event and also based
on stable ceph content.
We can also have nightly builds with ceph developement content from
shaman/chacra.

See https://github.com/ceph/ceph-container/pull/1411

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1327 from rishabh-d-dave/add-shrink_rgw-to-CI

ceph-ansible: add shrink_rgw scenario test to CI

Merge branch 'master' into add-shrink_rgw-to-CI

Merge pull request #1328 from tchaikov/wip-gcc-9-is-nice-but

scripts/build_utils.sh: only use GCC-9 for building crimson

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

scripts/build_utils.sh: only use GCC-9 for building crimson

using GCC-9 for building mimic and other stable releases has following
consequences:

* it brings runtime dependencies of libstdc++ runtime shipped along with
  GCC-9
* GCC-9 is more picky, so some 3rd party libraries fail to build due to
  -Werror.

let's continue using GCC-8 on bionic when building non-crimson flavors

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-ansible: add shrink_rgw scenario test to CI

Add shrink_rgw scenario to CI so that it can be tested against master
and nautilus.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1326 from rishabh-d-dave/remove-shrink_mds-from-oldstable

ceph-ansible: remove shrink_mds from oldstable

Merge branch 'master' into remove-shrink_mds-from-oldstable

Merge pull request #1325 from rishabh-d-dave/add-shrink_rbdmirror-to-CI

ceph-ansible: add shrink_rbdmirror to CI

ceph-ansible: remove shrink_mds from oldstable

We don't need to run shrink_mds against "oldstable" releases.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

ceph-ansible: add shrink_rbdmirror to CI

Add shrink_rbdmirror scenario to CI so that it can be tested against
master and nautilus.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1324 from rishabh-d-dave/add-shrink_mgr-to-CI

ceph-ansible: add shrink_mgr scenario to CI

ceph-ansible: add shrink_mgr scenario to CI

Add shrink_mgr scenario to CI so that it can be tested against master
and nautilus.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1323 from ceph/guits-use_start_tox

ceph-ansible: build job name properly

ceph-ansible: build job name properly

Since tox.ini file has been split, this is needed to be able to run the
individual scenario testing. Those parameters are used to build the job
name passed to start_tox() function.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1322 from ceph/guits-use_start_tox

ceph-ansible: use start_tox function in ansible-scenario job

ceph-ansible: use start_tox function in ansible-scenario job

otherwise it always tries to run scenario using tox.ini only.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1320 from rishabh-d-dave/add-shrink_mds-to-CI

add shrink_mds scenario to CI

add shrink_mds scenario to CI

Add shrink_mds scenario to CI so that it can be tested against master.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1319 from tchaikov/wip-1315

ceph-dev-new-setup: missing pieces in PR#1315

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

scripts/build_utils.sh: add missing closing quote

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-setup: pass "--with seastar" to rpmbuild for crimson flavor build

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: dyna link against libgcc when building crimson

as seastar requires dynamic linkage of libgcc, see the comment for more
details. see also

https://github.com/scylladb/seastar/blob/master/src/core/exception_hacks.cc

without this change, we will have errors like:

/usr/lib/gcc/x86_64-linux-gnu/9/libgcc_eh.a(unwind-dw2.o): In function
`_Unwind_RaiseException':
(.text+0x24b0): multiple definition of `_Unwind_RaiseException'
../../../lib/libseastar.a(exception_hacks.cc.o):./obj-x86_64-linux-gnu/src/seastar/./src/seastar/include/seastar/core/future.hh:738:
first defined here
collect2: error: ld returned 1 exit status
src/test/crimson/CMakeFiles/unittest_seastar_lru.dir/build.make:140:
recipe for target 'bin/unittest_seastar_lru' failed

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: install gnupg

so that `apt-key add -` can import the pubkey

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-setup: move "git diff" before install-deps.sh

as install-deps.sh now changes debian/control if we are building the
crimson flavor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1315 from tchaikov/wip-crimson-flavor

scripts,ceph-dev-new*: add "crimson" flavor

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

scripts/build_utils.sh: update ubuntu-toolchain-r mirrors

the removed two do not have GCC-9, but the newly added mirros do.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: install GCC-9 on bionic

GCC-8 is good enough for build Ceph. but we are trying to focus on
1 year later's mainstream toolchain, and only the PPA of
"ubuntu-toolchain-r" offers GCC-9 for bionic, so let's just use GCC-9
for building all flavors on bionic. simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: add a param for "setup_pbuilder_for_ppa()"

allowing it to install other GCC than GCC-7

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new*: build "crimson" flavor for octopus and master

* add "crimson" as another flavor
* build "default", "notcmalloc" flavors for branches whose name
  match jewel, kraken, luminous, mimic, nautilus
* build "default", "notcmalloc" and "crimson" flavors for branches whose name
  does not match any of jewel, kraken, luminous, mimic, nautilus.
  and build "crimson" on bionic only, because the "ubuntu-toolchain-r"
  PPA offers GCC-9, which is the latest GCC release at the time of
  writing. we want to target the mainstream hardware and toolchain
  available at least 1 year later. by then, GCC-9 will be available
  in most distros. and hopefully, it will be included by the
  devtoolset-9.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-setup: add "crimson" flavor

currently, we have two flavors for ceph builds

1. "default", which is, well, the default build.
2. "notcmalloc" -- "no tcmalloc", originally, it was only for applease
   valgrind. but now, it is more like a Debug build, which is less
   performant, but it is compiled with some facilities enabled to help
   debugging.

as crimson is not yet production ready, a new flavor is added in this
change -- "crimson". this flavor is built only for testing crimson
in sepia nodes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-setup: set editor variables/options

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1318 from ceph/guits-modify_collect_log

ceph-ansible: improve collect ceph config file and logs

ceph-ansible: improve collect ceph config file and logs

let's collect these files in one task instead of two tasks.
By the way, it will avoid undesired failures regarding ceph config files
which don't exist (it first tried ceph.conf > test.conf >
mycluster.conf)

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1317 from ceph/ceph_ansible_dashboard_stable_4.0

ceph-ansible: add dashboard scenario to stable-4.0

ceph-ansible: add dashboard scenario to stable-4.0

This commit adds the dashboard scenario to nautilus deployment.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1316 from ceph/guits-add_ovh_nodes

ceph-ansible: use more slave in CI

ceph-ansible: use more slave in CI

we stopped to use OVH nodes because of an old ceph-disk race condition.
Since ceph-disk isn't present anymore as of nautilus, let's use ovh
nodes again.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1306 from ceph/wip-submodules-trigger

ceph-pr-submodules: add trigger phrase

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-pr-submodules: add trigger phrase

Signed-off-by: Nathan Cutler <ncutler@suse.com>