ceph-website: Update build script for go-live

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1848 from ceph/wip-website-dev

ceph-website: Use proper build env per branch

Merge pull request #1840 from ceph/wip-tags

ansible: Give registration tasks a tag

Merge pull request #1849 from ceph/wip-cleaner-args

ansible: Set tmpwatch/tmpcleaner args correctly

ceph-website: Use proper build env per branch

Fixes: https://github.com/ceph/ceph.io/issues/161
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1847 from ceph/wip-cic8

ceph-iscsi-*: Support CentOS8

ansible: Set tmpwatch/tmpcleaner args correctly

https://github.com/ceph/ceph-build/pull/1838#pullrequestreview-686910470

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1838 from ceph/wip-suse-cleaner

ansible: It's called tmpwatch in OpenSUSE

ceph-iscsi-*: Support CentOS8

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1845 from ceph/wip-debugmore

ceph-ansible-prs: Debug vagrant problems

Merge pull request #1846 from ceph/ca_split_jobs

ceph-ansible-prs: don't run all_in_one on smithi

ceph-ansible-prs: don't run all_in_one on smithi

smithi nodes aren't enough powerful for that job, let's run it on either
braggi or amadi

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-ansible-prs: Debug vagrant problems

Occasionally having VMs come up but not get an IP.  Or vagrant thinks they got an IP but didn't.  This is impossible to debug after the VMs in the job are torn down so let's spit out some debug info before that.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1844 from tchaikov/wip-revert-92b5739f271a263d58bd7fe6ef8c8000f3cb4204

Revert "add repo update for arrow installation (ubunto/centos)."

Revert "add repo update for arrow installation (ubunto/centos)."

curl and sudo are missing.

This reverts commit 92b5739f271a263d58bd7fe6ef8c8000f3cb4204.

Merge pull request #1843 from ceph/wip-c7-vl

ansible: Pin CentOS7 vagrant-libvirt to 0.3.0

ansible: Pin CentOS7 vagrant-libvirt to 0.3.0

So the vagrant-libvirt project itself admits that it does not support CentOS8.

Jobs /usually/ pass on the smithi builders (CentOS7.9, Vagrant 2.2.3, and vagrant-libvirt 0.0.40 or 0.3.0) so let's try running that everywhere.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1827 from galsalomon66/wip-test-repo-update-for-arrow

scripts/build_utils.sh: add package repo for apache arrow (enables parquet objects processing)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1841 from ceph/wip-pin-vagrant-libvirt

ansible: Pin vagrant-libvirt to 0.3.0

ansible: Pin vagrant-libvirt to 0.3.0

The latest version (and possibly older) 0.5.3 is giving us tons of issues.

- VMs hanging on `vagrant up`
- `vagrant ssh-config` failing

Not to mention we have to manually build libssh as well as krb5.

Let's not mess with all that.

With this PR, we also don't /need/ to run centos8-vagrant.yml.  It can be used to RE-install vagrant but the separate playbook is no longer required.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1836 from ceph/wip-c8-v2210

ansible: Pin centos8 vagrant builders to 2.2.10

Merge pull request #1839 from ceph/split_smithi_braggi

ceph-ansible-prs: use braggi for biggest jobs

ansible: Give registration tasks a tag

This is so I can skip it when I'm setting up a new CentOS 8 Vagrant builder.

https://wiki.sepia.ceph.com/doku.php?id=production:jenkins.ceph.com&#setting_up_new_jenkins_builder

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-ansible-prs: use braggi for biggest jobs

smithi and ovh workers aren't enough powerful, let's use braggi and
adami instead for those jobs.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ansible: It's called tmpwatch in OpenSUSE

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1837 from ceph/wip-remove-saved

scripts/build_utils.sh: Remove virsh save images

scripts/build_utils.sh: Remove virsh save images

Fixes

```
+ sudo virsh undefine container_rgw0_1623696405_fb24199d9194c338f3d6
error: Refusing to undefine while domain managed save image exists
```
and subsequent
```
+ sudo service libvirt-guests restart
Redirecting to /bin/systemctl restart libvirt-guests.service
Job for libvirt-guests.service failed because the control process exited with error code.
See "systemctl status libvirt-guests.service" and "journalctl -xe" for details.
Build step 'Conditional step (single)' marked build as failure

Jun 15 15:40:10 braggi19 libvirt-guests.sh[44513]: Resuming guest container_rgw0_1623696405_fb24199d9194c338f3d6: error: Failed to start domain container_rgw0_1623696405_fb24199d9194c338f3d6
Jun 15 15:40:10 braggi19 libvirt-guests.sh[44513]: error: Network not found: no network with matching name 'vagrant-libvirt'
Jun 15 15:40:11 braggi19 systemd[1]: libvirt-guests.service: Main process exited, code=exited, status=1/FAILURE
Jun 15 15:40:11 braggi19 systemd[1]: libvirt-guests.service: Failed with result 'exit-code'.
Jun 15 15:40:11 braggi19 systemd[1]: Failed to start Suspend/Resume Running libvirt Guests.
```

Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible: Pin centos8 vagrant builders to 2.2.10

Seeing weird random failures with 2.2.16.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1835 from ceph/wip-vagrantdestroy

ceph-{ansible,container}*: Globally destroy all stale VMs

ceph-{ansible,container}*: Globally destroy all stale VMs

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1834 from ceph/wip-c8-vagrant

ansible: Update centos8-vagrant playbook to work for CentOS8.4

Merge pull request #1833 from ceph/wip-feature_branches-master

pr-api: trigger on feature branches

ansible: Update centos8-vagrant playbook to work for CentOS8.4

Signed-off-by: David Galloway <dgallowa@redhat.com>

pr-api: trigger on feature branches

While other Jenkins jobs don't filter by specific branches, the API
tests do, so we need to support 'feature-*' branch syntax.

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

add repo update for arrow installation (ubunto/centos).

Signed-off-by: gal salomon <gal.salomon@gmail.com>

Merge pull request #1831 from tchaikov/wip-build-deb-mem

scripts/build_utils: increase mem per job to 3000 MiB

Merge pull request #1832 from tchaikov/wip-no-master-on-bionic

ceph-dev-cron: remove leap15 from comment for job building master

ceph-dev-cron: remove leap15 from comment for job building master

as we don't build master on leap15 anymore

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

ceph-dev-cron: do not build master on bionic

see also 2cbeb00308051188337773b587ea5e2913a42b56

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

scripts/build_utils: increase mem per job to 3000 MiB

see also https://github.com/ceph/ceph/pull/41677

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1830 from tchaikov/wip-no-bionic-on-master

ceph-dev-*-trigger: stop building master on bionic

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-dev-trigger: remove leap15 from comment for job building master

as we don't build master on leap15 anymore

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: always pass -DALLOCATOR

this change reverts 901fe2e097548707350bc89c7650e0f72c45d994 and
17b40974b1cb10d1d8ba0373dbf578fb0e82f036.

since we don't need to buld master (quincy and up) on bionic anymore,
there is no need to worry about the FTBFS when compiling with a buggy
gperftools (<2.6.2) shipped along with bionic.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-*-trigger: stop building master on bionic

in short, we don't need to test on bionic anymore for couple reasons:

- for accessing newer GCC
- for accessing more packages

see the discussion at
https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/BW5NKQ4KTOPV7OYUBUHVDVCSH72LVIEO/

the only qa suite referencing ubuntu 18.04 is the qa/suites/orch/rook/smoke,
but it deploys ceph using Rook, and the container image of Ceph is
based on CentOS, so stop building on ubuntu/bionic does not
break it, despite the fact that 18.04 is included by
qa/suites/orch/rook/smoke/0-distro.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1829 from tchaikov/wip-viva-la-venv

*/build/teardown: create venv for running teardown_vagrant_tests()

ceph-volume-nightly: do not set $scenarios

teardown_vagrant_tests() takes care of this already.

Signed-off-by: Kefu Chai <kchai@redhat.com>

*/build/teardown: create venv for running teardown_vagrant_tests()

to address the regression introduced by
e302eaabb977f3e93b1a7005c879f4b2eba581fc, which failed to create venv
after removing the global `$(mktemp -td venv.XXXXXXXXXX)` call.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1826 from dmick/wip-ceph-grafana

Add ceph-grafana job

Add ceph-grafana job to build ceph-grafana container.

This job builds from the ceph workspace based on a ceph branch and an
architecture (x86_64 or arm64).  A parent job calls this job for
branches or architectures.  More versions etc. could be parameterized,
but for now they're hardcoded in the ceph branch in
monitoring/grafana/build/Makefile.

Merge pull request #1825 from ceph/wip-kernel-sha

kernel: Export sha1 as an env variable so shaman build URLs work

kernel: Export sha1 as an env variable so shaman build URLs work

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1824 from ceph/wip-oklist

ceph-website: Allow Pete-Robelou to build beta.ceph.io

ceph-website: Allow Pete-Robelou to build beta.ceph.io

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1821 from tchaikov/wip-teuthology-pull-requests

teuthology-pull-requests: call create_venv_dir()

teuthology-pull-requests: call create_venv_dir()

it was an oversight in e302eaabb977f3e93b1a7005c879f4b2eba581fc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1820 from ceph/create_venv_ca_syntax_check

ceph-ansible-syntax: create virtualenv after dir creation

ceph-ansible-syntax: create virtualenv after dir creation

add back venv creation that has been removed accidentally by
commit 3d832da831f0b938faa6f845617b0257bda184f4

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1819 from ceph/fix_ca_syntax

ceph-ansible-syntax: fix broken job

ceph-ansible-syntax: fix broken job

commit b09f6470745a09ce4fce0265cb7a22a56b049886 supposed to fix
regression introduced by e302eaabb977f3e93b1a7005c879f4b2eba581fc missed
the ceph-ansible-pr-syntax-check job, so the CI is still failing like
following:

```
[ceph-ansible-pr-syntax-check] $ /bin/bash /tmp/jenkins6031146002439656501.sh
+ set -e
+ virtualenv
You must provide a DEST_DIR
Usage: virtualenv [OPTIONS] DEST_DIR
```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1818 from tchaikov/wip-ansible

ceph-ansible*: pass $TEMPVENV to start_tox

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-ansible*: pass $TEMPVENV to start_tox

to address the regression introduced by e302eaabb977f3e93b1a7005c879f4b2eba581fc

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #1816 from ceph/wip-kernel-deb-source-name

kernel: specify KDEB_SOURCENAME for make bindeb-pkg

kernel: specify KDEB_SOURCENAME for make bindeb-pkg

Undo the damage caused by linux.git commit 82526ef43399 ("kbuild:
deb-pkg: change the source package name to linux-upstream").

We don't build a source package, but the source package name gets
embedded in the package metadata and ends up affecting the paths in
the repository generated by chacra.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1815 from tchaikov/wip-venv-dir

*: use create_venv_dir() helper

Reviewed-by: David Galloway <dgallowa@redhat.com>

*: use create_venv_dir() helper

* define create_venv_dir() helper to
  1. create a temp dir
  2. set a trap to clean it up upon exit
* use create_venv_dir() in all scripts where
  TEMPVENV and/or VENV variables are used.

this change can help us to avoid creating a tempdir in /tmp,
and keep the tempdir forever.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1813 from tchaikov/wip-venv-cleanup

scripts/build_utils.sh: pass VENV to helper functions

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1699 from tchaikov/wip-ceph-pr-commits-cleanup

ceph-pr-commits: wrap lines over 79 chars

Merge pull request #1812 from ceph/wip-sunilkumarn417

teuthology-pull-requests: Allowlist sunilkumarn417

Merge pull request #1814 from tchaikov/wip-allocator-tcmalloc

ceph-dev-*-setup: do not specify ALLOCATOR

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

ceph-dev-*-setup: do not specify ALLOCATOR

$DIST env variable is not set by ceph-dev-new-setup, as it prepares
the dist tarball for building on different distros. to build with
tcmalloc if gperftools 2.6.2 is available, we leave it to cmake to
decide the allocator to use. because, if ALLOCATOR is not specified,
cmake first tries to find gperftools, and falls back to JeMalloc, and
the libc allocator. after quincy, the fix for gperftools <2.6.2 is
dropped, and gperftools >= 2.6.2 is required, so it's safe. before
quincy, the fix gperftools <2.6.2 is still included, so it's also safe.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1811 from tchaikov/wip-no-tcmalloc-bionic

ceph-dev-*-setup: use libc allocator when building quincy on bionic

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

ceph-dev-*-setup: use libc allocator when building quincy on bionic

see also https://tracker.ceph.com/issues/39703, to silence the warning
std::allocator<void> is deprecated in C++17 and will be removed in
C++20, so stop using allocator<void> in quincy and up.

but the libtcmalloc shipped by bionic is still buggy (v2.5), so we should not
build qunincy and up with tcmalloc on this distro.

after this change, we only build quincy and up with tcmalloc on non-bionic distro.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: pass VENV to helper functions

pass VENV to following functions explicitly:

* teardown_vagrant_tests()
* start_tox()
* build_debs()
* install_python_packages()
* install_python_packages_no_binary()
* pip_download()
* check_binary_existence()

prepare for the cleanup to avoid creating TEMPVENV everytime the script
is included.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: reindent start_tox()

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology-pull-requests: Allowlist sunilkumarn417

https://github.com/ceph/teuthology/pull/1646#issuecomment-841353102

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1810 from ceph/update_cephadm-ansible

cephadm-ansible: update scripts

cephadm-ansible: update scripts

This updates the build and teardown scripts so the playbook in
cephadm-ansible repository can be tested with the upstream ci.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1809 from ceph/cephadm-ansible

cephadm-ansible: add initial job

cephadm-ansible: add initial job

This adds a job definition for cephadm-ansible.
For now it doesn't run anything, it's just adding the definition into
jenkins so we can start setting up the corresponding repository
(cephadm-ansible).

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1807 from mgfritch/opensuse-leap-15.3

ceph-dev-build: update {build,setup}_osc

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1808 from tchaikov/wip-ceph-pr-docs-check

ceph-pr-docs: add annotation to build result

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

ceph-pr-docs: add annotation to build result

* do not "set -e", so the script does not exit immediately. instead
  use a variable of "succeed" to store true if the document is built
  successfully, and false otherwise
* feed the output of the sphinx-build to github-check tool, so it
  is able to parse and call the GitHub Check API to update the
  Check web page on the PR if the test fails.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1684 from tchaikov/wip-cbt-lint

ceph-cbt-lint: add cbt lint test

Reviewed-by: David Galloway <dgallowa@redhat.com>

ceph-dev-build: update {build,setup}_osc

use openSUSE Leap 15.3 repository for pacific and quincy

Fixes: https://tracker.ceph.com/issues/50563
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #1777 from ceph/drop_luminous_from_cv_nightly

ceph-volume-nightly: remove luminous

Merge pull request #1806 from ceph/fix_ganesha_builds

nfs-ganesha: fix broken builds

nfs-ganesha: fix broken builds

This job must be updated due to recent chances in nfs-ganesha (CMakeLists.txt)

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1805 from tchaikov/wip-debug-build-of-crimson

scripts/build_utils.sh: build Debug builds for crimson flavor

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

scripts/build_utils.sh: build Debug builds for crimson flavor

for ease of testing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: use "+=" for appending string to a variable

less repeating this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-*setup: extract ceph_build_args_from_flavor() out

so this helper can be shared by ceph-dev-new-setup and ceph-dev-setup
jobs.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1793 from ceph/wip-1772

sync-push: Use sort -V

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1804 from tchaikov/wip-no-ppa-on-bionic

scripts/build_utils.sh: do not use ppa on bionic

Reviewed-by: Casey Bodley <cbodley@redhat.com>

scripts/build_utils.sh: do not use ppa on bionic

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1803 from tchaikov/wip-pr-docs-cleanup

ceph-pr-docs: remove more leftover source.list

ceph-pr-docs: remove more leftover source.list

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1801 from ceph/guits-add-ca-nightly-fs2bs

ceph-ansible: add a nightly job

ceph-ansible: add a nightly job

This adds a missing nightly job 'filestore_to_bluestore'

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1800 from rzarzynski/wip-crimson-only-builds

ceph-dev-new-trigger: add a crimson-only building variant.

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-trigger: add a crimson-only building variant.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>