ceph-dashboard-cephadm-e2e job: add E2E deps script

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #1891 from rhcs-dashboard/cephadm-e2e-improv

ceph-dashboard-cephadm-e2e job: update KCLI image version

ceph-dashboard-cephadm-e2e job: update KCLI image version

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #1842 from rhcs-dashboard/dashboard-cephadm-e2e

project config: dashboard-cephadm-e2e

Merge pull request #1888 from tchaikov/wip-venv

ansible/examples/slave.yml: add python3-venv to python3_debs of debian

Merge pull request #1887 from tchaikov/wip-build_utils-cleanups

scripts/build_utils: cleanups

ansible/examples/slave.yml: add python3-venv to python3_debs of debian

since we are going to replace virtualenv with "python3 -m venv",

let's add python3-venv.

keep python3-virtualenv around for a while just in case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: check retval of "command"

instead of checking the output of `command`, check its retval.
this is a more idiomatic way of using this tool.

and redirect its output to /dev/null. no news is good news.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: use "test" to test existence of a directory

otherwise we have scaring error message like:

++ ls -A /tmp/venv.flkiJIs23q
ls: cannot access '/tmp/venv.flkiJIs23q': No such file or directory
+

in build log of job of ceph-dev-new-build

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1886 from melissa-kun-li/asyncssh

Revert "ansible: install python3-asyncssh copr repo on EL8 for cephadm"

project config: dashboard-cephadm-e2e

Project: run Dashboard E2E tests with cephadm as orchestrator backend
  on dashboard/cephadm PRs.

Fixes: https://tracker.ceph.com/issues/51301
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Revert "ansible: install python3-asyncssh copr repo on EL8 for cephadm"

This reverts commit 733be223a72951db7e66eac0f28e379d83da54bd.

Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

Merge pull request #1885 from melissa-kun-li/asyncssh

ansible: install python3-asyncssh copr repo on EL8 for cephadm

ansible: install python3-asyncssh copr repo on EL8 for cephadm

install the copr repo so we can install the `python3-asyncssh` dependency for non-containerized tests

Fixes: https://tracker.ceph.com/issues/44676
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

Merge pull request #1857 from tchaikov/wip-reenable-arm64-pr

ceph-pull-requests-arm64: reenable test on arm64

Merge pull request #1884 from dmick/wip-ceph-grafana-trigger

ceph-grafana-trigger: change manifest list's name to final form

ceph-grafana-trigger: change manifest list's name to final form

remove the -combined; a test shows the combined image is working
as expected on arm64

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1879 from dmick/wip-ceph-grafana-trigger

add ceph-grafana-trigger job

Merge pull request #1881 from tchaikov/ansible-release

ansible: replace version number in project(..VERSION..)

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1883 from ceph/publish_ceph_container_image_on_quay.io

ceph-container: publish ceph/ceph to quay.io

ceph-container: publish ceph/ceph to quay.io

We're switching from docker.io registry for the ceph/ceph container images
to quay.io registry.
This requires to have a new credential-id in jenkins and update the
environment variables used by the ceph-container build script.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1882 from ceph/wip-nofrng

ansible: fakeroot-ng doesn't exist anymore

ansible: fakeroot-ng doesn't exist anymore

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1880 from aaryanporwal/add-applitools-key

ceph-dashboard-pull-requests: Add Applitools key credential

ansible: replace version number in project(..VERSION..)

should be replace the VERSION number in project(...). this is the
authentic place to set version number.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dashboard-pull-requests: Add Applitools key credential

Signed-off-by: Aaryan Porwal <aaryanporwal2233@gmail.com>

add ceph-grafana-trigger job

Build both x86_64 and arm64 containers using ceph-grafana job,
and then pull, create a combined manifest, and push, currently to
both docker.io and quay.io

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1878 from tchaikov/wip-dist

ceph-dev-new-setup: populate DEB_BUILD_PROFILES using dist/other_envvars

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph-dev*-setup/build: consolidate logic to write dist/*

extract write_dist_files() function to consolidate logic to write dist/*
files.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-setup: populate DEB_BUILD_PROFILES using dist/other_envvars

see also cd55aaf0dba5620f3da228100a65ca6cf7e22883

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1877 from tchaikov/wip-deb-build-profiles

scripts/build_utils: populate DEB_BUILD_PROFILES using dist/other_env…

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

scripts/build_utils: populate DEB_BUILD_PROFILES using dist/other_envvars

* rename PROFILES to DEB_BUILD_PROFILES so it's more explicit that
  DEB_BUILD_PROFILES is for debian package build
* populate DEB_BUILD_PROFILES using dist/other_envvars, so that
  this environment variable can be populate to the next jenkins job.
  as dist/other_envvars is specified as one of the "properties-file"s
  in ceph-*build/config/definitions/ceph-build.yml and
* pass DEB_BUILD_PROFILES to pbuilder via an environment
  variable with the same name. more consistent this way, as
  DEB_BUILD_OPTIONS is also passed this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1875 from tchaikov/wip-quote-profile

scripts/build_utils.sh: quote $PROFILES

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

scripts/build_utils.sh: quote $PROFILES

otherwise the following "--use-network" would be taken as a profile,
and "yes" is used as the path to package's dsc file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1873 from tchaikov/wip-profiles-nocheck

scripts/build_utils: do not include nocheck in $PROFILES

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #1874 from tchaikov/wip-ceph-dev-new-trigger-escape-in-grep

ceph-dev-new-trigger:  escape ) by adding \ before it

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph-dev-new-trigger: add escape ) by adding \ before it

this should address the regression introduced by
99ac2b60d3e9c236802daf752d9924076151bf82

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: do not include nocheck in $PROFILES

we don't use nocheck for disabling the "make check" dependencies,
instead, <pkg.ceph.check> is used to enable "make check" dependencies.
so they are not enabled by default.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1870 from ideepika/crimson-pkg-build

build_utils: specify buildprofiles for crimson build

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1862 from ideepika/backup

build_utils: set extra args for jaeger based on branch name

Reviewed-by: Kefu Chai <kchai@redhat.com>

build_utils: check if PROFILE for pbuilder exists

since not sure if --profile accepts comma or not, added this check

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph-dev-new-trigger: add jaeger-only trigger

if branch will have `jaeger` in branch name, it shall only build
      focal, centos8 x86_64

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

build_utils: export jaeger build args if FLAVOR == jaeger

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #1871 from ceph/wip-cleanupcephio

Prune old ceph.io builds

update extrapkg needed for crimson, to be added by debian buildprofiles

cleanup: add/ remove some missing go to have comments

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #1872 from ceph/wip-arm64-focal

ceph-build: arm64 focal builds too

ceph-build: arm64 focal builds too

https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/GAIDYRWBOXC5FQD4H4UHUHZ6NLNSEKRC/

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-website*: Remove TODO

The README isn't going to be updated all that often.  It's not a big deal to run the job anyway.

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-website-prs: Clean up old builds

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1869 from ceph/wip-split-cephio

ceph-website: Split job into two (PRs and main)

ceph-website: Split job into two (PRs and main)

The PRs job will build PRs.

The non-PRs job will build the production website.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1868 from tchaikov/wip-build_utils-arrow-repo

scripts/build_utils: use apache repo hosted by apache.jfrog.io

Reviewed-by: Gal Salomon <gsalomon@redhat.com>

scripts/build_utils: use apache repo hosted by apache.jfrog.io

the http://dl.bintray.com/apache/arrow/ mirror returns 403 "Forbidden"
now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1867 from ceph/ceph_ansible_prs_purge_dashboard

ceph-ansible-prs: add purge-dashboard jobs

ceph-ansible-prs: add purge-dashboard jobs

This adds the purge-dashboards jobs for both container and non container
scenarios.
Those scenarios aren't triggered by default.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1865 from ceph/wip-wipeout

ceph-website: Build PR checkouts

ceph-website: Build PR checkouts

The previous way this was configured would only build branches from this repo; no forks.  When a PR is submitted from a fork, the job would just build main.  Oops.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1864 from ceph/ceph_ansible_prs_cephadm_adopt_hosts

ceph-ansible-prs: don't run cephadm jobs on smithi

ceph-ansible-prs: don't run cephadm jobs on smithi

smithi nodes aren't enough powerful for that job, let's run it on either
braggi or amadi.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1861 from ceph/wip-normaldistro

build_utils.sh: Fix variable

build_utils.sh: Fix variable

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1860 from ceph/auth_cephadm_ansible

cephadm-ansible: use dedicated credential-id

cephadm-ansible: use dedicated credential-id

This makes the cephadm-ansible job use the right credential id

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1850 from galsalomon66/update-repo-for-arrow-parquet-installation

scripts/build_utils.sh: add package repo for apache arrow

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1859 from ceph/drop_syntax_check

ceph-ansible: drop pr-syntax-check

ceph-ansible: drop pr-syntax-check

This is now done using GitHub workflows.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

script/build_utils: update repo for arrow and parquet installation;

Signed-off-by: gal salomon <gal.salomon@gmail.com>

Merge pull request #1858 from ceph/wip-modules

ceph-pr-submodules: This job can run anywhere

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-pr-submodules: This job can run anywhere

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-pull-requests-arm64: reenable test on arm64

since we now perform the "make check" test only on ubuntu focal, on
which all the dependencies are available on both aarch64 and amd64,
and the "make check" passes on this architecture also. let's test the
PRs on this architecture.

this change reverts 1e0b902baa33b3b972891779d2950e580bb18e63

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1856 from ceph/wip-orirvingi

ceph-volume-*: Run on irvingi too.

Merge pull request #1855 from ceph/c-v-test-suite-cleanup

ceph-volume-test: cleanup

ceph-volume-*: Run on irvingi too.

I added irvingi01 and irvingi02 as CentOS7 Vagrant builders today.  We very briefly tried to use DHC for builders but that crashed and burned.

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-volume-test: cleanup

Those jobs don't exist anymore in tox.ini of ceph-volume functional
tests. Let's remove them.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1854 from ceph/fix_missing_subcommand

ceph-volume-test: add missing SUBCOMMAND var

ceph-volume-test: add missing SUBCOMMAND var

SUBCOMMAND is missing for `centos7-bluestore-mixed_type` test.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1853 from ceph/fix_cv_test_suite

ceph-volume-test: add missing SUBCOMMAND definition

ceph-volume-test: add missing SUBCOMMAND definition

This is making some tests fail in the CI like following:

```
+ /tmp/venv.LBZdiW0Nx4/bin/tox --workdir=/tmp/tox.pQhNDR2lrs -vre
  xenial-bluestore-mixed_type -- --provider=libvirt
  using tox.ini:
  /home/jenkins-build/build/workspace/ceph-volume-scenario/src/ceph-volume/ceph_volume/tests/functional/lvm/tox.ini
  (pid 1864)
  ERROR: unknown environment 'xenial-bluestore-mixed_type'
```

it's picking up the wrong tox.ini file because SUBCOMMAND isn't injected
as it should be.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1852 from ceph/wip-naginator

ansible: Install naginator on Jenkins master

ceph-ansible-prs: Retry failed builds w/ naginator

Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible: Install naginator on Jenkins master

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1851 from ceph/wip-website-live

ceph-website: Update build script for go-live

ceph-website: Update build script for go-live

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1848 from ceph/wip-website-dev

ceph-website: Use proper build env per branch

Merge pull request #1840 from ceph/wip-tags

ansible: Give registration tasks a tag

Merge pull request #1849 from ceph/wip-cleaner-args

ansible: Set tmpwatch/tmpcleaner args correctly

ceph-website: Use proper build env per branch

Fixes: https://github.com/ceph/ceph.io/issues/161
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1847 from ceph/wip-cic8

ceph-iscsi-*: Support CentOS8

ansible: Set tmpwatch/tmpcleaner args correctly

https://github.com/ceph/ceph-build/pull/1838#pullrequestreview-686910470

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1838 from ceph/wip-suse-cleaner

ansible: It's called tmpwatch in OpenSUSE

ceph-iscsi-*: Support CentOS8

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1845 from ceph/wip-debugmore

ceph-ansible-prs: Debug vagrant problems

Merge pull request #1846 from ceph/ca_split_jobs

ceph-ansible-prs: don't run all_in_one on smithi

ceph-ansible-prs: don't run all_in_one on smithi

smithi nodes aren't enough powerful for that job, let's run it on either
braggi or amadi

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-ansible-prs: Debug vagrant problems

Occasionally having VMs come up but not get an IP.  Or vagrant thinks they got an IP but didn't.  This is impossible to debug after the VMs in the job are torn down so let's spit out some debug info before that.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1844 from tchaikov/wip-revert-92b5739f271a263d58bd7fe6ef8c8000f3cb4204

Revert "add repo update for arrow installation (ubunto/centos)."

Revert "add repo update for arrow installation (ubunto/centos)."

curl and sudo are missing.

This reverts commit 92b5739f271a263d58bd7fe6ef8c8000f3cb4204.

Merge pull request #1843 from ceph/wip-c7-vl

ansible: Pin CentOS7 vagrant-libvirt to 0.3.0

ansible: Pin CentOS7 vagrant-libvirt to 0.3.0

So the vagrant-libvirt project itself admits that it does not support CentOS8.

Jobs /usually/ pass on the smithi builders (CentOS7.9, Vagrant 2.2.3, and vagrant-libvirt 0.0.40 or 0.3.0) so let's try running that everywhere.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1827 from galsalomon66/wip-test-repo-update-for-arrow

scripts/build_utils.sh: add package repo for apache arrow (enables parquet objects processing)

Reviewed-by: Kefu Chai <kchai@redhat.com>