ansible: Stop installing Jessie key

We don't build for Jessie anymore

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1910 from tchaikov/wip-venv

scripts/build_utils.sh: use "python3 -m venv" to create venv

scripts/build_utils.sh: use "python3 -m venv" to create venv

we started to have following error recently:

```
+ local path=/tmp/venv.0D0Gwn9bMl
+ '[' -d /tmp/venv.0D0Gwn9bMl ']'
+ command -v python3
+ virtualenv -p python3 /tmp/venv.0D0Gwn9bMl
/tmp/jenkins5609477504730019125.sh: line 106: virtualenv: command not found
```

instead of relying on `virtualenv`, let's use the builtin venv
module to create virtualenv.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1907 from ceph/wip-c9

ansible: Modify builder playbook to support CentOS 9 Stream

ansible: Modify builder playbook to support CentOS 9 Stream

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1889 from ceph/wip-applitools

ansible: Install applitools-eyes plugin on Jenkins

Merge pull request #1890 from ceph/wip-applitools_wrapper-master

dashboard: add Applitools wrapper

Merge pull request #1896 from ceph/wip-18491

ceph-dev-*build: Notify shaman of a build sooner

Merge pull request #1902 from ceph/wip-bullseye

ceph: Build for bullseye

Merge pull request #1894 from ceph/wip-pull-sign-push-auto

Scripts that pull sign and pushes the nfs-ganesha packages on the signer

Changed the folder name used in the signer box

Signed-off-by: Adam Kraitman <akraitma@redhat.com>

Merge pull request #1906 from ceph/wip-website-many

ceph-website*: Keep 20 builds

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

ceph-website*: Keep 20 builds

They're very low cost and 1 day's worth of builds is not useful

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1905 from ceph/wip-makejenkinsbuilduser

chacra-pull-requests: Now set the ansible interpreter

chacra-pull-requests: Now set the ansible interpreter

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1904 from ceph/wip-postgres2

chacra-pull-requests: Install correct version of psycopg2

chacra-pull-requests: Install correct version of psycopg2

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1903 from ceph/wip-postgres

chacra-pull-requests: Define postgresql_version on >= xenial

chacra-pull-requests: Define postgresql_version on >= xenial

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph: Build for bullseye

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1900 from tchaikov/wip-build_utils-cleanups

scripts/build_utils: remove create_build_status()

Merge pull request #1901 from ceph/fix-applitools-nightly-master

dashboard-nightly-e2e: add APPLITOOLS_API_KEY

dashboard-nightly-e2e: add APPLITOOLS_API_KEY

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

scripts/build_utils: remove create_build_status()

it's identical to update_build_status(), apart from that it is only used
for creating the build status. but from the semantic point of view,
create is also an "update", IMHO. so consolidating them sounds like a
good move.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #1897 from tchaikov/wip-dep-cleanup

ceph*-buiild/build/setup_deb: create tmp dir in setup_rpm_build_deps()

Merge pull request #1899 from ceph/rename_container_push_jobs

container: update push jobs display name

Merge pull request #1898 from ceph/publish_daemon_container_image_on_quay.io

container: publish daemon,daemon-base to quay.io

container: consume quay.io creds for prs/nightly

The ceph-container PRs and nightly jobs also need to consume the quay.io
credentials as we need to pull base images from that registry.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

container: update push jobs display name

Because we don't push to Docker Hub registry anymore, we don't need to have
that in the job display name.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

container: publish daemon,daemon-base to quay.io

We're switching from docker.io registry for the ceph/{daemon,daemon-base}
container images to quay.io registry.
This requires to have a new credential-id in jenkins and update the
environment variables used by the ceph-container build script.

This impacts image tags like:
  - latest-xxxxx
  - latest-xxxxx-devel
  - latest-bis
  - vX.Y.Z-stable-X.Y

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

ceph*-buiild/build/setup_deb: create tmp dir in setup_rpm_build_deps()

better maintainability this way.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

ceph-dev-*build: Notify shaman of a build sooner

Fixes: https://tracker.ceph.com/issues/18491
Signed-off-by: David Galloway <dgallowa@redhat.com>

Scripts that pull sign and push the nfs-ganesha packages on the signer

Signed-off-by: Adam Kraitman <akraitma@redhat.com>

Merge pull request #1892 from rhcs-dashboard/cephadm-e2e-improv

ceph-dashboard-cephadm-e2e job: add E2E deps script

Merge pull request #1893 from ceph/guits_quick_update

cephadm-ansible: change scenario name

cephadm-ansible: use braggi or adami nodes

smithi nodes aren't enough powerful, let's use braggi or adami instead.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

cephadm-ansible: change scenario name

change the current scenario name

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-dashboard-cephadm-e2e job: add E2E deps script

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #1891 from rhcs-dashboard/cephadm-e2e-improv

ceph-dashboard-cephadm-e2e job: update KCLI image version

ceph-dashboard-cephadm-e2e job: update KCLI image version

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #1842 from rhcs-dashboard/dashboard-cephadm-e2e

project config: dashboard-cephadm-e2e

Merge pull request #1888 from tchaikov/wip-venv

ansible/examples/slave.yml: add python3-venv to python3_debs of debian

Merge pull request #1887 from tchaikov/wip-build_utils-cleanups

scripts/build_utils: cleanups

dashboard: add Applitools wrapper

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

ansible: Install applitools-eyes plugin on Jenkins

Fixes: https://tracker.ceph.com/issues/51996
Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible/examples/slave.yml: add python3-venv to python3_debs of debian

since we are going to replace virtualenv with "python3 -m venv",

let's add python3-venv.

keep python3-virtualenv around for a while just in case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: check retval of "command"

instead of checking the output of `command`, check its retval.
this is a more idiomatic way of using this tool.

and redirect its output to /dev/null. no news is good news.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: use "test" to test existence of a directory

otherwise we have scaring error message like:

++ ls -A /tmp/venv.flkiJIs23q
ls: cannot access '/tmp/venv.flkiJIs23q': No such file or directory
+

in build log of job of ceph-dev-new-build

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1886 from melissa-kun-li/asyncssh

Revert "ansible: install python3-asyncssh copr repo on EL8 for cephadm"

project config: dashboard-cephadm-e2e

Project: run Dashboard E2E tests with cephadm as orchestrator backend
  on dashboard/cephadm PRs.

Fixes: https://tracker.ceph.com/issues/51301
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Revert "ansible: install python3-asyncssh copr repo on EL8 for cephadm"

This reverts commit 733be223a72951db7e66eac0f28e379d83da54bd.

Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

Merge pull request #1885 from melissa-kun-li/asyncssh

ansible: install python3-asyncssh copr repo on EL8 for cephadm

ansible: install python3-asyncssh copr repo on EL8 for cephadm

install the copr repo so we can install the `python3-asyncssh` dependency for non-containerized tests

Fixes: https://tracker.ceph.com/issues/44676
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

Merge pull request #1857 from tchaikov/wip-reenable-arm64-pr

ceph-pull-requests-arm64: reenable test on arm64

Merge pull request #1884 from dmick/wip-ceph-grafana-trigger

ceph-grafana-trigger: change manifest list's name to final form

ceph-grafana-trigger: change manifest list's name to final form

remove the -combined; a test shows the combined image is working
as expected on arm64

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1879 from dmick/wip-ceph-grafana-trigger

add ceph-grafana-trigger job

Merge pull request #1881 from tchaikov/ansible-release

ansible: replace version number in project(..VERSION..)

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1883 from ceph/publish_ceph_container_image_on_quay.io

ceph-container: publish ceph/ceph to quay.io

ceph-container: publish ceph/ceph to quay.io

We're switching from docker.io registry for the ceph/ceph container images
to quay.io registry.
This requires to have a new credential-id in jenkins and update the
environment variables used by the ceph-container build script.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1882 from ceph/wip-nofrng

ansible: fakeroot-ng doesn't exist anymore

ansible: fakeroot-ng doesn't exist anymore

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1880 from aaryanporwal/add-applitools-key

ceph-dashboard-pull-requests: Add Applitools key credential

ansible: replace version number in project(..VERSION..)

should be replace the VERSION number in project(...). this is the
authentic place to set version number.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dashboard-pull-requests: Add Applitools key credential

Signed-off-by: Aaryan Porwal <aaryanporwal2233@gmail.com>

add ceph-grafana-trigger job

Build both x86_64 and arm64 containers using ceph-grafana job,
and then pull, create a combined manifest, and push, currently to
both docker.io and quay.io

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1878 from tchaikov/wip-dist

ceph-dev-new-setup: populate DEB_BUILD_PROFILES using dist/other_envvars

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph-dev*-setup/build: consolidate logic to write dist/*

extract write_dist_files() function to consolidate logic to write dist/*
files.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-setup: populate DEB_BUILD_PROFILES using dist/other_envvars

see also cd55aaf0dba5620f3da228100a65ca6cf7e22883

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1877 from tchaikov/wip-deb-build-profiles

scripts/build_utils: populate DEB_BUILD_PROFILES using dist/other_env…

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

scripts/build_utils: populate DEB_BUILD_PROFILES using dist/other_envvars

* rename PROFILES to DEB_BUILD_PROFILES so it's more explicit that
  DEB_BUILD_PROFILES is for debian package build
* populate DEB_BUILD_PROFILES using dist/other_envvars, so that
  this environment variable can be populate to the next jenkins job.
  as dist/other_envvars is specified as one of the "properties-file"s
  in ceph-*build/config/definitions/ceph-build.yml and
* pass DEB_BUILD_PROFILES to pbuilder via an environment
  variable with the same name. more consistent this way, as
  DEB_BUILD_OPTIONS is also passed this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1875 from tchaikov/wip-quote-profile

scripts/build_utils.sh: quote $PROFILES

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

scripts/build_utils.sh: quote $PROFILES

otherwise the following "--use-network" would be taken as a profile,
and "yes" is used as the path to package's dsc file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1873 from tchaikov/wip-profiles-nocheck

scripts/build_utils: do not include nocheck in $PROFILES

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #1874 from tchaikov/wip-ceph-dev-new-trigger-escape-in-grep

ceph-dev-new-trigger:  escape ) by adding \ before it

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph-dev-new-trigger: add escape ) by adding \ before it

this should address the regression introduced by
99ac2b60d3e9c236802daf752d9924076151bf82

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: do not include nocheck in $PROFILES

we don't use nocheck for disabling the "make check" dependencies,
instead, <pkg.ceph.check> is used to enable "make check" dependencies.
so they are not enabled by default.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1870 from ideepika/crimson-pkg-build

build_utils: specify buildprofiles for crimson build

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1862 from ideepika/backup

build_utils: set extra args for jaeger based on branch name

Reviewed-by: Kefu Chai <kchai@redhat.com>

build_utils: check if PROFILE for pbuilder exists

since not sure if --profile accepts comma or not, added this check

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph-dev-new-trigger: add jaeger-only trigger

if branch will have `jaeger` in branch name, it shall only build
      focal, centos8 x86_64

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

build_utils: export jaeger build args if FLAVOR == jaeger

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #1871 from ceph/wip-cleanupcephio

Prune old ceph.io builds

update extrapkg needed for crimson, to be added by debian buildprofiles

cleanup: add/ remove some missing go to have comments

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #1872 from ceph/wip-arm64-focal

ceph-build: arm64 focal builds too

ceph-build: arm64 focal builds too

https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/GAIDYRWBOXC5FQD4H4UHUHZ6NLNSEKRC/

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-website*: Remove TODO

The README isn't going to be updated all that often.  It's not a big deal to run the job anyway.

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-website-prs: Clean up old builds

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1869 from ceph/wip-split-cephio

ceph-website: Split job into two (PRs and main)

ceph-website: Split job into two (PRs and main)

The PRs job will build PRs.

The non-PRs job will build the production website.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1868 from tchaikov/wip-build_utils-arrow-repo

scripts/build_utils: use apache repo hosted by apache.jfrog.io

Reviewed-by: Gal Salomon <gsalomon@redhat.com>

scripts/build_utils: use apache repo hosted by apache.jfrog.io

the http://dl.bintray.com/apache/arrow/ mirror returns 403 "Forbidden"
now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1867 from ceph/ceph_ansible_prs_purge_dashboard

ceph-ansible-prs: add purge-dashboard jobs

ceph-ansible-prs: add purge-dashboard jobs

This adds the purge-dashboards jobs for both container and non container
scenarios.
Those scenarios aren't triggered by default.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1865 from ceph/wip-wipeout

ceph-website: Build PR checkouts

ceph-website: Build PR checkouts

The previous way this was configured would only build branches from this repo; no forks.  When a PR is submitted from a fork, the job would just build main.  Oops.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1864 from ceph/ceph_ansible_prs_cephadm_adopt_hosts

ceph-ansible-prs: don't run cephadm jobs on smithi

ceph-ansible-prs: don't run cephadm jobs on smithi

smithi nodes aren't enough powerful for that job, let's run it on either
braggi or amadi.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1861 from ceph/wip-normaldistro

build_utils.sh: Fix variable

build_utils.sh: Fix variable

Signed-off-by: David Galloway <dgallowa@redhat.com>