Merge pull request #1934 from ionutbalutoiu/update-ceph-windows-test

Add branch param to the `ceph-windows-test` job

Add branch param to the `ceph-windows-test` job

The default value is `master`, since Windows builds for the master
branch were enabled by https://github.com/ceph/ceph-build/pull/1928.

The `get-bin.py` script is updated to support `--branchname` parameter:
https://github.com/ceph/ceph-win32-tests/pull/1

Merge pull request #1927 from ionutbalutoiu/ceph-windows-pull-requests

Add ceph-windows-pull-requests job & Update ceph-windows-test job

Merge pull request #1932 from rhcs-dashboard/cephadm-e2e-node-v12

ceph-dashboard-cephadm-e2e job: update nodejs to v12

ceph-dashboard-cephadm-e2e job: update nodejs to v12

'ceph/make-dist' file and Angular 12 (https://github.com/ceph/ceph/pull/43404) require nodejs >= 12.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #1928 from ceph/wip-master-win

ceph-dev*: Start building Windows binaries for master

Merge pull request #1931 from ceph/ceph_container_lint_flake_always_pull

container-{flake8,lint}: always pull the image

container-{flake8,lint}: always pull the image

Otherwise we will still use an outdated container image.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Add ceph-windows-pull-requests job

Used to run Windows tests against Ceph pull requests.

Update ceph-windows-test job

* Create `scripts/ceph-windows` directory with the common scripts
  for the Ceph Windows testing.
* Move `run_tests` script to `scripts/ceph-windows`.
* Remove VM image caching from the `run_tests` script.
* Add `cleanup_tests` script to `scripts/ceph-windows`, and call
  this as a post-build step when the job is aborted.
* Add `get_chacra_build` build script, and refactor away this logic
  from the `run_tests` script.
* Move cleanup Windows tests function to `scripts/build_utils.sh`.

Merge pull request #1930 from ceph/ceph_container_lint_fix_pipe

ceph-container-lint: add missing pipe

ceph-container-lint: add missing pipe

The pipe was removed in the previous change but it shouldn't.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1929 from ceph/ceph_container_lint_fix_files_removed

ceph-container-lint: filter out file removed

ceph-container-lint: filter out file removed

We don't want to run shellcheck on shell script removed by a pull
request.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

ceph-dev*: Start building Windows binaries for master

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1922 from ionutbalutoiu/ceph-windows-test

Add Ceph Windows testing job

Add Ceph Windows testing job

Add a new freestyle Jenkins job that can be used to test a Ceph
Windows build uploaded to Chacra, on the Jenkins agents labeled with
`amd64`, `focal`, `libvirt`.

The Ceph Windows build is identified in Chacra using the following
Jenkins job parameters:
* `CEPH_WINDOWS_VERSION`
* `CEPH_WINDOWS_SHA1`

As a build step, the new job will execute the `./build/run_tests`
bash script. This script sets up a KVM virtual machine with Windows
Server 2019, that it's used to run the Ceph Windows unit tests from
the Chacra build. On top of that, it takes care of the everything
needed to run the Ceph Windows tests.

Merge pull request #1926 from ceph/https-not-git

ceph-dev-new-setup/build/build: git:// -> https://

ceph-dev-new-setup/build/build: git:// -> https://

https://github.blog/2021-09-01-improving-git-protocol-security-github/

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #1925 from ceph/cv_nightly_remove_myself_from_notif

ceph-volume-nightly: remove dsavinea

ceph-volume-nightly: remove dsavinea

This removes dsavinea@redhat.com from the recipients list.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1924 from ceph/ca_tox

ceph-ansible-prs: use right tox config file

ceph-ansible-prs: use right tox config file

use tox-subset_update.ini for the subset_update job.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1923 from ceph/add_ca_subset_update

ceph-ansible-prs: add new scenario subset_update

ceph-ansible-prs: add new scenario subset_update

see ceph-ansible corresponding pr: ceph/ceph-ansible#6951

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1921 from ceph/wip-pin-ansible

scripts: Set LC_ALL=en_US.UTF-8 again

scripts: Set LC_ALL=en_US.UTF-8 again

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1920 from ceph/wip-n-yuri

build_utils.sh: Disable CI_CONTAINER for nautilus too

build_utils.sh: Disable CI_CONTAINER for nautilus too

Brad and Yuri are working on something for downstream and need some branches built in ceph-ci.git based on the nautilus branch.  The builds keep showing up as failed in shaman because the container is failing to build.  This minor change fixes that.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1919 from dmick/wip-cephadm-binary

ceph-*-build: only try extracting cephadm if it exists

ceph-*-build: only try extracting cephadm if it exists

https://github.com/ceph/ceph-build/pull/1913 added code to extract
the cephadm binary from its package and push it to chacra; however,
older builds don't include a cephadm package, and ceph-build must
work for those as well.  Broke for nautilus.

Fixes: https://tracker.ceph.com/issues/52847
Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1918 from ceph/wip-container-arch

ceph-dev*build: Wait for the appropriate arch's repo to be ready

ceph-dev*build: Wait for the appropriate arch's repo to be ready

Also use shaman's search API.

What was happening before this was, for example:

- master branch gets built
- x86_64 job, build, and repo are done way before arm64
- arm64 job finally gets to the point where it's ready to build its container
- arm64 job checks with shaman to ask if the branch's repo is ready
- shaman answers with the x86_64 repo status and says it is ready
- arm64 job proceeds but fails because the arm64 repo is not ready

```
$ echo $RELEASE
8
$ echo $SHA1
d5759e030bea9732b905c51d041fe16c934790ce
$ echo $BRANCH
master
$ echo $FLAVOR
default
$ for ARCH in x86_64 arm64; do echo "============== $ARCH =============="; curl -s "https://shaman.ceph.com/api/search/?project=ceph&distros=centos/${RELEASE}/${ARCH}&sha1=${SHA1}&ref=${BRANCH}&flavor=${FLAVOR}" | jq -r '.[0].status'; done
============== x86_64 ==============
ready
============== arm64 ==============
ready

```

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1908 from ceph/fix-applitools_error-master

dashboard-e2e: fix applitools BATCH_ID error

Merge pull request #1913 from dmick/wip-cephadm-binary

ceph-*-build: add step to extract and store cephadm binary

dashboard-e2e: fix applitools BATCH_ID error

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1915 from ceph/wip-utf8

ansible: Set file.encoding=UTF-8

ansible: Set file.encoding=UTF-8

Requested by Ernesto via e-mail.  See https://stackoverflow.com/questions/27960996/jenkins-console-output-characters

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1914 from ceph/refact-ca-collect-logs

ceph-ansible-prs: refact collect-logs

ceph-ansible-prs: refact collect-logs

This would be easier to maintain this directly in ceph-ansible.
so let's move this playbook to ceph-ansible repo.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Fix tabs vs. spaces, add vim modeline for future vim users

Signed-off-by: Dan Mick <dmick@redhat.com>

ceph-*-build: add step to extract and store cephadm binary

For deb builds, this is all in build_utils.sh; for rpm, it's in
build_rpm for each build

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1912 from ceph/wip-focal-libvirt

ansible: CentOS and Ubuntu share libvirt service names again

Merge pull request #1911 from ceph/wip-jessie

ansible: Stop installing Jessie key

ansible: CentOS and Ubuntu share libvirt service names again

Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible: Stop installing Jessie key

We don't build for Jessie anymore

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1910 from tchaikov/wip-venv

scripts/build_utils.sh: use "python3 -m venv" to create venv

scripts/build_utils.sh: use "python3 -m venv" to create venv

we started to have following error recently:

```
+ local path=/tmp/venv.0D0Gwn9bMl
+ '[' -d /tmp/venv.0D0Gwn9bMl ']'
+ command -v python3
+ virtualenv -p python3 /tmp/venv.0D0Gwn9bMl
/tmp/jenkins5609477504730019125.sh: line 106: virtualenv: command not found
```

instead of relying on `virtualenv`, let's use the builtin venv
module to create virtualenv.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1907 from ceph/wip-c9

ansible: Modify builder playbook to support CentOS 9 Stream

ansible: Modify builder playbook to support CentOS 9 Stream

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1889 from ceph/wip-applitools

ansible: Install applitools-eyes plugin on Jenkins

Merge pull request #1890 from ceph/wip-applitools_wrapper-master

dashboard: add Applitools wrapper

Merge pull request #1896 from ceph/wip-18491

ceph-dev-*build: Notify shaman of a build sooner

Merge pull request #1902 from ceph/wip-bullseye

ceph: Build for bullseye

Merge pull request #1894 from ceph/wip-pull-sign-push-auto

Scripts that pull sign and pushes the nfs-ganesha packages on the signer

Changed the folder name used in the signer box

Signed-off-by: Adam Kraitman <akraitma@redhat.com>

Merge pull request #1906 from ceph/wip-website-many

ceph-website*: Keep 20 builds

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

ceph-website*: Keep 20 builds

They're very low cost and 1 day's worth of builds is not useful

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1905 from ceph/wip-makejenkinsbuilduser

chacra-pull-requests: Now set the ansible interpreter

chacra-pull-requests: Now set the ansible interpreter

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1904 from ceph/wip-postgres2

chacra-pull-requests: Install correct version of psycopg2

chacra-pull-requests: Install correct version of psycopg2

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1903 from ceph/wip-postgres

chacra-pull-requests: Define postgresql_version on >= xenial

chacra-pull-requests: Define postgresql_version on >= xenial

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph: Build for bullseye

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1900 from tchaikov/wip-build_utils-cleanups

scripts/build_utils: remove create_build_status()

Merge pull request #1901 from ceph/fix-applitools-nightly-master

dashboard-nightly-e2e: add APPLITOOLS_API_KEY

dashboard-nightly-e2e: add APPLITOOLS_API_KEY

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

scripts/build_utils: remove create_build_status()

it's identical to update_build_status(), apart from that it is only used
for creating the build status. but from the semantic point of view,
create is also an "update", IMHO. so consolidating them sounds like a
good move.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #1897 from tchaikov/wip-dep-cleanup

ceph*-buiild/build/setup_deb: create tmp dir in setup_rpm_build_deps()

Merge pull request #1899 from ceph/rename_container_push_jobs

container: update push jobs display name

Merge pull request #1898 from ceph/publish_daemon_container_image_on_quay.io

container: publish daemon,daemon-base to quay.io

container: consume quay.io creds for prs/nightly

The ceph-container PRs and nightly jobs also need to consume the quay.io
credentials as we need to pull base images from that registry.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

container: update push jobs display name

Because we don't push to Docker Hub registry anymore, we don't need to have
that in the job display name.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

container: publish daemon,daemon-base to quay.io

We're switching from docker.io registry for the ceph/{daemon,daemon-base}
container images to quay.io registry.
This requires to have a new credential-id in jenkins and update the
environment variables used by the ceph-container build script.

This impacts image tags like:
  - latest-xxxxx
  - latest-xxxxx-devel
  - latest-bis
  - vX.Y.Z-stable-X.Y

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

ceph*-buiild/build/setup_deb: create tmp dir in setup_rpm_build_deps()

better maintainability this way.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

ceph-dev-*build: Notify shaman of a build sooner

Fixes: https://tracker.ceph.com/issues/18491
Signed-off-by: David Galloway <dgallowa@redhat.com>

Scripts that pull sign and push the nfs-ganesha packages on the signer

Signed-off-by: Adam Kraitman <akraitma@redhat.com>

Merge pull request #1892 from rhcs-dashboard/cephadm-e2e-improv

ceph-dashboard-cephadm-e2e job: add E2E deps script

Merge pull request #1893 from ceph/guits_quick_update

cephadm-ansible: change scenario name

cephadm-ansible: use braggi or adami nodes

smithi nodes aren't enough powerful, let's use braggi or adami instead.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

cephadm-ansible: change scenario name

change the current scenario name

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-dashboard-cephadm-e2e job: add E2E deps script

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #1891 from rhcs-dashboard/cephadm-e2e-improv

ceph-dashboard-cephadm-e2e job: update KCLI image version

ceph-dashboard-cephadm-e2e job: update KCLI image version

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #1842 from rhcs-dashboard/dashboard-cephadm-e2e

project config: dashboard-cephadm-e2e

Merge pull request #1888 from tchaikov/wip-venv

ansible/examples/slave.yml: add python3-venv to python3_debs of debian

Merge pull request #1887 from tchaikov/wip-build_utils-cleanups

scripts/build_utils: cleanups

dashboard: add Applitools wrapper

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

ansible: Install applitools-eyes plugin on Jenkins

Fixes: https://tracker.ceph.com/issues/51996
Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible/examples/slave.yml: add python3-venv to python3_debs of debian

since we are going to replace virtualenv with "python3 -m venv",

let's add python3-venv.

keep python3-virtualenv around for a while just in case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: check retval of "command"

instead of checking the output of `command`, check its retval.
this is a more idiomatic way of using this tool.

and redirect its output to /dev/null. no news is good news.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: use "test" to test existence of a directory

otherwise we have scaring error message like:

++ ls -A /tmp/venv.flkiJIs23q
ls: cannot access '/tmp/venv.flkiJIs23q': No such file or directory
+

in build log of job of ceph-dev-new-build

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1886 from melissa-kun-li/asyncssh

Revert "ansible: install python3-asyncssh copr repo on EL8 for cephadm"

project config: dashboard-cephadm-e2e

Project: run Dashboard E2E tests with cephadm as orchestrator backend
  on dashboard/cephadm PRs.

Fixes: https://tracker.ceph.com/issues/51301
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Revert "ansible: install python3-asyncssh copr repo on EL8 for cephadm"

This reverts commit 733be223a72951db7e66eac0f28e379d83da54bd.

Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

Merge pull request #1885 from melissa-kun-li/asyncssh

ansible: install python3-asyncssh copr repo on EL8 for cephadm

ansible: install python3-asyncssh copr repo on EL8 for cephadm

install the copr repo so we can install the `python3-asyncssh` dependency for non-containerized tests

Fixes: https://tracker.ceph.com/issues/44676
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

Merge pull request #1857 from tchaikov/wip-reenable-arm64-pr

ceph-pull-requests-arm64: reenable test on arm64

Merge pull request #1884 from dmick/wip-ceph-grafana-trigger

ceph-grafana-trigger: change manifest list's name to final form