setup_pbuilder_for_new_gcc doesnt mention version number

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #1969 from ceph/quincy

ceph-dev-cron: Drop nautilus | Add quincy

ceph-dev-cron: Drop nautilus | Add quincy

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1967 from ceph/revert-207adf189

Revert "use alpha tag for ceph-ansible until we have a stable-7.0 bra…

Revert "use alpha tag for ceph-ansible until we have a stable-7.0 branch"

This reverts commit 207adf189e25753bfc68b2998d430b16fb50d389.

This breaks ceph-volume/ceph-ansible testing.

`v7.0.0alpha1` is a tag present on stable-6.0.
ceph-ansible stable-6.0 doesn't support quincy.

We have to keep using ceph-ansible@master for now.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1964 from vumrao/wip-vumrao-quincy

add quincy targets

use alpha tag for ceph-ansible until we have a stable-7.0 branch

Signed-off-by: Vikhyat Umrao <vikhyat@redhat.com>

add quincy targets

Fixes: https://tracker.ceph.com/issues/54134
Signed-off-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge pull request #1962 from ceph/just-container

ceph-dev-container-only: Create job

Merge pull request #1965 from rhcs-dashboard/dashboard-e2e-dep-script

dep: delete the network before creating it again

dep: delete the network before creating it again

Need to recreate the network to avoid issues like ssh ing into the
environment.

Fixes: https://tracker.ceph.com/issues/54131
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #1937 from ceph/14

ansible: Clean up tmp files older than 14

Merge pull request #1916 from ceph/wip-jenkins-default

ansible: Update Jenkins master java args

Merge pull request #1959 from ceph/disable-docs

{ceph-pr-render,teuthology}-docs: Disable job

ceph-dev-container-only: Create job

This job will allow users to build a container for https://quay.ceph.io/repository/ceph-ci/ceph?tab=tags given a branch name.  Should work for ceph.git and ceph-ci.git.

This is mostly useful for when packages and a repo successfully get built for a ceph-dev*build job but the container build fails.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1917 from ceph/wip-more-builds

ceph-dev-cron: Keep more builds

Merge pull request #1961 from ceph/no-dockermirror

ceph-dev*build: Stop using docker-mirror

Merge pull request #1963 from ceph/wip-kernel-get-rpm-dist

kernel: make CentOS Stream builds work

kernel: drop local get_rpm_dist() definition

The local definition is missing a couple of things, most notably
CentOSStream addition from commit d4dfd1210124 ("scripts: Support
CentOS Stream").  This fixes repository creation breakage caused
by DIST and DISTRO set to unknown.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

scripts/build_utils: set DIST for fedora

Set it to what it was set in a duplicate definition that got
removed in commit 8e83d2dd1b4b ("scripts/build_utils: remove
duplicated get_rpm_dist()").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-dev*build: Stop using docker-mirror

Non-stream CentOS8 containers no longer work because the BaseOS and AppStream repos moved.  We now use the centos/centos:stream8 container image as our BASEOS_TAG.  Except that container lives on quay.io and not dockerhub.

docker-mirror.front.sepia.ceph.com:5000 can only mirror one registry.  We'll keep it as mirroring dockerhub.

Signed-off-by: David Galloway <dgallowa@redhat.com>

{ceph-pr-render,teuthology}-docs: Disable job

We're using RTD now.  The docs Jenkins builder is retired.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1958 from ceph/wip-fix7core

scripts: Support CentOS Stream

scripts: Support CentOS Stream

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1955 from cbodley/wip-arrow

arrow: remove apache jfrog repositories

Merge pull request #1948 from chrisphoffman/wip-classic-libarchive-zstd

ceph-perf-pull-requests: update libarchive

Merge pull request #1956 from ceph/api-quincy

api: allowlist quincy

api: allowlist quincy

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

revert the apache.jfrog.io repos

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #1954 from rhcs-dashboard/add-quincy-dash-nightly

config: add quincy to the dashboard nightlies

config: add quincy to the dashboard nightlies

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #1952 from dmick/wip-cephadm-package-test

ceph-{,dev,dev-new} build: fix conditional for cephadm package test

ceph-{,dev,dev-new} build: fix conditional for cephadm package test

https://github.com/ceph/ceph-build/pull/1919 added a conditional
test to check for cephadm package existence (so the code to extract
cephadm could work on older builds that did not have a cephadm package).
However, it used bash's [[ operator, and that doesn't expand file globs
in its arguments, so it effectively stopped creating the cephadm
binary altogether.

Switch to [, which doesn't have that problem.

Signed-off-by: Dan Mick <dmick@redhat.com>

ceph-perf-pull-requests: update libarchive

Ensure version of libarchive is installed due to cmake dependency: https://access.redhat.com/solutions/6068431
Error message encountered:
cmake3: symbol lookup error: cmake3: undefined symbol: archive_write_add_filter_zstd

Related: https://github.com/ceph/ceph/pull/42294

Signed-off-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #1951 from ionutbalutoiu/add-daily-triggers

Add triggers for `wnbd-build` and `ceph-windows-installer-build`

Add triggers for `wnbd-build` and `ceph-windows-installer-build`

Add daily triggers for `wnbd-build` and `ceph-windows-installer-build`.

The ceph-windows installer trigger time is offset by 1 hour to
allow the `wnbd` build to be uploaded to Chacra.

Merge pull request #1949 from ionutbalutoiu/ceph-windows-image-build

[ceph-windows-image-build] Add job to build the CI Windows image

[ceph-windows-image-build] Add job to build the CI Windows image

Adds a new freestyle Jenkins job that builds, unattended, the Windows
Server 2019 qcow2 image used in the Ceph CI with libvirt.

Merge pull request #1950 from ceph/cephadm-ansible-creds

cephadm-ansible: add new credentials

cephadm-ansible: add new credentials

This will allow using default images from cephadm instead of
maintaining what needs to be used in the tox.ini file of cephadm-ansible.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1945 from rhcs-dashboard/cephadm-e2e-nightly-fix

ceph-dashboard-cephadm-e2e-job: update workspace path

Merge pull request #1946 from chrisphoffman/wip-llvmtoolset-repo-method

ceph-perf-pull-requests: enable llvm-toolset-10 repo using distribution-version-architecture format.

ceph-perf-pull-requests: enable llvm-toolset-10 repo using
distribution-version-architecture format.

Signed-off-by: Christopher Hoffman <choffman@redhat.com>

ceph-dashboard-cephadm-e2e-job: update workspace path

In the nightly runs we saw that the ceph_cluster.yml file which we used
to deploy kcli plans is not up-to-date with the master one. It cause
problems in the nightlies. We've narrowed down the problem is with the basedir
and and also with the path difference in the dependency script and the location of
ceph. So changing the workspace path to workspace/ceph.

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #1941 from ionutbalutoiu/ceph-windows-installer-job

[ceph-windows-installer-build] Add Jenkins job to build the ceph-windows-installer project

Merge pull request #1944 from ionutbalutoiu/refactor-windows-jobs

Refactor `ceph-windows-test` and `ceph-windows-pull-requests` jobs

Add `ceph-windows-installer-build` job to build Ceph Windows installer

The `ceph-windows-installer` project creates a Windows MSI installer
that can be used to setup Ceph on Windows.

It bundles the `wnbd` driver, and the `ceph` binaries.
These are downloaded from Chacra prior to building the installer.

Refactor `ceph-windows-test` and `ceph-windows-pull-requests` jobs

Refactor the Ceph Windows Jenkins jobs (`ceph-windows-test` and
`ceph-windows-pull-requests`) with common scripts from
`scripts/ceph-windows` and `scripts/build_utils.sh`.

Merge pull request #1943 from tchaikov/wip-submodules

ceph-pr-submodules: check for commit message noting submodule change

Reviewed-by: David Galloway <dgallowa@redhat.com>

ceph-pr-submodules: check for commit message noting submodule change

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

ceph-pr-submodules: print out the changed submodule only

instead of printing:

Project has modified submodules!
Submodule src/seastar f021c161816...ce2652d39dc (commits not present)

just print:

Project has modified submodules: src/seastar

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

ceph-pr-submodules: extract the check into a function

prepare for the next commit which add an escape rule

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #1935 from ionutbalutoiu/add-wnbd-job

[wnbd-build] Add new Jenkins job to build the wnbd project

[wnbd-build] Add Jenkins job to build the wnbd project

* Add the Jenkins job used to build the `wnbd` project.
* Add the following utils functions: `ssh_exec`, `scp_upload`, `scp_download`.
* Add `./scripts/ceph-windows/cleanup` script that can be used as a
  post-build step to clean up the Windows test / build environment.
* Add `./scripts/ceph-windows/setup_libvirt_vm` script used to spawn a
  libvirt VM with the Windows test / build environment.
* Upload the build artifact to chacra.

Merge pull request #1936 from rhcs-dashboard/dashboard-cephadm-e2e-nightly

project config: ceph-dashboard-cephadm-e2e-nightly

Merge pull request #1940 from rhcs-dashboard/dashboard-e2e-avoid-unit-tests-build

ceph-dashboard-pull-requests (e2e) job: do not build tests, only build ceph

project config: ceph-dashboard-cephadm-e2e-nightly

Project: run the ceph dashboard cephadm e2e tests against master,
pacific and octopus
Signed-off-by: Nizamudeen A <nia@redhat.com>

ceph-dashboard-pull-requests (e2e) job: do not build tests, only build ceph

We don't need to build tests for this job and also this change is a workaround for: https://tracker.ceph.com/issues/53582

Also do not build tests for nautilus nightly jobs.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #1939 from ceph/wip-doc-arm

ceph-pull-requests-arm64: Skip make check on doc-only change

ceph-pull-requests-arm64: Skip make check on doc-only change

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1938 from ceph/wip-which-machines

sepia-fog-images: Optionally allow specifying machines

sepia-fog-images: Optionally allow specifying machines

This is useful if you already have machine(s) locked

Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible: Clean up tmp files older than 14

Some of the braggi were still running out of space

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1933 from ionutbalutoiu/update-ceph-windows-pull-requests

Update ceph-windows-pull-requests job

Merge pull request #1934 from ionutbalutoiu/update-ceph-windows-test

Add branch param to the `ceph-windows-test` job

Add branch param to the `ceph-windows-test` job

The default value is `master`, since Windows builds for the master
branch were enabled by https://github.com/ceph/ceph-build/pull/1928.

The `get-bin.py` script is updated to support `--branchname` parameter:
https://github.com/ceph/ceph-win32-tests/pull/1

Update ceph-windows-pull-requests job

Run the job only against PRs that target the `master` branch.

Merge pull request #1927 from ionutbalutoiu/ceph-windows-pull-requests

Add ceph-windows-pull-requests job & Update ceph-windows-test job

Merge pull request #1932 from rhcs-dashboard/cephadm-e2e-node-v12

ceph-dashboard-cephadm-e2e job: update nodejs to v12

ceph-dashboard-cephadm-e2e job: update nodejs to v12

'ceph/make-dist' file and Angular 12 (https://github.com/ceph/ceph/pull/43404) require nodejs >= 12.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #1928 from ceph/wip-master-win

ceph-dev*: Start building Windows binaries for master

Merge pull request #1931 from ceph/ceph_container_lint_flake_always_pull

container-{flake8,lint}: always pull the image

container-{flake8,lint}: always pull the image

Otherwise we will still use an outdated container image.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Add ceph-windows-pull-requests job

Used to run Windows tests against Ceph pull requests.

Update ceph-windows-test job

* Create `scripts/ceph-windows` directory with the common scripts
  for the Ceph Windows testing.
* Move `run_tests` script to `scripts/ceph-windows`.
* Remove VM image caching from the `run_tests` script.
* Add `cleanup_tests` script to `scripts/ceph-windows`, and call
  this as a post-build step when the job is aborted.
* Add `get_chacra_build` build script, and refactor away this logic
  from the `run_tests` script.
* Move cleanup Windows tests function to `scripts/build_utils.sh`.

Merge pull request #1930 from ceph/ceph_container_lint_fix_pipe

ceph-container-lint: add missing pipe

ceph-container-lint: add missing pipe

The pipe was removed in the previous change but it shouldn't.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1929 from ceph/ceph_container_lint_fix_files_removed

ceph-container-lint: filter out file removed

ceph-container-lint: filter out file removed

We don't want to run shellcheck on shell script removed by a pull
request.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

ceph-dev*: Start building Windows binaries for master

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1922 from ionutbalutoiu/ceph-windows-test

Add Ceph Windows testing job

Add Ceph Windows testing job

Add a new freestyle Jenkins job that can be used to test a Ceph
Windows build uploaded to Chacra, on the Jenkins agents labeled with
`amd64`, `focal`, `libvirt`.

The Ceph Windows build is identified in Chacra using the following
Jenkins job parameters:
* `CEPH_WINDOWS_VERSION`
* `CEPH_WINDOWS_SHA1`

As a build step, the new job will execute the `./build/run_tests`
bash script. This script sets up a KVM virtual machine with Windows
Server 2019, that it's used to run the Ceph Windows unit tests from
the Chacra build. On top of that, it takes care of the everything
needed to run the Ceph Windows tests.

Merge pull request #1926 from ceph/https-not-git

ceph-dev-new-setup/build/build: git:// -> https://

ceph-dev-new-setup/build/build: git:// -> https://

https://github.blog/2021-09-01-improving-git-protocol-security-github/

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #1925 from ceph/cv_nightly_remove_myself_from_notif

ceph-volume-nightly: remove dsavinea

ceph-volume-nightly: remove dsavinea

This removes dsavinea@redhat.com from the recipients list.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1924 from ceph/ca_tox

ceph-ansible-prs: use right tox config file

ceph-ansible-prs: use right tox config file

use tox-subset_update.ini for the subset_update job.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1923 from ceph/add_ca_subset_update

ceph-ansible-prs: add new scenario subset_update

ceph-ansible-prs: add new scenario subset_update

see ceph-ansible corresponding pr: ceph/ceph-ansible#6951

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1921 from ceph/wip-pin-ansible

scripts: Set LC_ALL=en_US.UTF-8 again

scripts: Set LC_ALL=en_US.UTF-8 again

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1920 from ceph/wip-n-yuri

build_utils.sh: Disable CI_CONTAINER for nautilus too

build_utils.sh: Disable CI_CONTAINER for nautilus too

Brad and Yuri are working on something for downstream and need some branches built in ceph-ci.git based on the nautilus branch.  The builds keep showing up as failed in shaman because the container is failing to build.  This minor change fixes that.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1919 from dmick/wip-cephadm-binary

ceph-*-build: only try extracting cephadm if it exists

ceph-*-build: only try extracting cephadm if it exists

https://github.com/ceph/ceph-build/pull/1913 added code to extract
the cephadm binary from its package and push it to chacra; however,
older builds don't include a cephadm package, and ceph-build must
work for those as well.  Broke for nautilus.

Fixes: https://tracker.ceph.com/issues/52847
Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1918 from ceph/wip-container-arch

ceph-dev*build: Wait for the appropriate arch's repo to be ready

ceph-dev*build: Wait for the appropriate arch's repo to be ready

Also use shaman's search API.

What was happening before this was, for example:

- master branch gets built
- x86_64 job, build, and repo are done way before arm64
- arm64 job finally gets to the point where it's ready to build its container
- arm64 job checks with shaman to ask if the branch's repo is ready
- shaman answers with the x86_64 repo status and says it is ready
- arm64 job proceeds but fails because the arm64 repo is not ready

```
$ echo $RELEASE
8
$ echo $SHA1
d5759e030bea9732b905c51d041fe16c934790ce
$ echo $BRANCH
master
$ echo $FLAVOR
default
$ for ARCH in x86_64 arm64; do echo "============== $ARCH =============="; curl -s "https://shaman.ceph.com/api/search/?project=ceph&distros=centos/${RELEASE}/${ARCH}&sha1=${SHA1}&ref=${BRANCH}&flavor=${FLAVOR}" | jq -r '.[0].status'; done
============== x86_64 ==============
ready
============== arm64 ==============
ready

```

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-dev-cron: Keep more builds

Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible: Update Jenkins master java args

Latest version of Jenkins didn't support any of the removed arguments anymore

Signed-off-by: David Galloway <dgallowa@redhat.com>