Merge pull request #2098 from ceph/pin-tox-cv-scenario

ceph-volume-scenario: pin the tox version

ceph-volume-scenario: pin the tox version

Due to a regression introduced in tox 4.3.1, we have to pin
the version used in the CI in order to avoid the following error:

```
tox.report.HandledError: replace failed in centos8-filestore-create.commands with MatchRecursionError('circular chain between set env VAGRANT_UP_FLAGS, CEPH_DEV_BRANCH, CEPH_DEV_SHA1')
```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #2097 from ceph/ca-pin-tox

ceph-ansible-prs: pin tox version

ceph-ansible-prs: pin tox version

Due to a regression introduced in tox 4.3.1, multiple environment variable substitution
is broken.
```
tox.report.HandledError: replace failed in centos-non_container-update.commands with MatchRecursionError('circular chain between set env DOCKER_HUB_USERNAME, DOCKER_HUB_PASSWORD, INVENTORY')
```

Let's pin to the last tox stable version where it used to work fine (4.2.8).

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #2096 from Matan-B/wip-matanb-c-cephfs-java-disable-revert

Revert "build_utils.sh: Disable cephfs_java with Crimson"

Reviewed-by: Samuel Just <sjust@redhat.com>

Revert "build_utils.sh: Disable cephfs_java with Crimson"

This reverts commit dd03c322eaded5f1bb274bb46fb13fbfe7a191aa.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #2095 from Matan-B/wip-matanb-c-cephfs-java-disable

build_utils.sh: Disable cephfs_java with Crimson

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

build_utils.sh: Disable cephfs_java with Crimson

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #2094 from ionutbalutoiu/update-ceph-win-tests

ceph-windows: Use PowerShell script to run tests

ceph-windows: Use PowerShell script to run tests

We have a new PowerShell script that runs all the Ceph Windows tests
from the main repository.

Signed-off-by: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>

Merge pull request #2093 from ionutbalutoiu/update-ceph-win-tests

ceph-windows: Use locally cloned script, if possible

ceph-windows: Use locally cloned script, if possible

Use the locally cloned `test_rbd_wnbd.py` script if `$WORKSPACE/ceph`
repository was cloned as part of the Jenkins job.

Signed-off-by: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>

Merge pull request #2091 from petrutlucian94/win_stamp_test

ceph-windows: re-enable rbd-wnbd stamp test

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

ceph-windows: re-enable rbd-wnbd stamp test

We had to disable one of the rbd-wnbd tests while investigating
the reason why disks started coming up as read-only.

This was actually caused by the wnbd bus type, which changed from
virtual to SAS. The default Windows policy (offlineShared) doesn't
automatically bring online such disks, SAS being considered a
shared bus.

While at it, we're also enabling the new *Fs* tests, which use
a Windows partition instead of the raw block devices.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2076 from rzarzynski/wip-kickoff-reef

kickoff v18 reef

Merge pull request #2090 from ceph/ceph-ansible-infinite-loop

ceph-ansible-nightly: Don't search tags endlessly

ceph-ansible-nightly: Don't search tags endlessly

If the reponse is in an unexpected format (e.g. it's an error response),
we were looping infinitely; one job got to over 60k pages!

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2089 from petrutlucian94/disable_wnbd_test

temporarily disable windows RbdStampTest test

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

temporarily disable windows rbd-wnbd test

One of the rbd-wnbd tests has started failing after the Windows
image got updated. The disks now show up as "read-only" by default.

https://pastebin.com/raw/JXmLXFnN

We'll temporarily disable this test to unblock the Windows job
while doing further investigation.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2088 from ceph/cdn-quay

ceph-dev-new: Default to internal quay instance

ceph-dev-new: Default to internal quay instance

Temporarily, so we can unblock building on git push.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2087 from ceph/quay-ceph-io-ceph-ci

Replace dmick-quay creds with quay-ceph-io-ceph-ci

Replace dmick-quay creds with quay-ceph-io-ceph-ci

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2086 from ionutbalutoiu/add-windows-tests

ceph-windows: Add more Windows testing

Merge pull request #2085 from ionutbalutoiu/update-ceph-win-img-build

ceph-windows: Update `ceph-windows-image-build` setup script

ceph-windows: Add more Windows testing

Run `test_rbd_wnbd.py` script from the upstream Ceph QA Windows
workunit scripts as part of the `script/ceph-windows/run_tests`.

The new test cases don't take a lot of time to execute (unless
something is wrong with Ceph on Windows or Linux). So, we won't
have any noticeable jobs' execute time penalty.

This adds more Windows tests to the Jenkins jobs:
* `ceph-windows-test`
* `ceph-windows-pull-requests`

Signed-off-by: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>

ceph-windows: Update `ceph-windows-image-build` setup script

* Bump Python to `3.11`.
* Add `fio` as part of Windows CI image setup.
* Add `prettytable` pip dependency. We'll use this for the Python
  `test_rbd_wnbd.py` script.

Signed-off-by: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>

Merge pull request #2084 from lxbsz/fs_encryption

kernel: enable filesystem encryption option

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

kernel: enable filesystem encryption option

This will enable us to test the filesystem encryption feature in
cephfs kclient.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #2083 from dmick/main

Fix slave.yml symlink

Fix slave.yml symlink

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2082 from ceph/wip-kernel-trigger-master

kernel: re-add master branch to the trigger list

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

ceph-dev-build: update {build,setup}_osc for reef

Based on 18c3c7719c3ded64dbec815774defa6166a492d1.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

scripts/build_utils: map 18.* to reef

Based on e75fe3903f750811d7f4067ec9c9f2597203ebfe.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #2075 from ceph/cv-fix-nightly

ceph-volume-nightly: fix broken tests

ceph-volume: drop testing against focal

due the a too few number of available jenkins workers allocated
to ceph-volume, we can't afford testing against both centos and ubuntu.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume-nightly: fix broken tests

ceph-volume nightly jobs are broken for a while because it
picks 'main' as ceph-ansible branch for any ceph release being tested.

With this commit, the right ceph-ansible branch will be picked.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #2074 from ceph/revert-refactor-collect-logs

build_utils: revert collect-logs refactor

Revert "ceph-ansible-prs: refact collect-logs"

This reverts commit 21cbce81e1759852b9e2b167d43c7d2e80f7aeba.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Revert "build_utils: fix collect_ceph_logs function"

This reverts commit 14f91f9d7a2b2bd9ca17cc1a89b5a48191e46d1f.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #2073 from ceph/refactor-collect-logs

build_utils: fix collect_ceph_logs function

build_utils: fix collect_ceph_logs function

typical error from ceph-volume tests:

```
ERROR! the playbook: /home/jenkins-build/build/workspace/ceph-volume-prs-lvm-centos8-bluestore-dmcrypt/tests/functional/collect-logs.yml could not be found
```

Since `collect-logs.yml` is maintained in ceph-ansible repository,
when `collect_ceph_logs` is called from a job different from
ceph-ansible, the path is incorrect.
The idea is to make ceph-volume jobs override this path.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #2072 from ionutbalutoiu/collect-win-logs

ceph-windows-pull-requests: Collect tests logs

ceph-windows-pull-requests: Collect tests logs

Also, set `artifact-days-to-keep` and `artifact-num-to-keep` to make
sure we don't clutter the Jenkins server with log files.

Signed-off-by: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>

Merge pull request #2071 from ceph/cv-nightlies-update

ceph-volume-nightly: cleanup

ceph-volume-nightly: cleanup

update ceph-volume nightly jobs.
drop nautilus testing as it's EOL for a while.
drop testing against el7
drop testing against xenial (replace with focal)
test against quincy and pacific

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #2070 from dsavineau/tcmalloc_in_mock

nfs-ganesha/rpm: Fix tcmalloc allocator setup

nfs-ganesha/rpm: Fix tcmalloc allocator setup

The previous commit installed the gperftools-devel package and set the
cmake ALLOCATION variable to tcmalloc but this isn't used at all during
the RPM package build via mock.
We also need those changes in the RPM spec file.
This is a temporary solution until the upstream nfs-ganesha RPM spec file
will support such change.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #2069 from ceph/fix-ganesha-builds

nfs-ganesha: build with tcmalloc

nfs-ganesha: bump debian build to focal

we don't build for bionic anymore, this should be updated.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

nfs-ganesha: build with tcmalloc

since https://github.com/ceph/ceph/pull/47628 any development builds of nfs-ganesha
are broken.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #2066 from ceph/builtin

ceph-dev-container-only: Use Jenkins controller for main job

ceph-dev-container-only: Use Jenkins controller for main job

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #2065 from ceph/fix-sync-pull2

Updates for sync-pull signing script

sync-pull: Fix ssh command

chacra.ceph.com used to live in OVH and listened for SSH on port 2222.  The VM is in Sepia now and listens on 22.

Signed-off-by: David Galloway <dgallowa@redhat.com>

sync-pull: Automate download.ceph.com tarball upload

I swapped the paths in `for endpoint in` because there may not be a centos/9 repo.  That means `$version` gets left unset and isn't available to `get-tarballs.sh` later in the script.

Signed-off-by: David Galloway <dgallowa@redhat.com>

sync-pull: Update possible distro combos

Removed EOL releases

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #2030 from rhcs-dashboard/change-log-dir

dashboard-e2e: change artifact dir to mstart ones

Merge pull request #2064 from tchaikov/ceph-windows-cleanup

ceph-windows: remove installed packages

ceph-windows: remove installed packages

`ceph-windows/run_tests` installs client side dependencies using
`cephadm install ceph-common`, which leaves us with a bunch ceph
packages after the test finishes. and they inteferes with succeeding
tests which, for instance, try to import python modules from test's
building directory.
but they end up with importing the python modules installed in system.

in this change, the installed package and its ceph dependencies are
removed in the cleanup script. the list of packages is collected by
inspecting `/var/log/dpkg.log` on the system where the test failures
were spotted and `ceph/debian/control` 's "ceph-common" section.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #2058 from ceph/quincy-c9

ceph-dev-cron: Build quincy on el9 daily

Merge pull request #2055 from ceph/touch-mgr-file

ceph-pr-api: Avoid artifact archive plugin hang

Merge pull request #2043 from cbodley/wip-boost-valgrind

add WITH_BOOST_VALGRIND to default build flavor

Merge pull request #2061 from tchaikov/pbuilder

build_utils/build_debs: use variable when appropriate

Merge pull request #2063 from ceph/no-more-lsb

Stop using lsb for rpm-based distros

Merge pull request #2060 from ceph/ceph-ci-c9

ceph-dev-new: Build centos9 for main and quincy based ceph-ci branches

Stop using lsb for rpm-based distros

lsb_release was removed in EL9 and we already have functions to read /etc/os-release and return the values we want.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #2059 from ceph/scripts-c9

scripts: Update signing scripts for el9

build_utils/build_debs: use variable when appropriate

less "magic numbers" this way.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #1999 from ceph/wip-pbuilder-dir

pbuilder: build in $WORKSPACE

ceph-dev-new: Build centos9 for main and quincy based ceph-ci branches

Signed-off-by: David Galloway <dgallowa@redhat.com>

scripts: Update signing scripts for el9

Signed-off-by: David Galloway <dgallowa@redhat.com>

pbuilder: build in $WORKSPACE or tmpfs if available

Come to find out unless --buildplace is defined, pbuilder builds in /var/cache/pbuilder/build.

The braggi and adami builders have separate faster and larger SSDs mounted at /home/jenkins-build but pbuilder hasn't been using them.

Additionally, it is possible to build in memory.  The braggi and adami have plenty of space available to build in it so let's use it.

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-dev-cron: Build quincy on el9 daily

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #2057 from ceph/c9-cron

ceph-dev-cron: Build main on centos9 daily

ceph-dev-cron: Build main on centos9 daily

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #2054 from ceph/el9

ceph-dev: Automatically build main on centos9

Merge pull request #2056 from ceph/no-rhel9-container

ceph*build: Do not build containers on CentOS9 for now

Merge pull request #2044 from ceph/trigger-branches

ceph-docs-rtd: Create job to keep releases docs up-to-date

ceph-docs-rtd: Create job to keep releases docs up-to-date

See ceph/ceph#46917

This Jenkins job will receive webhooks from the github plugin.  It will then check if the main branch is what was updated and if the `doc/releases` dir was updated.

If true, it will trigger a build for each active Ceph release's docs.

ceph/ceph#47442
ceph/ceph#47443
ceph/ceph#47444

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #2052 from ceph/windows-jammy

More fixes to get Windows jobs running again

ceph*build: Do not build containers on CentOS9 for now

ceph-container isn't ready yet.  There is work being done in https://github.com/ceph/ceph-container/pull/2038 but our build scripts still need a lot of adjustments.

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-pr-api: Avoid artifact archive plugin hang

More often than not, docs-only PRs hang during this job for 45-50 minutes then crash like this:

```
+ '[' true = true ']'
+ echo 'Only the doc/ dir changed.  No need to run make check or API tests.'
Only the doc/ dir changed.  No need to run make check or API tests.
+ exit 0
[PostBuildScript] - [INFO] Executing post build scripts.
[PostBuildScript] - [INFO] Build does not have any of the results [ABORTED]. Did not execute build step #0.
Archiving artifacts
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3689)
at java.base/java.util.Vector.grow(Vector.java:261)
at java.base/java.util.Vector.ensureCapacity(Vector.java:241)
at org.apache.tools.ant.util.VectorSet.doAdd(VectorSet.java:86)
at org.apache.tools.ant.util.VectorSet.addElement(VectorSet.java:98)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1245)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1194)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1156)
at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:954)
at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:912)
at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:528)
at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:3159)
Caused: java.io.IOException: Remote call on JNLP4-connect connection from 8.43.84.3/8.43.84.3:42699 failed
at hudson.remoting.Channel.call(Channel.java:1004)
at hudson.FilePath.act(FilePath.java:1194)
at hudson.FilePath.act(FilePath.java:1183)
at hudson.FilePath.validateAntFileMask(FilePath.java:3095)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:271)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:79)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:816)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:765)
at hudson.model.Build$BuildExecution.post2(Build.java:179)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:709)
at hudson.model.Run.execute(Run.java:1922)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
No artifacts found that match the file pattern "build/out/mgr.*.log". Configuration error?
```

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-windows: Cleanup repo files

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-windows: Install dev version of ceph-common

We don't have an official Ubuntu Jammy ceph-common package yet and won't until the next version of quincy.

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-dev: Automatically build main on centos9

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #2053 from ceph/enable-linger

ansible: loginctl enable-linger jenkins-build

ansible: loginctl enable-linger jenkins-build

https://github.com/containers/podman/issues/9002

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-windows: Use quincy

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #2051 from ceph/cleanupfix

windows: Check for cephadm binary before attempting to use it

windows: Check for cephadm binary before attempting to use it

Fixes:
```
+ [[ -x /home/jenkins-build/build/workspace/wnbd-build/cephadm ]]
+ sudo /home/jenkins-build/build/workspace/wnbd-build/cephadm rm-repo
[sudo] password for jenkins-build:
sudo: no password was provided
sudo: a password is required
Build step 'Execute scripts' changed build result to FAILURE
Build step 'Execute scripts' marked build as failure
```

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #2050 from tchaikov/drop-stale-comment

ceph-build-pull-requests: drop stale comment

ceph-build-pull-requests: drop stale comment

now that we've bumped up the urllib3's version in
f2f9bec9da002ac07b27c5caf5131aeab196509c. the comment does not
apply anymore.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #2048 from tchaikov/wip-newer-urllib3

ceph-build-pull-requests: use a newer urllib3 when installing jjb

Merge pull request #2047 from tchaikov/node14

dashboard-cephadm-e2e: bump node version to 14

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

ceph-build-pull-requests: use a newer urllib3 when installing jjb

as urllib3 v1.22 is not compatible with python3.10 which is shipped by
ubuntu jammy. see https://pypi.org/project/urllib3/1.22/

so, to run jjb in a python3.10 venv we should use a newer urllib.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

dashboard-cephadm-e2e: bump node version to 14

as node.js 12 is being deprecated. and the oldest node.js which is
packaged by https://github.com/nodesource is node.js 14. see
https://github.com/nodesource/distributions#debinstall

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #2046 from ceph/windows-cleanup

ceph-windows: Clean up repo files

ceph-windows: Clean up repo files

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #2045 from ionutbalutoiu/fix-windows-jenkins-jobs

Improve Windows libvirt VM IP detection