Stefan Chivu [Tue, 31 Jan 2023 12:08:02 +0000 (14:08 +0200)]
ceph-windows: Added script for collecting Windows event logs
In order to collect more information for the build artifacts,
the collect-event-logs.ps1 script will be run in order to extract
the event logs from the Windows client machine.
It will dump all the event logs as evtx and then convert them to
txt in order to be accessible on all platforms.
If the -IncludeEvtxFiles flag is used, the evtx files can be kept.
By default they are deleted.
Also, if the -CleanupEventLog flag is used, then the machine's
events will get cleared after the dump. By default they are kept.
The dumped event log files can be found in the directory sent as
parameter using the mandatory -LogDirectory parameter.
Signed-off-by: Stefan Chivu <schivu@cloudbasesolutions.com>
Lucian Petrut [Wed, 25 Jan 2023 12:52:45 +0000 (14:52 +0200)]
ceph-windows: update test timeouts
The Python rbd-wnbd tests time out on stable branches. The reason
is that if the test script is missing, we're fetching it from
the main branch and then run the tests individually.
The issue is that while on the main branch the entire suite uses
a 30m timeout, on stable branches we're excuting each individual
test with a 5m timeout, which isn't enough for some of the tests
(e.g. the FIO one).
For this reason, we're going to increase the timeout.
While at it, we're pinning the commit id when fetching the test
script on older branches. That will allow us to move it or
refactor it.
Due to a regression introduced in tox 4.3.1, we have to pin
the version used in the CI in order to avoid the following error:
```
tox.report.HandledError: replace failed in centos8-filestore-create.commands with MatchRecursionError('circular chain between set env VAGRANT_UP_FLAGS, CEPH_DEV_BRANCH, CEPH_DEV_SHA1')
```
Due to a regression introduced in tox 4.3.1, multiple environment variable substitution
is broken.
```
tox.report.HandledError: replace failed in centos-non_container-update.commands with MatchRecursionError('circular chain between set env DOCKER_HUB_USERNAME, DOCKER_HUB_PASSWORD, INVENTORY')
```
Let's pin to the last tox stable version where it used to work fine (4.2.8).
Lucian Petrut [Thu, 24 Nov 2022 08:01:52 +0000 (10:01 +0200)]
ceph-windows: re-enable rbd-wnbd stamp test
We had to disable one of the rbd-wnbd tests while investigating
the reason why disks started coming up as read-only.
This was actually caused by the wnbd bus type, which changed from
virtual to SAS. The default Windows policy (offlineShared) doesn't
automatically bring online such disks, SAS being considered a
shared bus.
While at it, we're also enabling the new *Fs* tests, which use
a Windows partition instead of the raw block devices.
Ionut Balutoiu [Mon, 31 Oct 2022 16:16:56 +0000 (18:16 +0200)]
ceph-windows: Add more Windows testing
Run `test_rbd_wnbd.py` script from the upstream Ceph QA Windows
workunit scripts as part of the `script/ceph-windows/run_tests`.
The new test cases don't take a lot of time to execute (unless
something is wrong with Ceph on Windows or Linux). So, we won't
have any noticeable jobs' execute time penalty.
This adds more Windows tests to the Jenkins jobs:
* `ceph-windows-test`
* `ceph-windows-pull-requests`
* Bump Python to `3.11`.
* Add `fio` as part of Windows CI image setup.
* Add `prettytable` pip dependency. We'll use this for the Python
`test_rbd_wnbd.py` script.
```
ERROR! the playbook: /home/jenkins-build/build/workspace/ceph-volume-prs-lvm-centos8-bluestore-dmcrypt/tests/functional/collect-logs.yml could not be found
```
Since `collect-logs.yml` is maintained in ceph-ansible repository,
when `collect_ceph_logs` is called from a job different from
ceph-ansible, the path is incorrect.
The idea is to make ceph-volume jobs override this path.
update ceph-volume nightly jobs.
drop nautilus testing as it's EOL for a while.
drop testing against el7
drop testing against xenial (replace with focal)
test against quincy and pacific
The previous commit installed the gperftools-devel package and set the
cmake ALLOCATION variable to tcmalloc but this isn't used at all during
the RPM package build via mock.
We also need those changes in the RPM spec file.
This is a temporary solution until the upstream nfs-ganesha RPM spec file
will support such change.
I swapped the paths in `for endpoint in` because there may not be a centos/9 repo. That means `$version` gets left unset and isn't available to `get-tarballs.sh` later in the script.
Signed-off-by: David Galloway <dgallowa@redhat.com>
Kefu Chai [Sat, 20 Aug 2022 00:17:45 +0000 (08:17 +0800)]
ceph-windows: remove installed packages
`ceph-windows/run_tests` installs client side dependencies using
`cephadm install ceph-common`, which leaves us with a bunch ceph
packages after the test finishes. and they inteferes with succeeding
tests which, for instance, try to import python modules from test's
building directory.
but they end up with importing the python modules installed in system.
in this change, the installed package and its ceph dependencies are
removed in the cleanup script. the list of packages is collected by
inspecting `/var/log/dpkg.log` on the system where the test failures
were spotted and `ceph/debian/control` 's "ceph-common" section.
David Galloway [Wed, 10 Aug 2022 18:41:35 +0000 (14:41 -0400)]
ceph-docs-rtd: Create job to keep releases docs up-to-date
See ceph/ceph#46917
This Jenkins job will receive webhooks from the github plugin. It will then check if the main branch is what was updated and if the `doc/releases` dir was updated.
If true, it will trigger a build for each active Ceph release's docs.
ceph/ceph#47442
ceph/ceph#47443
ceph/ceph#47444
Signed-off-by: David Galloway <dgallowa@redhat.com>
David Galloway [Wed, 10 Aug 2022 17:29:01 +0000 (13:29 -0400)]
ceph*build: Do not build containers on CentOS9 for now
ceph-container isn't ready yet. There is work being done in https://github.com/ceph/ceph-container/pull/2038 but our build scripts still need a lot of adjustments.
Signed-off-by: David Galloway <dgallowa@redhat.com>
David Galloway [Wed, 10 Aug 2022 17:19:02 +0000 (13:19 -0400)]
ceph-pr-api: Avoid artifact archive plugin hang
More often than not, docs-only PRs hang during this job for 45-50 minutes then crash like this:
```
+ '[' true = true ']'
+ echo 'Only the doc/ dir changed. No need to run make check or API tests.'
Only the doc/ dir changed. No need to run make check or API tests.
+ exit 0
[PostBuildScript] - [INFO] Executing post build scripts.
[PostBuildScript] - [INFO] Build does not have any of the results [ABORTED]. Did not execute build step #0.
Archiving artifacts
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3689)
at java.base/java.util.Vector.grow(Vector.java:261)
at java.base/java.util.Vector.ensureCapacity(Vector.java:241)
at org.apache.tools.ant.util.VectorSet.doAdd(VectorSet.java:86)
at org.apache.tools.ant.util.VectorSet.addElement(VectorSet.java:98)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1245)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1267)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1194)
at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1156)
at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:954)
at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:912)
at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:528)
at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:3159)
Caused: java.io.IOException: Remote call on JNLP4-connect connection from 8.43.84.3/8.43.84.3:42699 failed
at hudson.remoting.Channel.call(Channel.java:1004)
at hudson.FilePath.act(FilePath.java:1194)
at hudson.FilePath.act(FilePath.java:1183)
at hudson.FilePath.validateAntFileMask(FilePath.java:3095)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:271)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:79)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:816)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:765)
at hudson.model.Build$BuildExecution.post2(Build.java:179)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:709)
at hudson.model.Run.execute(Run.java:1922)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
No artifacts found that match the file pattern "build/out/mgr.*.log". Configuration error?
```
Signed-off-by: David Galloway <dgallowa@redhat.com>