Merge pull request #2164 from petrutlucian94/whitelist_ceph

windows: add Ceph dir to Windows Defender exclusion list

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: add Ceph dir to Windows Defender exclusion list

We're getting occasional false positives from Windows Defender.
For this reason, we'll add the Ceph dir to the exclusion list.

  ERROR - Test exception: [WinError 225] Operation did not complete
  successfully because the file contains a virus or potentially unwanted
  software. Total exceptions: 22

   Windows Defender Antivirus has taken action to protect this machine from
   malware or other potentially unwanted software.
    For more information please see the following:
   https://go.microsoft.com/fwlink/?linkid=37020&name=Trojan:Win32/Bearfoos.A!m
   l&threatid=2147731250&enterprise=0
        Name: Trojan:Win32/Bearfoos.A!ml
        ID: 2147731250
        Severity: Severe
        Category: Trojan
        Path: file:_C:\ceph\rbd-wnbd.exe
        Detection Origin: Local machine
        Detection Type: Concrete
        Detection Source: Real-Time Protection
        User: NT AUTHORITY\SYSTEM
        Process Name: C:\Program Files\Python311\python.exe
        Action: Quarantine
        Action Status:  No additional actions required
        Error Code: 0x80508023
        Error description: The program could not find the malware and other
                       potentially unwanted software on this device.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2163 from dmick/wip-sepia-fog-images-build

sepia-fog-images/build/build: bootstrap teuthology unconditionally

Whether or not we will teuthology-lock machines, we always need
teuthology-queue --pause

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2162 from petrutlucian94/win32_sv_start

windows: use "net.exe start" instead of Start-Service

windows: use "net.exe start" instead of Start-Service

The "Start-Serice" command doesn't really say why the service could
not be started:

  Start-Service : Service 'ceph-rbd (ceph-rbd)' cannot be started
  due to the following error: Cannot start service

We'll use the "net.exe start" command instead, which provides
additional information.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2161 from tserlin/wip-tserlin-pin-to-V5.5

nfs-ganesha: pin the daily build to V5.5

nfs-ganesha: pin the daily build to V5.5

Instead of just building the tip of whatever is in the "next"
branch, let's pin the daily nfs-ganesha build to the latest
stable tag: V5.5.

Merge pull request #2159 from ljflores/wip-lower-concurrency

scripts: lower concurrency for many-core machines

scripts: lower concurrency for many-core machines

adami nodes have 96 processing units (nproc). Based on the old script
logic, we were compiling 90 build jobs with 96 processing units. This
combination (96/90) ends up causing many instances of memory
overconsumption on adami nodes.

With this new logic, we take into account that jobs take up more memory
than we expect. This will make it so adami nodes will compile 67 build
jobs on 96 processing units, which will hopefully avoid so many instances
of memory overconsumption. (Total memory on adami nodes is generally
~270036 MiB, so 270036 / 4000 = 67)

braggi nodes do not have this problem; they have 48 processing units.
It has been working so far to compile 48 build jobs with 48 processing
units on the braggi nodes, and this new logic will not change that or
any other node with nproc <= 50.

Fixes: https://tracker.ceph.com/issues/57296
Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #2157 from dmick/wip-jenkins-builders

Update jenkins-builder support

Merge pull request #2158 from ceph/fix-ca-flake8

ceph-container-flake8: migrate to alpine/flake8 image

ceph-container-flake8: migrate to alpine/flake8 image

the current image is old and uses a flake8 build based on
py2.7.

It means we are checking python 3.x code with a python 2.7 version
of flake8. That doesn't work very well.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Standardize on "label" name for parameter

The Jenkins UI (or at least the Python binding) uses "label", not
"labels" (for reconfig only, because it uses XML, and the XML contains
the tag 'label'.  Sadly that's not true for create.)

Avoid the confusion that wasted my time.

Signed-off-by: Dan Mick <dmick@redhat.com>

ansible/builder.yml: add bookworm deb signing keys

Signed-off-by: Dan Mick <dmick@redhat.com>

Treat 8.43-IP hosts as "permanent" builders too (for ceph.io, which builds HTML)

Signed-off-by: Dan Mick <dmick@redhat.com>

ansible: use 'inventory_hostname' to look up jenkins labels

inventory_hostname is as it appears in the inventory file, and keeping
that consistent is easier than dealing with short vs long names
Note: this requires a change to ceph-sepia-secrets to standardize on
the inventory form of the name

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2156 from ceph/fix-container-flake8

ceph-container-flake8: fix test

ceph-container-flake8: fix test

Looks like this job has never worked.
The file should be bind-mounted in /code, not in /
There's no need to pass arguments, the entrypoint automatically
scan /code and run flake8 against any present files.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2154 from Matan-B/wip-matanb-crimson-centos9

ceph-dev*: build crimson on centos9 (main and reef)

Merge pull request #2155 from petrutlucian94/log_disk_space

windows: log available disk space

Some Windows test jobs have failed after the OSDs ran out of disk
space. Those jobs use Linux VMs with 128GBs of storage and run
3 OSDs, each having 15GB.

In order to get a better picture, we'll log the disk space usage,
including the vstart directory contents.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2147 from dmick/wip-jenkins-builders

ansible: enhance library/jenkins_node to allow node update

ceph-dev*: build crimson on centos9 (main and reef)

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #2152 from dmick/wip-vault-pass

ansible: remove vault_password_file; jenkins-build doesn't have one

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

ansible: remove vault_password_file; jenkins-build doesn't have one

Jenkins also uses ansible for some builds, and this config item breaks
the run, because although the ansible secret isn't needed, if you configure
it, ansible requires it.  <sigh>

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2150 from lxbsz/centos9

kernel: enable centos9 build

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

kernel: enable the centos9 build

commit 4b61a221e4e2f53dc7df3137fc34f418c3f3b879 just forgot to enable
it after adding centos9.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #2149 from dmick/wip-preserve-temp-jobs

jenkins-job-builder: don't delete jobs named 'preserve-*'

jenkins-job-builder: don't delete jobs named 'preserve-*'

This is to allow job development without losing the jobs and past builds
as other ceph-build PRs are pushed.  It's assumed that if you create a job
with 'preserve-' in its title that you will clean it up when you're done.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2141 from ceph/wip-tcmu-runner-repo-url

tcmu-runner: set $RELEASE before it's used

Reviewed-by: Adam Kraitman <akraitma@redhat.com>

Merge pull request #2148 from ljflores/wip-increase-timeout

ceph-dev-new-setup/config/definitions: increase timeout for cloning ci repo

ceph-dev-new-setup/config/definitions: increase timeout for cloning ci repo

Might help address failures like
```
fatal: fetch-pack: invalid index-pack output`
```
and
```
error: rev-list died of signal 15
error: github.com:ceph/ceph-ci.git did not send all necessary objects
```

Signed-off-by: Laura Flores <lflores@redhat.com>

library/jenkins_node: improve diagnostics on failure.

Capture exception args for exceptions that have no msg or message,
and capture a traceback.

Signed-off-by: Dan Mick <dmick@redhat.com>

examples/builder.yml, library/jenkins_node: support node update

Original implementation would only permit nodes to be created if they
did not exist.  Add support to update certain attributes of the node
with a run against extant nodes.  Only permit a few attribute updates.
This is primarily motivated by the desire to support maintaining node
labels with ansible.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2139 from dmick/wip-jenkins-builders

Fix up a number of problems with ansible/examples/builder.yml

Merge pull request #2146 from rhcs-dashboard/cleanup-script-cephadm-e2e-nightly

dashboard-cephadm-e2e-nightly: move the cleanup script to postbuild step

library/jenkins_node: use python3

Signed-off-by: Dan Mick <dmick@redhat.com>

files/ssh/hostkeys/github.com.pub: update the github host key

github reissued its host key in March 2023 (see
https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/).
Record here for use in setting up jenkins builders.

Signed-off-by: Dan Mick <dmick@redhat.com>

ansible/examples/builder.yml: kill any rogue agent/slave.jar procs

Some time ago slave.jar was renamed agent.jar, and there were
builders running the old version, sometimes as root (which caused
problems when the job would check out git workspaces as root
that could then not be removed by a job running as jenkins-build).
Clean up the crufty procs, if any.

Signed-off-by: Dan Mick <dmick@redhat.com>

ansible/examples/builder.yml: fix egregious spelling error

also add a tag for the register step; will be useful for updating
tags once I get updating working

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2144 from ljflores/wip-crimson-pacific-2

ceph-dev-cron, ceph-dev-new-trigger: don't build crimson for pacific

ansible/examples/builder.yml: use chown -R to change perms

chown -R is *much* faster than the ansible file module

Signed-off-by: Dan Mick <dmick@redhat.com>

ansible/examples/builder.yml: set builder labels from group_vars

Signed-off-by: Dan Mick <dmick@redhat.com>

ansible/examples/builder.yml: use secret for jenkins_api_token

Put it into variable 'token' for all the other tasks.

Signed-off-by: Dan Mick <dmick@redhat.com>

dashboard-cephadm-e2e-nightly: move the cleanup script to postbuild step

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #2145 from rhcs-dashboard/cleanup-script-cephadm-e2e

dashboard-cephadm-e2e: move the cleanup script to postbuild step

dashboard-cephadm-e2e: move the cleanup script to postbuild step

Signed-off-by: Nizamudeen A <nia@redhat.com>

ceph-dev-cron, ceph-dev-new-trigger: don't build crimson for pacific

Followup to PR https://github.com/ceph/ceph-build/pull/2142; removes
crimson pacific from additional build jobs.

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #2143 from dmick/wip-sync-pull

scripts/sync-pull: update argument check to look for $3

scripts/sync-pull: update argument check to look for $3

Catch the error where three arguments are not supplied.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2142 from ljflores/wip-crimson-pacific

ceph-dev-trigger/config/definitions: don't build crimson for pacific

Crimson changes will not be backported to pacific, so it doesn't make
sense to keep building it.

Signed-off-by: Laura Flores <lflores@redhat.com>

tcmu-runner: fix CEPH_BRANCH parameter description

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

tcmu-runner: set $RELEASE before it's used

get_rpm_dist() call is misplaced.  The Ceph repo URL ends up missing
the distro version which causes the job to fail after some timeouts:

  REPO_URL=https://shaman.ceph.com/api/repos/ceph/main/latest/centos//flavors/default/repo
  ...
  curl --fail -L https://shaman.ceph.com/api/repos/ceph/main/latest/centos//flavors/default/repo
  ...
  curl: (22) The requested URL returned error: 504 Gateway Timeout

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #2140 from dmick/wip-ceph-release-rpm

ceph-release-rpm build script missing space before =

ceph-release-rpm build script missing space before =

This causes a syntax error.  Also, the script didn't have -e set, so
the error went unnoticed

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2137 from ljflores/wip-ceph-dev-new-build

ceph-dev-new-build/config/definitions: give proper permission to remove a folder

ceph-dev-new-build/config/definitions: give proper permission to remove a folder

Solution to fix this in ceph-dev-new-build:
```
[EnvInject] - Inject global passwords.
[EnvInject] - Mask passwords that will be passed as build parameters.
Failed to evaluate name macro:org.jenkinsci.plugins.tokenmacro.MacroEvaluationException: Error processing tokens
[gigantic] $ /bin/sh -xe /tmp/jenkins6948878898411108393.sh
+ echo 'Cleaning up top-level workarea (shared among workspaces)'
Cleaning up top-level workarea (shared among workspaces)
+ sudo rm -rf dist
+ rm -rf venv
+ rm -rf release
rm: cannot remove 'release/18.0.0-4267-g48627125/ceph-18.0.0-4267-g48627125': Permission denied
rm: cannot remove 'release/18.0.0-4267-g48627125/version': Permission denied
rm: cannot remove 'release/18.0.0-4267-g48627125/ceph_18.0.0-4267-g48627125.orig.tar.gz': Permission denied
rm: cannot remove 'release/18.0.0-4267-g48627125/branch': Permission denied
rm: cannot remove 'release/18.0.0-4267-g48627125/ceph_18.0.0-4267-g48627125-1.dsc': Permission denied
rm: cannot remove 'release/18.0.0-4267-g48627125/other_envvars': Permission denied
rm: cannot remove 'release/18.0.0-4267-g48627125/ceph.spec': Permission denied
...
rm: cannot remove 'release/18.0.0-4267-g48627125/rpm/el8/BUILD/ceph-18.0.0-4267-g48627125/qa/suites/rgw/dbstore/cluster.yaml': Permission denied
rm: cannot remove 'release/18.0.0-4267-g48627125/rpm/el8/BUILD/ceph-18.0.0-4267-g48627125/qa/suites/rgw/dbstore/overrides.yaml': Permission denied
rm: cannot remove 'release/18.0.0-4267-g48627125/rpm/el8/BUILD/ceph-18.0.0-4267-g48627125/qa/suites/rgw/dbstore/s3tests-branch.yaml': Permission denied
rm: cannot remove 'release/18.0.0-4267-g48627125/rpm/el8/BUILD/ceph-18.0.0-4267-g48627125/qa/suites/rgw/dbstore/supported-random-distro$': Permission denied
rm: cannot remove 'release/18.0.0-4267-g48627125/rpm/el8/BUILD/ceph-18.0.0-4267-g48627125/qa/suites/rgw/dbstore/tasks/rgw_s3tests.yaml': Permission denied
rm: cannot remove 'release/18.0.0-4267-g48627125/rpm/el8/BUILD/ceph-18.0.0-4267-g48627125/qa/suites/rgw/dbstore/tasks/.qa': Permission denied
rm: cannot remove 'release/18.0.0-4267-g48627125/rpm/el8/BUILD/ceph-18.0.0-4267-g48627125/qa/suites/rgw/dbstore/.qa': Permission denied
rm: cannot remove 'release/18.0.0-4267-g48627125/rpm/el8/BUILD/ceph-18.0.0-4267-g48627125/qa/suites/rgw/hadoop-s3a/%': Permission denied
rm: cannot remove 'release/18.0.0-4267-g48627125/rpm/el8/BUILD/ceph-18.0.0-4267-g48627125/qa/suites/rgw/hadoop-s3a/clusters/fixed-2.yaml': Permission denied
```

Also add a line to list ownership before removing folders.

Signed-off-by: Laura Flores <lflores@redhat.com>
sdf

Merge pull request #2136 from petrutlucian94/win_bluestore

windows: use bluestore instead of memstore by default

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: use bluestore instead of memstore by default

We've been using memstore in order to put as much load as possible
on the Windows clients in a relatively short timeframe.

However, we've been hitting memstore segfaults:
https://tracker.ceph.com/issues/61416

For this reason, we'll make the OSD backend configurable,
defaulting to bluestore for now.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2135 from dmick/main

ceph-dev-new-trigger: keep 50 jobs to help diagnose dual builds

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2127 from dmick/main

Merge pull request #2134 from dmick/wip-build-utils

scripts/build_utils.sh: limit .gz files copied to chacra

Don't copy *every* .gz file from the release tree
(which includes the source tree); just ceph*.gz

Fixes: https://tracker.ceph.com/issues/61306
Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2132 from ceph/jjb-wheel

build_utils.sh: Install wheel

Signed-off-by: Zack Cerza <zack@redhat.com>

jenkins-job-builder: Install dataclasses

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2130 from ljflores/wip-requests-toolbelt

scripts/build_utils.sh: pin urllib3 and requests

scripts/build_utils.sh: pin urllib3 and requests

The latest version of chacractl requires requests-toolbelt==0.9.1.
This is not compatible with the latest version of urllib3, which
dropped support for appengine.

We need to pin urllib3 to an older version that still supports
appengine, and requests to an older version that still supports
the older urllib3 until charactl allows a higher version of
requests-toolbelt.

Fixes: https://tracker.ceph.com/issues/59652
Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #2129 from ceph/cephadm-ansible-reef

cephadm-ansible: add el9 testing

cephadm-ansible: add el9 testing

el9 RPMs weren't available so `el9` distribution was left commented out so
far. Given that we now have el9 packages, let's add el9 testing.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2128 from dmick/wip-remove-ceph-qa-emails

ceph-api-nightly, ceph-dashboard-cephadm-e2e-nightly: stop emails

ceph-api-nightly, ceph-dashboard-cephadm-e2e-nightly: stop emails

Remove the email notifications from the configuration.  They're
just cluttering the ceph-qa inbox and no one is acting on them.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph-dev-setup: save only 25 builds (rather than 2 weeks)

ceph-dev-setup is using 67G out of the 200G on /var/lib/jenkins,
far larger than any other job.  There's no reason to save all these jobs.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2122 from ceph/add_centos9_main

ceph-release-rpm: add CentOS 9 build

Merge pull request #2125 from dmick/wip-sign-rpms

Use createrepo_c, and change order of signing operations to avoid invalid repomd.xml sigs

scripts/sign-rpms: signing repomd.xml has to come after all updates

We had been signing, and then running createrepo, which changed repomd
and thus invalidated the signature.

Signed-off-by: Dan Mick <dmick@redhat.com>

scripts/sign-rpms: use createrepo_c on newer Ubuntu releases (post-focal)

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2124 from ceph/signing-fixes

scripts: signing fixes

sync-pull: use jq instead of python

We installed jq on the signer host so we do not need /usr/bin/python

scripts: make signing scripts executable

This makes it easier to run these interactively.

Merge pull request #2123 from rhcs-dashboard/grafana-update

ceph-grafana-trigger: update the grafana version

ceph-grafana-trigger: update the grafana version

Signed-off-by: Nizamudeen A <nia@redhat.com>

ceph-release-rpm: add CentOS 9 build

Merge pull request #2121 from ceph/cv-test-suite-update

ceph-volume: do not test filestore against reef onward

ceph-volume: do not test filestore against reef onward

filestore support has been dropped as of Reef release.
This commits drops ceph-volume filestore testing for reef onward.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #2120 from petrutlucian94/mgr_deps

windows: install python3-bcrypt dependency

windows: install python3-bcrypt dependency

The Windows job is failing because of a missing mgr dependency.

We'll update the setup script, ensuring that python3-bcrypt is
installed.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2119 from ceph/wip-reef

Update releases - add reef, remove octopus and older

Reviewed-by: Neha Ojha <nojha@redhat.com>

Update releases - add reef, remove octopus and older

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2118 from ceph/ceph-devstack

Add ceph-devstack job

Add ceph-devstack job

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2117 from ionutbalutoiu/fix-ssh-exec

utils: fix ssh_exec function

utils: fix ssh_exec function

We need to use `return` inside the function to properly return the
exit code of the function. Using `exit` will abruptly exit the
entire script, which is not what we want.

Signed-off-by: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>

Merge pull request #2116 from ionutbalutoiu/enable-win-event-logs

ceph-windows: Re-enable Windows event logs collection

ceph-windows: Enable Windows event log collection

The Windows event log collection was failing due `ssh_exec` abruptly
closing the connection, because the default timeout of `30 secs` was exceeded.

This change re-enables the Windows event log collection by increasing the
SSH timeout to `30 minutes`.

Additionally, the `ssh_exec` function is updated to print an appropriate
error message when SSH timeout is exceeded.

Signed-off-by: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>

ceph-windows: Cleanup collect-event-logs.ps1 script

* Rely on the default `$ErrorActionPreference` value (which is `Continue`).
* Add new function `SanitizeName` that it's used to sanitize the name of the log files.
  The function code existed before, but it was duplicated in the script.
* General PowerShell code cleanup.

Signed-off-by: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>

Merge pull request #2115 from petrutlucian94/log_collection_errors

windows: install dokany