Merge pull request #1398 from ceph/wip-build-wips-in-ceph-client

kernel: update the trigger list

kernel: update the trigger list

We didn't trigger on wip* branches like other ceph projects because
there was a lot of old branches matching that mask in the repository.
Now that they are cleaned up, drop the workarounds (testing*/master*)
and add wip*.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1397 from ceph/repo-cleanup

clean up shaman repos left behind after a build

tcmu-runner: remove custom shaman repos added after a build

Signed-off-by: Alfredo Deza <adeza@redhat.com>

nfs-ganesha-stable: remove custom repo/list configuration files added

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1396 from ceph/wip-centos-sclo-rh-source-skip

ceph-*-build: skip unavailable repo when yum-builddep

ceph-*-build: skip unavailable repo when yum-builddep

As centos-sclo-rh-source leads us to 404 at this moment. and we are not
using the source repo for building ceph. so we can just skip any
unavailable repo.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #1394 from ceph/revert-scm-changes

Revert "*/scm: only pull the PR under test"

Revert "*/scm: only pull the PR under test"

This reverts commit 4eae5698a6a1383c69338ec7781e4cee6c087a9f.

It breaks all pull-request jobs for ceph-volume at least, haven't been
able to investigate all other affected

Merge pull request #1393 from tchaikov/wip-crimson-perf

ceph-perf-pull-requests: add ceph-perf-pull-requests

Merge pull request #1392 from tchaikov/wip-scm-refspec

*/scm: only pull the PR under test

ceph-perf-pull-requests: add ceph-perf-pull-requests

for running cbt test against PR and master, then comparing them

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1378 from dmick/wip-dmick

Add container builds using ceph-container.git for centos7

Merge pull request #1391 from hjensas/issue#1390

pr-syntax-check - install all requirements

pr-syntax-check - install all requirements

This change removes the grep for 'ansible' when installing
requirements in ceph-ansible-pr-syntax-check. The grep
cause onlye the ansible requirement to be installed, ignoring
the other packages. This causes an issue in PR[1] which
introduce a filter plugin that depend on the netaddr package.

[1] https://github.com/ceph/ceph-ansible/pull/4339

Closes: #1390
Signed-off-by: Harald Jensås <hjensas@redhat.com>

ceph-build-pull-requests: pass --config-xml to "jenkins-jobs test"

to silence warnings like

WARNING:jenkins_jobs.cli.subcommand.test:(Deprecated) The default output
behavior of `jenkins-jobs test` when given the --output flag will change
in JJB 3.0. Instead of writing jobs to OUTPUT/jobname; they will be
written to OUTPUT/jobname/config.xml. The new behavior can be enabled by
the passing `--config-xml` parameter.

also redirect the test result to /dev/null, otherwise the test will fail
because the output directory may already exist.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-build-pull-requests: ping jjb to 3.0.2

bump jjb's version

Signed-off-by: Kefu Chai <kchai@redhat.com>

*/scm: only pull the PR under test

see
https://github.com/jenkinsci/ghprb-plugin/blob/master/README.md#creating-a-job

no need to pull all PR references of the remote repo. it'd be faster
just pull the references of a single PR. actually, what we need is but

- refs/pull/${pr}/head
- refs/pull/${pr}/merge

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1389 from tchaikov/wip-python-rpm-macro

ceph-*build: install python36-devel for rpm macros

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-*build: install python36-devel for rpm macros

see also 100a0fb5
see also https://github.com/ceph/ceph/pull/30190

Fixes: https://tracker.ceph.com/issues/41603
Signed-off-by: Kefu Chai <kchai@redhat.com>

Conditionally add container builds for wip- branches

New parameter CI_CONTAINER ('true' or 'false') controls behavior;
if set to 'true', ceph-container.git will be used to build a centos7
daemon-base default flavor container and push it to a repo server.
This requires waiting until the shaman repo is built.

New parameters CONTAINER_REPO_{HOSTNAME, ORGANIZATION} set where
to push the resultant container.

If repo doesn't become ready or if the actual build/push fails, don't
fail the overall build.  Maybe we can come up with a better way
to indicate this than just a log notation

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph-dev-new-build: add quay.io credentials for cephci repo

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph-dev-new-*: add CONTAINER_BRANCH and check it out

In case a container build is needed, check out $CONTAINER_BRANCH
of ceph-container.git to use to build it

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #1388 from ceph/prune-fix

scripts: use a loop instead of globbing directly

scripts: use a loop instead of globbing directly

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1387 from tchaikov/wip=40646

Revert "ceph-*build/build/setup_rpm: downgrade to gcc-c++-8.2.1"

Revert "ceph-*build/build/setup_rpm: downgrade to gcc-c++-8.2.1"

This reverts commit cf2c26b181a86738e2a8bb17545f45d6aa1c7c45.

since https://bugzilla.redhat.com/show_bug.cgi?id=1726630 has been
fixed. and we have devtoolset-8-gcc-c++-8.3.1-3.1.el7.x86_64.rpm.

no reason to stick to devtoolset-8-gcc-c++-8.2.1-3 anymore.

Fixes: https://tracker.ceph.com/issues/40646
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1386 from ceph/prune-fix

scripts: can't use globbing with quotes

scripts: can't use globbing with quotes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1385 from ceph/prune-fix

scripts: do not specify directories with find -delete, rm the path

Reviewed-by: Kefu Chai <kchai@redhat.com>

scripts: do not specify directories with find -delete, rm the path

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1383 from ceph/prune-fix

scripts: use find's -delete option to prevent 'argument too long'

Reviewed-by: Kefu Chai <kchai@redhat.com>

scripts: use find's -delete option to prevent 'argument too long'

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1382 from ceph/fix-cv-tox

ceph-volume-pr: remove escape which doubles the curly brackets

ceph-volume-pr: remove escape which doubles the curly brackes resulting in syntax errors

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1381 from ceph/submodules-wipe

ceph-pr-submodules: wipe the workspace to prevent repo corruption

ceph-pr-submodules: wipe the workspace to prevent repo corruption

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1380 from ceph/ceph-signed-wipe

ceph-pr-commits: wipe workspace to prevent repo corruption

ceph-pr-commits: wipe workspace to prevent repo corruption

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1377 from ceph/ceph-volume-cleanup

ceph-volume: cleanup workspace before running tests

ceph-volume use wipe-workspace in Git to cleanup before running

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1379 from ceph/wip-signed-master

ceph-pr-commits: add trigger phrase

ceph-pr-commits: add trigger phrase

When Jenkins fails to clone a branch, signed-off checks will fail. If
all other checks have passed, there's no reason to force a full run of
Jenkins jobs again on that PR. This PR introduces `jenkins test signed`
trigger phrase to allow re-running the pr-commit check.

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1376 from ceph/wip-enable-leap15

enable leap15 for ceph-dev-trigger and ceph-dev-new-trigger

ceph-dev-trigger: enable leap15 distro

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

ceph-dev-new-trigger: enable leap15 distro

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1374 from ceph/wip-scsi-checks

ceph-iscsi-stable: Check for specific package files

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-iscsi-stable: Check for specific package files

The -f `-*` got expanded too many times if there are multiple matches which broke the `if` statement.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1373 from ceph/wip-configshell-fb

configshell-fb: initial builder for python-configshell packages

configshell-fb: initial builder for python-configshell packages

This updated package is required by ceph-iscsi but won't be available
until CentOS 7.7(ish).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1372 from ceph/wip-ceph-iscsi-stable

ceph-iscsi-stable: fix revision to '-1' if building at tag

ceph-iscsi-stable: fix revision to '-1' if building at tag

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1371 from ceph/wip-ceph-iscsi-stable

ceph-iscsi-stable: inject missing credentials

ceph-iscsi-stable: inject missing credentials

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1370 from ceph/wip-ceph-iscsi-stable

ceph-iscsi-stable: query correct chacra host for uploads

ceph-iscsi-stable: query correct chacra host for uploads

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1369 from ceph/wip-ceph-iscsi-stable

ceph-iscsi-stable: incorporate consolidated repo changes

ceph-iscsi-stable: incorporate consolidated repo changes

ceph-iscsi-config and ceph-iscsi-cli are now merged into
the ceph-iscsi repo.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1368 from ceph/wip-leap15-available-dist

leap15: add leap15 to available dist

leap15: add leap15 to available dist

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1351 from callithea/wip-ceph-dashboard-prs-backend-api

ceph-dashboard-pr-backend: add job config

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-dashboard-pr-backend: add job config

Signed-off-by: Laura Paduano <lpaduano@suse.com>
Signed-off-by: alfonsomthd <almartin@redhat.com>

Merge pull request #1365 from ceph/wip-osc-build-scripts

osc build scripts

Merge pull request #1367 from tchaikov/wip-upload-ddeb-to-chacra

ceph*build/build/build_deb: upload *.ddeb packages

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph*build/build/build_deb: upload *.ddeb packages

ubuntu names *dbgsym* packages .ddeb in hope to put them into a
separated repo. while debian just uses the plain .deb extension for them.
so, to include the dbgsym packages created by ubuntu, we need to upload
the .ddeb packages as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1343 from ceph/github-token

ceph-volume: allow passing GITHUB_TOKEN as a password to manually set github status

ceph-dev-new-build: add osc build scripts

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

ceph-dev-build: add osc build scripts

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

validate_rpm: reuse rpm validation from another project

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

osc: add osc related scripts:
- validate_osc
- setup_osc
- build_osc
Supposed for adding them later to ceph-dev-new-build or ceph-dev-build

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1366 from ceph/wip-jb-group

slave.yml: Add jenkins-build user to jenkins-build group

slave.yml: Add jenkins-build user to jenkins-build group

On ubuntu, at least, with the addition of the "create a jenkins-build group" task, the creation of the user wasn't adding the user to the group.  I discovered this by seeing a docker command fail after the socket was 'chgrp'ed to jenkins-build

```
+ sudo chgrp jenkins-build /var/run/docker.sock
+ do_clean
+ make clean.all
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock
```

I logged in to the slave and confirmed this was the case.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1361 from ceph/wip-release-from-version

build_utils: add release from version

build_utils: use virtualenv -p python2.7 instead virtualenv

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

build_utils: fix get_rpm_dist openSUSE detection

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

build_utils: add release from version function

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1359 from ceph/wip-leap15-slave

ansible/slave: add opensuse leap15 osc support

ansible/slave: add opensuse leap15 osc support

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1360 from ceph/wip-tserlin-nfs-ganesha-disable-xenial

nfs-ganesha: don't build on xenial

nfs-ganesha: don't build on xenial

Ceph no longer builds its master branch on Xenial. See:
https://github.com/ceph/ceph-build/pull/1345

So we can no longer build the latest nfs-ganesha V2.9-dev on Xenial.

Merge pull request #1357 from tchaikov/wip-ceph-pr-docs/python3

ceph-pr-docs: install python3 packages

Merge pull request #1355 from ceph/fix-validate

ceph-dev-build: fix deb and rpm validate

ceph-pr-docs: install python3 packages

see also https://github.com/ceph/ceph/pull/29528

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-build: fix deb and rpm validate

validate_deb - not redhat does not mean it is deb distro,
there can be other packaging systems
validate_rpm - we do not want to proceed for zypper based,
we only want to proceed for redhat like repo

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1346 from ceph/nfs_ganesha_disable_lizardfs

nfs-ganesha: Disable LizardFS FSAL

Merge pull request #1350 from ceph/guits-remove-dashboard

ceph-ansible: remove dashboard scenarios

ceph-ansible: remove dashboard scenarios

since dashboard is enabled by default, we test it in all_daemons
scenario.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1349 from tchaikov/wip-grep-cleanup

ceph-dev-new-trigger: only use positive match for filter

Merge pull request #1354 from tchaikov/wip-pin-jjb

ceph-build-pull-requests: ping jjb to 2.10.1

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-build-pull-requests: ping jjb to 2.10.1

jjb 3.0 brings some incompatible behavior:

* it tries to expand the strings like ${FOO} using the parameters,
  so if we continue using include-raw, jjb test will fail like

```
Failure formatting job
....
ERROR:root:Failure formatting job '{
with itself
Traceback (most recent call last):
...
  File
"/tmp/venv.Pa7CUfDCFK/local/lib/python2.7/site-packages/jenkins_jobs/formatter.py",
line 43, in deep_format
    raise JenkinsJobsException(desc)
jenkins_jobs.errors.JenkinsJobsException:
...
...
TEMPVENV=$(mktemp -td venv.XXXXXXXXXX)
```
* also, it fails to pass down some parameters to the build job, for
  instance, the `WORKSPACE` is not passed anymore.

both these change break the "test" job, as we are using `include-raw`
everywhere, and we also use `${WORKSAPCE}` in some yml files.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1352 from tchaikov/wip-include-raw

s/!include-raw/!include-raw:/

s/!include-raw/!include-raw:/

the former is deprecated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-trigger: only use positive match for filter

there is no need to put
```
grep -v '\(jewel\|kraken\|luminous\)' | grep '\(mimic\|nautilus\)'
```

as we are not likely to have both "jewel" and "mimic" in a branch name.

this change is a follow-up of the discussion when reviewing #1345

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1347 from ceph/jan--f-add-notifications

add Jans email for nightly notifications

Merge branch 'master' into jan--f-add-notifications

Merge pull request #1348 from ceph/master-fix

ceph-dev-trigger: add missing escape for grep on master

ceph-dev-trigger: add missing escape for grep on master

Signed-off-by: Alfredo Deza <adeza@redhat.com>

add Jans email for nightly notifications

Merge pull request #1345 from ceph/rm-xenial

ceph-dev-*: remove xenial for master and octopus

ceph-dev-trigger: add conditional step for master and octopus

Signed-off-by: Alfredo Deza <adeza@redhat.com>