Merge pull request #1715 from tchaikov/wip-debian-profiles

ceph-*build: pass --profiles nocheck to pbuilder

Merge pull request #1718 from ceph/fix-brach-api

ceph-{api,dashboard}-pr: fix source branch ref

ceph-{api,dashboard}-pr: fix source branch ref

Applying same change as already applied by
https://github.com/ceph/ceph-build/pull/1426. A divergence has been
observed between `make check` job (ceph-pull-request) and the `make` in
ceph-pr-api job. For the same PR:

* ceph-pull-request `make check` passes
* ceph-pr-api `make` fails

This fix has been suggested by @sebastian-philipp

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1717 from ceph/wip-stopit-apt

ansible: Disable daily apt transaction on Ephemeral slaves

ansible: Disable daily apt transaction on Ephemeral slaves

Signed-off-by: David Galloway <dgallowa@redhat.com>

scripts/build_utils.sh: pass --profiles nocheck to pbuilder

so we can do profile-based build on debian derivative distros.

"nocheck" is one of the registered profile names. be specifying
"nocheck", we can reduce the build depends required by building Ceph
once the related Build-Depends in debian/control are marked with "nocheck".

see https://wiki.debian.org/BuildProfileSpec

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-*build: extract build_deb into scripts/build_utils.sh

so we can modify them in a single place

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-build: define chacra_url in 'setup_deb'

and use this variable in build_deb. this change pave the road to a
consolidated build_deb file which can be shared by all ceph*-build jobs

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1714 from ceph/fix_teardown

ceph-ansible: fix teardown

ceph-ansible: only exclude non_container-update

c83cd218b3a6fd387b5de2a0e9380c3fb3a26a3c excludes both `container` and
`non_container` jobs.

Only `non_container` should be excluded

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-ansible: fix teardown

If this playbook fails for some reason, it interrupts the teardown so
VMs aren't destroyed.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1713 from ceph/ca_disable_update_5.0

ceph-ansible: disable update scenario on octopus

ceph-ansible: disable update scenario on octopus

`ceph-volume simple scan` in octopus is broken and make the upgrade
testing scenario failing.
Until 15.2.8 is out, which will be the next release including the fix
for this. We can't make this scenario passing the CI.
Let's disable it temporarily.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1712 from ceph/wip-insecure

build_utils: curl docker-mirror insecurely

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

build_utils: curl docker-mirror insecurely

I had initially planned on installing the self-signed cert globally but that was not advised.  See https://github.com/ceph/ceph-build/pull/1698#issuecomment-725840305

So we should curl insecurely.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1711 from ceph/guits-cv_ca

ceph-volume: set CEPH_ANSIBLE_BRANCH properly

ceph-volume: set CEPH_ANSIBLE_BRANCH properly

At the moment it's using ceph-ansible@master to deploy octopus and
nautilus which is wrong.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1710 from ceph/guits-fix_cv_nautilus_pr

ceph-volume: fix nautilus PRs job

ceph-volume: fix nautilus PRs job

ceph-volume PRs against nautilus are failing for a while because it
tries to run against centos8 while the tox environment only defines
centos7 environments.

Let's enforce `DISTRO` environment variable when `ghprbTargetBranch` is
`nautilus` so we override the environment variable injected by Jenkins.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1692 from ceph/jan--f-patch-1

ceph-volume-nightly: Drop mimic from nightlies

Merge pull request #1708 from tchaikov/wip-revert-skip-when-not-needed

ceph-pr,api-pr: Revert" skip when not needed"

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

ceph-pr,api-pr: Revert" skip when not needed"

This reverts commit ae9a52ba2faa0b7faf0b9e0c6d3fd289cdd576ea.

Merge pull request #1593 from tchaikov/wip-enable-javapackages-tools

script/build_utils: enable javapackages-tools for building java binding

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

script/build_utils: enable javapackages-tools for building java binding

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1706 from ceph/wip-sepia-create

sepia-fog-images: Create FOG image template if missing

Merge pull request #1705 from ceph/wip-exclude_region-master

ceph-pr,api-pr: skip when not needed

Reviewed-by: Kefu Chai <kchai@redhat.com>

sepia-fog-images: Create FOG image template if missing

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-pr,api-pr: skip when not needed

Skip running make-check and ceph-api tests when the PR only contains
changes in certain dirs (doc/, .github/) or certain extensions (.rst,
.md, .txt). In any case, a Jenkins job can still be manually triggered
(`jenkins test...`).

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1704 from neha-ojha/wip-fix-48162

scripts/build_utils.sh: use correct DIST value for el7

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

scripts/build_utils.sh: use correct DIST value for el7

Fixes a3f26c8ab781a5ffe54aedf45be9e373bede4d6c, which ended up
comparing [[ el7 == centos7 ]].

Follow-on fix for: https://tracker.ceph.com/issues/48162
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #1702 from ceph/ceph_container_inject_docker_credentials

ceph-container: inject registry credentials

ceph-container: inject registry credentials

To avoid docker hub rate limit, we should use the authenticate docker
hub user like we're currently using in the build and publish jobs.
This adds the credentials for both ceph-container-prs and
ceph-container-nightly jobs.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1698 from ceph/wip-dockercerts

Use internal docker registry mirror

Merge pull request #1701 from ceph/wip-disable-leap15-on-nautilus

ceph-dev-new-trigger: disable leap15 for nautilus

ansible: Install docker/podman and docker-mirror TLS cert

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-dev*build: Pull centos images from our registry mirror

Since docker started rate limiting on Nov2, we've often seen errors that we hit our `pull` limit.  This should greatly reduce the number of times we're going to docker.io.  Only ephemeral builders would pull from docker.io once this is merged and those builders' IP changes when they're spun up.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1700 from tchaikov/wip-48162

ceph-dev-*-build: reset CI_CONTAINER when building octopus for el7

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

ceph-dev-new-trigger: disable leap15 for nautilus

We do not need for now opensuse-15.1 builds, since
support of it is limited.

The current version of Leap is 15.2 and the 15.1
is going to be EOL, thus we are not going to spend
resources on it.

We can disable it back if required in future.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

ceph-dev-*-build: reset CI_CONTAINER when building octopus for el7

because the consumers of ceph container images are referencing the
required images using
"${CONTAINER_REPO_HOSTNAME}/${CONTAINER_REPO_ORGANIZATION}/ceph:${SHA1}".
see ceph-container/contrib/build-push-ceph-container-imgs.sh.

but in a recent change, we push both the images built for el7 and for
el8 with the same tag. and the last one wins. this breaks the tests
which expect the cephadm and other tools included in an image built for
el8. but these tools are not included by el7 because of the missing
dependencies.

so we have at least two options

- stop building octopus+el7 container images
- use a different tag for octopus+el7 to avoid the racing and name collision.

before the 2nd change lands on ceph-container, we should have the 1st
one in ceph-build to unblock the builds of octopus.

Fixes: https://tracker.ceph.com/issues/48162
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1695 from ceph/wip-osc-build-root-typo

ansible: fix osc build-root typo for opensuse worker

ansible: fix osc build-root typo for opensuse worker

Fixes: https://tracker.ceph.com/issues/48141
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1685 from ceph/wip-change-osc-builroot

ceph-dev-build: Change build path for Leap builds

Merge pull request #1694 from tchaikov/wip-seastar-allocator

ceph-*-setup: use default allocator for crimson flavor

Reviewed-by: Samuel Just <sjust@redhat.com>

ceph-dev-build: Change build path for Leap builds

Previously, packages were being written to `/var/tmp` which does not live on the intended 1TB SSD.

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-*-setup: use default allocator for crimson flavor

since we've merged https://github.com/ceph/ceph/pull/37887

This reverts commit 3aa35a92b466212c0c3c715781a8aab8a5e0136c.

Merge pull request #1628 from tchaikov/wip-images-with-flavors

ceph-*-build: pass CONTAINER_FLAVOR to build-push-ceph-container-imgs.sh

Revert "ceph-dev-*-build: reset CI_CONTAINER when building nautilus"

This reverts commit d86ed63576a7dcc213c211b7678da2af6b6de108.

if we are able to build container images for nautilus on centos7 and
centos8, there is no need to skip this step.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-*-build: pass CONTAINER_FLAVOR to build-push-ceph-container-imgs.sh

so we can specify the distro and its version

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1693 from ceph/ceph_container_flake8_selinux

ceph-container-flake8: add selinux flag to volume

ceph-container-flake8: add selinux flag to volume

Without the selinux flag on the container volume, the flake8 command
fails with:

E902 IOError: [Errno 13] Permission denied: '/envglobals.py'

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Drop mimic from nightlies

Not sure anyone is interested in these, they are currently broken.

Merge pull request #1691 from huww98/patch-1

Fix Signed-off-by check with non-ASCII names

Reviewed-by: Kefu Chai <kchai@redhat.com>

Fix Signed-off-by check with non-ASCII names

Blocking https://github.com/ceph/ceph/pull/37810 https://github.com/ceph/ceph/pull/37849

Merge pull request #1690 from ceph/wip-pipeline-master

ceph: multibranch: trust contributors' forks

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph: multibranch: trust contributors' forks

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1687 from ceph/wip-docsdocs

ceph-pr-render-docs: Use RTD URL

Merge pull request #1689 from ceph/ceph_container_lint_rm_option

ceph-container-lint: use --rm docker option

ceph-container-lint: use --rm docker option

The container lint job, which run shellcheck in a docker container,
doesn't use the --rm docker option so the container aren't removed
after their execution.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

ceph-pr-render-docs: Use RTD URL

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1686 from tchaikov/wip-ceph-pr-commits

ceph-pr-commits: use origin/master as the base commit

ceph-pr-commits: use origin/master as the base commit

instead of using `${sha1}^1...${sha1}^2` use
`origin/master...origin/pr/${ghprbPullId}/head` for getting the range of
commits introduced by a PR. as per `man git-diff`

> "git diff A...B" is equivalent to "git diff $(git-merge-base A B) B".

and `origin/${ghprbTargetBranch}` and `origin/pr/${ghprbPullId}/head` are fetched by
the ghprb-plugin plugin, so it's safe to use them. see
https://plugins.jenkins.io/ghprb/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1683 from ceph/wip-sha1

ceph-pr-commits: Use sha1 instead of GIT_COMMIT

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-pr-commits: Use sha1 instead of GIT_COMMIT

I think GIT_COMMIT is the merge commit.  Everything Kefu and I could find indicated it should be valid to use here.

Switching to ${sha1} works though.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1682 from tchaikov/wip-ceph-pr-commits-decode

ceph-pr-commits: skip non-ascii chars

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

ceph-pr-commits: skip non-ascii chars

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1681 from ceph/wip-bump-jjb

jjb: Unpin jjb version

jjb: Unpin jjb version

Pinning it keeps biting us more often than fixing/updating job yamls would.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1668 from ceph/wip-pipeline-master

ceph: enable multibranch pipeline

Merge pull request #1678 from tchaikov/wip-drop-bashism

ceph-perf-pull-requests: s/source/./

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1680 from tchaikov/wip-ceph-pr-commits-junit

ceph-pr-commits: skip test for doc only changes

Reviewed-by: Zack Cerza <zack@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>

ceph-pr-commits: add a doc_test checking title of commit message

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-pr-commits: port test_commits.py to py3

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-pr-commits: skip "code_test" tests for commits changing "doc/"

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-pr-commits: add "code_test" for test_signed_off_by() test

this test should only apply to changes which modifies code. for the doc
only changes, we should skip this test for lowering the bar for
contributors willing to improve the document.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-perf-pull-requests: s/source/./

turns out we have ubuntu test nodes labeled with "performance", and
they use dash for "sh", and jenkins use "sh" for running the embedded
scripts.

let's avoid bashism, and use "." instead.

this change is a follow-up of 4e0a8987c7d19004d33c153b107057f684b547de

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1677 from ceph/wip-dpkg-a

ceph-pr-api: Clean dpkg db on job abort

ceph-pr-api: Clean dpkg db on job abort

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1676 from ceph/wip-xml-signed

ceph-pr-commits: Allow empty junit results

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-pr-commits: Allow empty junit results

report.xml wasn't getting written for docs-only changes because the job `exit 0`s.  The junit plugin was failing the build in those instances but we don't need or want it to.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1675 from ceph/guits-remove_ooo_collocation

ceph-ansible: remove ooo_collocation

ceph-ansible: remove ooo_collocation

This job is not needed anymore, we already have 'collocation' for that.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph: add multibranch pipeline

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1672 from tchaikov/wip-ceph-deploy-tag

ceph-deploy-tag: use ceph-build:origin/master

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1674 from dmick/master

ceph-dev{,-new}-build: add SHA1 to build name

ceph-deploy-build: inject chacractl credentials

otherwise we would have

requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://chacra.ceph.com/binaries/ceph-deploy/test/c6ff0a05523cdeda86ebac732dd186620261dc01/centos/7/source/

when posting the created package to chacra

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: remove duplicated get_rpm_dist()

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-deploy-*: install build deps for python3

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-deploy-build: build on bionic and centos8 for python3 package

* s/xenial/bionic/ for building python3 package
* add centos8

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-deploy-tag: use ceph-build:origin/master

otherwise it fails when trying to use the release.yml playbook:

ERROR! the playbook: release.yml could not be found
Build step 'Execute shell' marked build as failure

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1673 from ceph/revert-1648-wip-junit_dashboard-master

Revert "test: analyze JUnit results in PR jobs"

ceph-dev{,-new}-build: add SHA1 to build name

Signed-off-by: Dan Mick <dmick@redhat.com>

Revert "test: analyze JUnit results in PR jobs"

Merge pull request #1671 from ceph/wip-signed-docs

ceph-pr-commits: Don't require signed commits for doc/ dir

ceph-pr-commits: Don't require signed commits for doc/ dir

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1648 from ceph/wip-junit_dashboard-master

test: analyze JUnit results in PR jobs

Merge pull request #1670 from dmick/master

ceph-dev, ceph-dev-new: get shorter branch name, add SHA1 to build name

test: add JUnit result analysis to jobs

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

ceph-dev, ceph-dev-new: add SHA1 to build names

Signed-off-by: Dan Mick <dmick@redhat.com>

ceph-dev, ceph-dev-new: copy branch artifact too for short name

BRANCH is set by the github plugin to, say, origin/master.  The
-setup jobs change that back to just 'master', so, copy their
saved env-setting artifact 'branch' as well so the parent job
can also use the short branch name in its build name.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1669 from dmick/master

ceph-dev, ceph-dev-new: copyartifact permission: no spaces