prune-quay.py: handle aarch64 images

prune-quay.py: limit shaman search to ceph project

Some of the searches were broad enough to find hits in other projects

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1758 from ceph/wip-tmpreaper-notimeout

ansible: Disable timeout on tmp{watch,reaper}

ansible: Disable timeout on tmp{watch,reaper}

From the man page:

```
       -T x, --runtime=x
              Execution of tmpreaper will aborted after x seconds; this is to prevent attacks that create many, many files.  By default the timeout is set to 55 seconds.  A value of 0 will disable this feature, which is not advised as this feature prevents possible
              race-conditions between different instances of tmpreaper.
```

So tmpreaper wasn't running for the past week since I set this up because there was so much to clean up that it'd time out.  These are Jenkins builders where the primary user already has sudo access so there's no real use to locking down tmpreaper.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1757 from ceph/ceph_ansible_nightly_pacific_centos_release

ceph_ansible_nightly: add pacific centos release

ceph_ansible_nightly: add pacific centos release

This adds Pacific release based on CentOS 8 like Octopus.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1756 from ceph/wip-win32-build

ceph-dev-new-build: cleanup root-owned mingw64 pbuilder environment

Reviewed-by: David Galloway <dgallowa@redhat.com>

ceph-dev-new-build: cleanup root-owned mingw64 pbuilder environment

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1754 from ceph/wip-win32-build-trigger

ceph-dev-new-trigger: build windows for pacific and master

Merge pull request #1755 from petrutlucian94/win_pbuilder

ceph-dev-new-build: cross-compile Windows targets using pbuilder

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-dev-new-build: cross-compile Windows targets using pbuilder

The Windows builds that are generated right now are unusable due to
known Mingw runtime bugs.

We need Ubuntu Focal or later, for which reason we're going to use
pbuilder.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #1753 from ceph/wip-win32-build

ceph-dev-new-build: tweak chacra binary and repo endpoints for windows

ceph-dev-new-build: use dynamic chacra url for windows build

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-dev-new-trigger: build windows for pacific and master

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-dev-new-build: tweak chacra binary and repo endpoints for windows

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1752 from ceph/wip-win32-build

ceph-dev-new-build: extract artifacts prior to attempting windows build

ceph-dev-new-build: extract artifacts prior to attempting windows build

Also fixed the chacra endpoints to be consistent between the setup and
build steps.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1751 from ceph/wip-win32-build

ceph-dev-new-build: fixed windows build environment variables

ceph-dev-new-build: fixed windows build environment variables

The environment variables should be placed before the command.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1722 from petrutlucian94/win32

ceph-dev-new: add Windows build

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1750 from ceph/wip-notcmalloc

Disable notcmalloc builds!

ceph-dev-new: add Windows build

We're going to do a Windows build using Mingw as part of the
ceph-dev-new job.

"windows" is added as a valid distro. The "validate_mingw" script
will ensure that we're running on top of a Linux distro that the
Windows build script supports (currently just Ubuntu and Suse
distros).

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Disable notcmalloc builds!

We no longer need notmcalloc packages.  Valgrind works now.

See https://github.com/ceph/ceph/pull/39561

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1749 from ceph/wip-defaults

Update ceph ARCH defaults

Update ceph ARCH defaults

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1747 from ceph/wip-noarmcrimson

ceph-dev: Disable arm64 notcmalloc and crimson builds

ceph-dev: Disable arm64 notcmalloc and crimson builds

Nobody will use them and crimson is failing anyway

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1746 from ceph/wip-always

always set facts

ansible: Always set_facts

Without these set_fact tasks having the always tag, a few conditionals (e.g., permanent|bool) and vars don't get set.  So if we're running `ansible-playbook --tags tmp`, for example, all the tasks get skipped because `permanent|bool` didn't get set automatically.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1745 from ceph/wip-tmpcleaner

ansible: Install tmpreaper/tmpwatch and clean /tmp daily

ansible: Install tmpreaper/tmpwatch and clean /tmp daily

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1744 from ceph/wip-make-ken-happy-again

s/ceph-dev-nightly/ceph-dev-cron

s/ceph-dev-nightly/ceph-dev-cron

Fixes: https://github.com/ceph/ceph-build/pull/1743#issuecomment-780840190
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1743 from ceph/wip-cephnightly

Switch to building ceph.git branches twice daily

ceph-dev-trigger: Disable job for now

We no longer want to build packages for every merge in active branches.  ceph-dev-nightly will build each active branch twice daily instead.

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-dev-nightly: Create job

Build active branches twice daily if changes are found

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1741 from ceph/remove_ca_add_osds

ceph-ansible: remove add_osds jobs

ceph-ansible: remove add_osds jobs

The add-osds.yml playbook was dropped as of 5.0

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1740 from ceph/quick_patch_cv

ceph-volume-prs: use ceph-ansible@stable-6.0 for pacific

ceph-volume-prs: use ceph-ansible@stable-6.0 for pacific

Since, ceph-ansible stable-6.0 has been branched for deploying pacific,
we should use this branch in ceph-volume testing.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1739 from ceph/ca_6.0

ceph-ansible: branch stable-6.0

ceph-ansible: branch stable-6.0

This commit brings the required modification to branch stable-6.0

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1738 from ceph/wip-reenable-arm

ceph-dev*: Enable dev builds for arm64

ceph-dev*: Enable dev builds for arm64

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1737 from ceph/wip-docs-easier

Stop building binaries for docs-only PRs

Merge pull request #1734 from dmick/wip-quay-fix-ref-delete

Fix prune-quay.py's deletion of busy (well-known) refs

ceph-pr-api: Don't run for docs-only PRs

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-pull-requests: Skip make check on a docs-only PR

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-pr-commits: Move 'docs only' check to build_utils for reuse

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1736 from ceph/wip-notcmalloc-boost-valgrind

build: notcmalloc builds enable WITH_BOOST_VALGRIND

Reviewed-by: Kefu Chai <kchai@redhat.com>

build: notcmalloc builds enable WITH_BOOST_VALGRIND

this flag silences a false-positive from valgrind in boost::context:
```
<error>
  <unique>0x0</unique>
  <tid>124</tid>
  <kind>InvalidRead</kind>
  <what>Invalid read of size 8</what>
  <stack>
    <frame>
      <ip>0x52FF7E7</ip>
      <obj>/usr/lib64/libradosgw.so.2.0.0</obj>
      <fn>spawn::detail::continuation_context::resume()</fn>
    </frame>
  </stack>
  <auxwhat>Address 0x57f65728 is in a rw- anonymous segment</auxwhat>
</error>
```

Fixes: https://tracker.ceph.com/issues/48963
Signed-off-by: Casey Bodley <cbodley@redhat.com>

prune-quay.py: fix deletion of busy ref names (master etc.)

We must only delete the ref name if it refers to the same image
as the ref-shortsha1-el form of the name; if it's been pushed as a
new build of ref from a new sha1, it's no longer the image we
need to delete.  Verify the ref tag points to the same image as the
ref-shortsha1-el tag.

Signed-off-by: Dan Mick <dmick@redhat.com>

prune-quay.py: improve verbose logging

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1735 from dmick/wip-tcmu-runner-arm64

tcmu-runner: fix for arm64/aarch64

tcmu-runner: fix for arm64/aarch64

Add "available arch" arm64, and fix build_rpm script to use aarch64
Also limit aarch64 build to CentOS 8

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1733 from dmick/master

scripts/build_utils.sh: handle lowercase CentOS dnf repo names

Reviewed-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: handle lowercase CentOS dnf repo names

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1732 from ceph/wip-disable-leap15-for-master

disable leap15 for master and pacific

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

disable leap15 for master and pacific

SUSE is not going to support ceph except of octopus anymore

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1731 from tchaikov/wip-opensuse-q

ceph-dev-build: update {build,setup}_osc for quincy

ceph-dev-build: update {build,setup}_osc for quincy

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1730 from tchaikov/wip-build_utils-q

scripts/build_utils: map 17.* to quincy

Merge pull request #1664 from tchaikov/wip-zbd

ceph-pull-requests: enable WITH_ZBD

Reviewed-by: David Galloway <dgallowa@redhat.com>

ceph-pull-requests: enable WITH_ZBD on ubuntu

to avoid zbd backend bitrot, and for build test it in our "make check"
builds.

currently, libzbd-devel and libzbd packages were only prebuilt on bionic,
so only enable this on ubuntu distros.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: map 17.* to quincy

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: map 0.80.* to firefly

it's not giant. since both branches are EOL, but this change helps to
silence a warning from ShellCheck.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1728 from ceph/wip-pacific-cephci

ceph*trigger: Add pacific conditional steps

ceph*trigger: Add pacific conditional steps

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1726 from ceph/wip-t-test

teuthology-pull-requests: whitelist red-hat-storage gh group

Merge pull request #1727 from ceph/wip-docs-timeout

ceph-pr-docs: Timeout after an hour

Merge pull request #1724 from ceph/wip-pacific

add pacific targets

ceph-pr-docs: Timeout after an hour

Signed-off-by: David Galloway <dgallowa@redhat.com>

teuthology-pull-requests: whitelist red-hat-storage gh group

Most of Ceph QE is in this group but may not necessarily be in (or need to be in) the 'ceph' group.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1725 from ceph/wip-a

teuthology-pull-requests: Do not run on arm64

teuthology-pull-requests: Do not run on arm64

Signed-off-by: David Galloway <dgallowa@redhat.com>

use tag for ceph-ansible until we have a stable-6.0 branch

add pacific targets

Merge pull request #1723 from ceph/ca-sleep

ceph-ansible: refact write_collect_logs_playbook

ceph-ansible: refact write_collect_logs_playbook

This playbook doesn't work in containerized context.
Importing ceph-defaults and ceph-facts fixes this issue.

This also adds some output that might be helpful.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1721 from tchaikov/wip-gcc-toolset-9

scripts/build_utils: do not add tchaikov/gcc-toolset-9 copr repo anymore

Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

scripts/build_utils: do not add tchaikov/gcc-toolset-9 copr repo anymore

since both CentOS and RHEL now include the updated
gcc-toolset-9-gcc-9.2.1-2.3.el8.

see also https://bugzilla.redhat.com/show_bug.cgi?id=1853900

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1720 from tchaikov/wip-get_bptag

*/build_deb: extract get_bptag() into build_utils.sh

Reviewed-by: David Galloway <dgallowa@redhat.com>

scripts/build_utils: do not handle wheezy anymore

wheezy is the only "baddist" which does not have
"libbabeltrace-ctf-dev". since we don't build wheezy anymore, let's stop
handling wheezy.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-*build: drop wheezy builds

per David Galloway

> we don't build for wheezy.

so drop it!

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1719 from dmick/master

quay-pruner.py: find orphaned SHA1-based tags and delete

scripts/build_utils: reuse get_bptag() in gen_debian_version()

less repeating this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

*/build_deb: extract get_bptag() into build_utils.sh

all of the related job definitions include build_utils.sh before
including build_deb, so it's safe to extract this function into
scripts/build_utils.sh.

it's safer and simpler to maintain this function in a single place.

Signed-off-by: Kefu Chai <kchai@redhat.com>

*/build_deb: extract gen_debian_version() into build_utils.sh

all of the related job definitions include build_utils.sh before
including build_deb, so it's safe to extract this function into
scripts/build_utils.sh.

it's safer and simpler to maintain this function in a single place.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: do not consider squeeze anymore

we don't build debian squeeze, a.k.a debian 6.0 anymore. squeeze was EOL
in February 2016. see https://www.debian.org/releases/squeeze/

Signed-off-by: Kefu Chai <kchai@redhat.com>

quay-pruner: look for orphaned SHA1 tags and delete

This required the ability to query shaman for SHA1 alone as
well as the existing ref/sha1/CentOS-minor-version.  Add some
refactoring to pull out the actual shaman request, and some
caching to minimize the actual number of requests.

Signed-off-by: Dan Mick <dmick@redhat.com>

quay-pruner: long lines, sort delete tags when printing

Signed-off-by: Dan Mick <dmick@redhat.com>

quay-pruner: add page limit constants for hacking/debugging

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1715 from tchaikov/wip-debian-profiles

ceph-*build: pass --profiles nocheck to pbuilder

Merge pull request #1718 from ceph/fix-brach-api

ceph-{api,dashboard}-pr: fix source branch ref

ceph-{api,dashboard}-pr: fix source branch ref

Applying same change as already applied by
https://github.com/ceph/ceph-build/pull/1426. A divergence has been
observed between `make check` job (ceph-pull-request) and the `make` in
ceph-pr-api job. For the same PR:

* ceph-pull-request `make check` passes
* ceph-pr-api `make` fails

This fix has been suggested by @sebastian-philipp

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1717 from ceph/wip-stopit-apt

ansible: Disable daily apt transaction on Ephemeral slaves

ansible: Disable daily apt transaction on Ephemeral slaves

Signed-off-by: David Galloway <dgallowa@redhat.com>

scripts/build_utils.sh: pass --profiles nocheck to pbuilder

so we can do profile-based build on debian derivative distros.

"nocheck" is one of the registered profile names. be specifying
"nocheck", we can reduce the build depends required by building Ceph
once the related Build-Depends in debian/control are marked with "nocheck".

see https://wiki.debian.org/BuildProfileSpec

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-*build: extract build_deb into scripts/build_utils.sh

so we can modify them in a single place

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-build: define chacra_url in 'setup_deb'

and use this variable in build_deb. this change pave the road to a
consolidated build_deb file which can be shared by all ceph*-build jobs

Signed-off-by: Kefu Chai <kchai@redhat.com>