nfs-ganesha: remove centos7 and xenial, build arm64 (centos8 only)

Signed-off-by: Dan Mick <dmick@redhat.com>

nfs-ganesha: fix patch for debian/rules

Some branches of nfs-ganesha-debian don't have the right format for
the patch file to work (Cmake features can be turned on or off
with various words, ON/OFF, YES/NO, etc.).  Use sed instead to be
more resilient to such chicanery.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1797 from dmick/wip-build-utils

Wrong variable use / shell syntax errors in build_utils.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

build_utils.sh: delete unescaped newlines in sources.list addition

Signed-off-by: Dan Mick <dmick@redhat.com>

build_utils.sh: check correct version of variable for distro

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1796 from ceph/wip-unattende

ansible: Remove unattended-upgrades after package install

ansible: Remove unattended-upgrades after package install

One of the packages we *do* want to install (software-properties-common maybe) pulls in unattended upgrades.  It's not a strict dependency so we can remove it after package install.  It's the only package that gets removed.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1795 from tchaikov/wip-hookdir

scripts/build_utils.sh: set hookdir using pbuilderrc

Reviewed-by: David Galloway <dgallowa@redhat.com>

scripts/build_utils.sh: set hookdir using pbuilderrc

per our testing, --hookdir does not work even the pbuilderrc(5) puts

> HOOKDIR=/usr/lib/pbuilder/hooks
>    Specifies the default location for the user hooks directory.  This is overridden with the option --hookdir.

so let use HOOKDIR as we did before e81fd7baa0e49242cef2db66eeb3fe6dc4269905

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: no need to update repo if ppa repo is not used

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: make user script executable

as per pbuilder(8),

> The  scripts  must be executable and may be either binaries or
> interpreted scripts.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1792 from tchaikov/wip-updates

scripts/build_utils.sh: install gnupg before ppa packages

scripts/build_utils: use -security and -updates repo

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1790 from tchaikov/wip-no-quotes-thanks

scripts/build_utils.sh: should not quote --mirror option

scripts/build_utils.sh: should not quote --mirror option

otherwise the URL is not reachable with quotes in it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1781 from dmick/wip-nfs-ganesha-arm64

Fix up nfs-ganesha for arm64/centos8 builds

Merge pull request #1788 from tchaikov/wip-xunit-thresholds

ceph-pull-requests: specify threshold for a failure

Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1789 from ceph/wip-allow-focal

multiple PR jobs: Run on any Ubuntu node

multiple PR jobs: Run on any Ubuntu node

All of Jenkins builders are running Ubuntu Focal.  They all have the bionic label.  They also have the xenial label but that's only because we're still building packages *for* Xenial.  But not *on* Xenial.

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-pull-requests: specify threshold for a failure

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #1785 from tchaikov/wip-pbuilder

scripts/build_utils: write settings to /root/.pbuilderrc

Reviewed-by: David Galloway <dgallowa@redhat.com>

scripts/build_utils: write settings to /root/.pbuilderrc

according to pbuilder(8)

> ${HOME}/.pbuilderrc
>       The personal configuration file for pbuilder, which overrides settings set in other configuration files.
>
>       Note that ${HOME} is usually /root (if you are running pbuilder through sudo).

since we always call `pbuilder` via `sudo`, we should overwrite
/root/.pbuilderrc with $HOME/.pbuilderrc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: use "pbuilder clean"

"clean" is now a command instead of an option.

Signed-off-by: Kefu Chai <kchai@redhat.com>

script/build_utils.sh: use a single variable for pbuilder options

for better maintainability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1780 from dmick/wip-xunit-skip

ceph-pull-requests: skip xunit publisher if no test files present

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1782 from tchaikov/wip-gcc-9-bionic

scripts/build_utils.sh: use gcc-9 from ppa on bionic

Merge pull request #1784 from tchaikov/wip-dashboard-py3

ceph-dashboard-pull-requests: install python3 packages only

Reviewed-by: Laura Paduano <lpaduano@suse.com>

ceph-dashboard-pull-requests: install python3 packages only

since we've moved to ubuntu focal for running "make check", "ceph-dashboard-pull-requests"
and "ceph-pr-api" tests. and ubuntu focal does not package python2 packages
for python-scipy or python-routes. let's just install the python3
packages.

this change is related to nautilus builds. because nautilus is the last
release which supports python2.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1783 from tchaikov/wip-api-test-python2

ceph-{api-nightly,pr-api}: install python3 packages on debian derivat…

Reviewed-by: Laura Paduano <lpaduano@suse.com>

ceph-{api-nightly,pr-api}: install python3 packages on debian derivatives

since we've moved to ubuntu focal for running "make check" and
"ceph-pr-api" tests. and ubuntu focal does not package python2 packages
for python-scipy or python-routes. let's just install the python3
packages.

this change is related to nautilus builds. because nautilus is the last
release which supports python2.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1778 from dmick/wip-tcmu-runner-arm64

tcmu-runner: add arm64

scripts/build_utils.sh: use gcc-9 from ppa on bionic

for using more C++17 compliant compiler

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

nfs-ganesha: remove unused variable

Signed-off-by: Dan Mick <dmick@redhat.com>

nfs-ganesha: mock arch is aarch64, not arm64

Signed-off-by: Dan Mick <dmick@redhat.com>

nfs-ganesha: enable centos8 for arm64, handle lowercase repo names

Signed-off-by: Dan Mick <dmick@redhat.com>

ceph-pull-requests: skip xunit publisher if no test files present

otherwise make check, at least, fails

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1779 from tchaikov/wip-pr-ctest

ceph-pull-requests: use XUnit for publishing the test result of ctest

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

ceph-pr-commits: remove non-ascii char

otherwise `jenkins-jobs test` fails when reading it in non-utf-8 locale,

Traceback (most recent call last):
  File "/tmp/venv.KVnqGXyZ3a/bin/jenkins-jobs", line 8, in <module>
    sys.exit(main())
  File "/tmp/venv.KVnqGXyZ3a/lib/python3.6/site-packages/jenkins_jobs/cli/entry.py", line 165, in main
    jjb.execute()
  File "/tmp/venv.KVnqGXyZ3a/lib/python3.6/site-packages/jenkins_jobs/cli/entry.py", line 146, in execute
    ext.obj.execute(self.options, self.jjb_config)
  File "/tmp/venv.KVnqGXyZ3a/lib/python3.6/site-packages/jenkins_jobs/cli/subcommand/test.py", line 66, in execute
    config_xml=options.config_xml)
  File "/tmp/venv.KVnqGXyZ3a/lib/python3.6/site-packages/jenkins_jobs/builder.py", line 279, in update_jobs
    output.write(job.output())
  File "/tmp/venv.KVnqGXyZ3a/lib/python3.6/codecs.py", line 836, in write
    return self.writer.write(data)
  File "/tmp/venv.KVnqGXyZ3a/lib/python3.6/codecs.py", line 376, in write
    data, consumed = self.encode(object, self.errors)
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2014' in position 57823: ordinal not in range(256)

before this change:

$ iconv -f ascii -t ascii < ./ceph-pr-commits/build/build
...
input sequence at position 197

after this change,

$ iconv -f ascii -t ascii < ./ceph-pr-commits/build/build

finishes without issues.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-pull-requests: use XUnit for publishing the test result of ctest

for better developer experience

Signed-off-by: Kefu Chai <kchai@redhat.com>

tcmu-runner: add arm64

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1775 from ceph/wip-nomore10

build_utils: Start using pip 21.0.1 instead of 10.0.0

build_utils: Start using pip 20.3.4 instead of 10.0.0

Started seeing
```
+ /tmp/venv.ivc8u2qcCO/bin/pip install --no-binary=:all: --upgrade --exists-action=i --find-links=file:///home/jenkins-build/.cache/pip --no-index 'jenkins-job-builder>=3.5.0'
Looking in links: file:///home/jenkins-build/.cache/pip
Collecting jenkins-job-builder>=3.5.0
Collecting six>=1.9.0 (from jenkins-job-builder>=3.5.0)
Collecting PyYAML<6,>=3.10.0 (from jenkins-job-builder>=3.5.0)
  Could not find a version that satisfies the requirement wheel (from versions: )
No matching distribution found for wheel
```

Even 21.0.1 on Ubuntu 16.04 was failing.  It's just time to pin a newer version.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1774 from ceph/wip-preferpy3

build_utils.sh: Prefer python3 instead of python2 for virtualenvs

Merge pull request #1709 from ceph/wip-login-dockerhub

ceph*build: Login to docker.io if needed

Merge pull request #1761 from ceph/wip-spc

ansible: Install software-properties-common on deb builders

Merge pull request #1769 from ceph/wip-use-network

build_utils: Use network in pbuilder env

Merge pull request #1773 from ceph/wip-or-pacific

build_utils: Don't build pacific containers for el7

Merge pull request #1772 from ceph/wip-oh-noes

sync-push: Print releases in order

Merge pull request #1776 from ceph/wip-laterjjb

ceph-build-pull-requests: Use newer jjb

ceph-build-pull-requests: Use newer jjb

Signed-off-by: David Galloway <dgallowa@redhat.com>

build_utils.sh: Prefer python3 instead of python2 for virtualenvs

https://github.com/ceph/ceph-build/blob/master/scripts/build_utils.sh#L94-L105

Signed-off-by: David Galloway <dgallowa@redhat.com>

build_utils: Don't build pacific containers for el7

Signed-off-by: David Galloway <dgallowa@redhat.com>

sync-push: Print releases in order

Previous directory listing behavior:

```
octopus-15.2.10
octopus-15.2.4
octopus-15.2.5
octopus-15.2.6
octopus-15.2.7
octopus-15.2.8
octopus-15.2.9
```

Desired (and current with this PR) listing behavior:

```
octopus-15.2.4
octopus-15.2.5
octopus-15.2.6
octopus-15.2.7
octopus-15.2.8
octopus-15.2.9
octopus-15.2.10
```

This wasn't discovered until now because the `$newgen` stuff I do in this script only applies to Octopus for now.  Octopus only had single digit minor release versions (15.x.{1..9}) until 15.2.10.  The symlink didn't get updated to download.ceph.com/rpm-octopus still pointed to download.ceph.com/rpm-15.2.9 since it was last in the directory listing.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1767 from ceph/drop-3.2-5.0

ceph-ansible: drop stable-3.2 and stable-5.0 nightlies

Merge pull request #1771 from ceph/wip-pip3

Revert "build_utils.sh: Prefer python3 instead of python2 for virtual envs

Revert "build_utils.sh: Prefer python3 instead of python2 for virtualenvs"

This reverts commit 58afbde0aead09adc1f7af06e8491670e1acd614.

build_utils: Use network in pbuilder env

Not sure what happened between 0.229.1 vs 0.230.4 but seems `--use-network no` is now the default.  Discovered on a Focal builder.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1766 from ceph/wip-prefer-py3

build_utils.sh: Prefer python3 instead of python2 for virtualenvs

ceph-ansible: drop stable-3.2 and stable-5.0 nightlies

stable-3.2 no longer receives patches
stable-5.0 won't be productized

there so few CI resources allocated to ceph-ansible that we can't
affoard testing all of this.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1764 from tchaikov/wip-ceph-pull-request-kill-tests

ceph-pull-requests: kill orphan processes after job is aborted

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-pull-requests: kill orphan processes after job is aborted

Fixes: https://tracker.ceph.com/issues/47830
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1763 from ceph/fix_ganesha_builds

nfs-ganesha: fix wrong url for setting REPO_URL

nfs-ganesha: fix wrong url for setting REPO_URL

7173ede7ec472b555c137093cb30e272ce1868cd was an attempt to fix the
nfs-ganesha job but it's using the wrong url. This commit fixes it.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1762 from ceph/fix_ganesha_builds

nfs-ganesha: fix devel builds

nfs-ganesha: fix devel builds

since recent changes in shaman, the query for setting `REPO_URL` must be
updated.
This should fix the builds that are currently broken.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1688 from ceph/wip-force-vagrant

ansible: New playbook to force reinstall vagrant on CentOS8

Merge pull request #1760 from ceph/new-c-v-maintainer

swap ceph-volume maintainers

ansible: Install software-properties-common on deb builders

Provides `add-apt-repository`

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1759 from dmick/wip-arm

prune-quay.py: Provision for aarch64 containers, manifest lists, minor improvements

swap ceph-volume maintainers

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

prune-quay.py: don't try to examine manifest-list tags

They don't have image_ids, so don't consider them for "same as"
comparisons

Signed-off-by: Dan Mick <dmick@redhat.com>

prune-quay.py: reinstate verbose log about form of tag

Signed-off-by: Dan Mick <dmick@redhat.com>

prune-quay.py: parse and hold possible arch value

currently only for debugging ease; may be useful in future

Signed-off-by: Dan Mick <dmick@redhat.com>

prune-quay.py: handle aarch64 images

prune-quay.py: limit shaman search to ceph project

Some of the searches were broad enough to find hits in other projects

Signed-off-by: Dan Mick <dmick@redhat.com>

build_utils.sh: Prefer python3 instead of python2 for virtualenvs

https://github.com/ceph/ceph-build/blob/master/scripts/build_utils.sh#L94-L105

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1758 from ceph/wip-tmpreaper-notimeout

ansible: Disable timeout on tmp{watch,reaper}

ansible: Disable timeout on tmp{watch,reaper}

From the man page:

```
       -T x, --runtime=x
              Execution of tmpreaper will aborted after x seconds; this is to prevent attacks that create many, many files.  By default the timeout is set to 55 seconds.  A value of 0 will disable this feature, which is not advised as this feature prevents possible
              race-conditions between different instances of tmpreaper.
```

So tmpreaper wasn't running for the past week since I set this up because there was so much to clean up that it'd time out.  These are Jenkins builders where the primary user already has sudo access so there's no real use to locking down tmpreaper.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1757 from ceph/ceph_ansible_nightly_pacific_centos_release

ceph_ansible_nightly: add pacific centos release

ceph_ansible_nightly: add pacific centos release

This adds Pacific release based on CentOS 8 like Octopus.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1756 from ceph/wip-win32-build

ceph-dev-new-build: cleanup root-owned mingw64 pbuilder environment

Reviewed-by: David Galloway <dgallowa@redhat.com>

ceph-dev-new-build: cleanup root-owned mingw64 pbuilder environment

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1754 from ceph/wip-win32-build-trigger

ceph-dev-new-trigger: build windows for pacific and master

Merge pull request #1755 from petrutlucian94/win_pbuilder

ceph-dev-new-build: cross-compile Windows targets using pbuilder

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-dev-new-build: cross-compile Windows targets using pbuilder

The Windows builds that are generated right now are unusable due to
known Mingw runtime bugs.

We need Ubuntu Focal or later, for which reason we're going to use
pbuilder.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #1753 from ceph/wip-win32-build

ceph-dev-new-build: tweak chacra binary and repo endpoints for windows

ceph-dev-new-build: use dynamic chacra url for windows build

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-dev-new-trigger: build windows for pacific and master

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-dev-new-build: tweak chacra binary and repo endpoints for windows

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1752 from ceph/wip-win32-build

ceph-dev-new-build: extract artifacts prior to attempting windows build

ceph-dev-new-build: extract artifacts prior to attempting windows build

Also fixed the chacra endpoints to be consistent between the setup and
build steps.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1751 from ceph/wip-win32-build

ceph-dev-new-build: fixed windows build environment variables

ceph-dev-new-build: fixed windows build environment variables

The environment variables should be placed before the command.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1722 from petrutlucian94/win32

ceph-dev-new: add Windows build

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1750 from ceph/wip-notcmalloc

Disable notcmalloc builds!

ceph-dev-new: add Windows build

We're going to do a Windows build using Mingw as part of the
ceph-dev-new job.

"windows" is added as a valid distro. The "validate_mingw" script
will ensure that we're running on top of a Linux distro that the
Windows build script supports (currently just Ubuntu and Suse
distros).

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Disable notcmalloc builds!

We no longer need notmcalloc packages.  Valgrind works now.

See https://github.com/ceph/ceph/pull/39561

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1749 from ceph/wip-defaults

Update ceph ARCH defaults

Update ceph ARCH defaults

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1747 from ceph/wip-noarmcrimson

ceph-dev: Disable arm64 notcmalloc and crimson builds