ceph-dashboard-pull-requests: install python3 packages only

since we've moved to ubuntu focal for running "make check", "ceph-dashboard-pull-requests"
and "ceph-pr-api" tests. and ubuntu focal does not package python2 packages
for python-scipy or python-routes. let's just install the python3
packages.

this change is related to nautilus builds. because nautilus is the last
release which supports python2.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1783 from tchaikov/wip-api-test-python2

ceph-{api-nightly,pr-api}: install python3 packages on debian derivat…

Reviewed-by: Laura Paduano <lpaduano@suse.com>

ceph-{api-nightly,pr-api}: install python3 packages on debian derivatives

since we've moved to ubuntu focal for running "make check" and
"ceph-pr-api" tests. and ubuntu focal does not package python2 packages
for python-scipy or python-routes. let's just install the python3
packages.

this change is related to nautilus builds. because nautilus is the last
release which supports python2.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1778 from dmick/wip-tcmu-runner-arm64

tcmu-runner: add arm64

Merge pull request #1779 from tchaikov/wip-pr-ctest

ceph-pull-requests: use XUnit for publishing the test result of ctest

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

ceph-pr-commits: remove non-ascii char

otherwise `jenkins-jobs test` fails when reading it in non-utf-8 locale,

Traceback (most recent call last):
  File "/tmp/venv.KVnqGXyZ3a/bin/jenkins-jobs", line 8, in <module>
    sys.exit(main())
  File "/tmp/venv.KVnqGXyZ3a/lib/python3.6/site-packages/jenkins_jobs/cli/entry.py", line 165, in main
    jjb.execute()
  File "/tmp/venv.KVnqGXyZ3a/lib/python3.6/site-packages/jenkins_jobs/cli/entry.py", line 146, in execute
    ext.obj.execute(self.options, self.jjb_config)
  File "/tmp/venv.KVnqGXyZ3a/lib/python3.6/site-packages/jenkins_jobs/cli/subcommand/test.py", line 66, in execute
    config_xml=options.config_xml)
  File "/tmp/venv.KVnqGXyZ3a/lib/python3.6/site-packages/jenkins_jobs/builder.py", line 279, in update_jobs
    output.write(job.output())
  File "/tmp/venv.KVnqGXyZ3a/lib/python3.6/codecs.py", line 836, in write
    return self.writer.write(data)
  File "/tmp/venv.KVnqGXyZ3a/lib/python3.6/codecs.py", line 376, in write
    data, consumed = self.encode(object, self.errors)
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2014' in position 57823: ordinal not in range(256)

before this change:

$ iconv -f ascii -t ascii < ./ceph-pr-commits/build/build
...
input sequence at position 197

after this change,

$ iconv -f ascii -t ascii < ./ceph-pr-commits/build/build

finishes without issues.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-pull-requests: use XUnit for publishing the test result of ctest

for better developer experience

Signed-off-by: Kefu Chai <kchai@redhat.com>

tcmu-runner: add arm64

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1775 from ceph/wip-nomore10

build_utils: Start using pip 21.0.1 instead of 10.0.0

build_utils: Start using pip 20.3.4 instead of 10.0.0

Started seeing
```
+ /tmp/venv.ivc8u2qcCO/bin/pip install --no-binary=:all: --upgrade --exists-action=i --find-links=file:///home/jenkins-build/.cache/pip --no-index 'jenkins-job-builder>=3.5.0'
Looking in links: file:///home/jenkins-build/.cache/pip
Collecting jenkins-job-builder>=3.5.0
Collecting six>=1.9.0 (from jenkins-job-builder>=3.5.0)
Collecting PyYAML<6,>=3.10.0 (from jenkins-job-builder>=3.5.0)
  Could not find a version that satisfies the requirement wheel (from versions: )
No matching distribution found for wheel
```

Even 21.0.1 on Ubuntu 16.04 was failing.  It's just time to pin a newer version.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1774 from ceph/wip-preferpy3

build_utils.sh: Prefer python3 instead of python2 for virtualenvs

Merge pull request #1709 from ceph/wip-login-dockerhub

ceph*build: Login to docker.io if needed

Merge pull request #1761 from ceph/wip-spc

ansible: Install software-properties-common on deb builders

Merge pull request #1769 from ceph/wip-use-network

build_utils: Use network in pbuilder env

Merge pull request #1773 from ceph/wip-or-pacific

build_utils: Don't build pacific containers for el7

Merge pull request #1772 from ceph/wip-oh-noes

sync-push: Print releases in order

Merge pull request #1776 from ceph/wip-laterjjb

ceph-build-pull-requests: Use newer jjb

ceph-build-pull-requests: Use newer jjb

Signed-off-by: David Galloway <dgallowa@redhat.com>

build_utils.sh: Prefer python3 instead of python2 for virtualenvs

https://github.com/ceph/ceph-build/blob/master/scripts/build_utils.sh#L94-L105

Signed-off-by: David Galloway <dgallowa@redhat.com>

build_utils: Don't build pacific containers for el7

Signed-off-by: David Galloway <dgallowa@redhat.com>

sync-push: Print releases in order

Previous directory listing behavior:

```
octopus-15.2.10
octopus-15.2.4
octopus-15.2.5
octopus-15.2.6
octopus-15.2.7
octopus-15.2.8
octopus-15.2.9
```

Desired (and current with this PR) listing behavior:

```
octopus-15.2.4
octopus-15.2.5
octopus-15.2.6
octopus-15.2.7
octopus-15.2.8
octopus-15.2.9
octopus-15.2.10
```

This wasn't discovered until now because the `$newgen` stuff I do in this script only applies to Octopus for now.  Octopus only had single digit minor release versions (15.x.{1..9}) until 15.2.10.  The symlink didn't get updated to download.ceph.com/rpm-octopus still pointed to download.ceph.com/rpm-15.2.9 since it was last in the directory listing.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1767 from ceph/drop-3.2-5.0

ceph-ansible: drop stable-3.2 and stable-5.0 nightlies

Merge pull request #1771 from ceph/wip-pip3

Revert "build_utils.sh: Prefer python3 instead of python2 for virtual envs

Revert "build_utils.sh: Prefer python3 instead of python2 for virtualenvs"

This reverts commit 58afbde0aead09adc1f7af06e8491670e1acd614.

build_utils: Use network in pbuilder env

Not sure what happened between 0.229.1 vs 0.230.4 but seems `--use-network no` is now the default.  Discovered on a Focal builder.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1766 from ceph/wip-prefer-py3

build_utils.sh: Prefer python3 instead of python2 for virtualenvs

ceph-ansible: drop stable-3.2 and stable-5.0 nightlies

stable-3.2 no longer receives patches
stable-5.0 won't be productized

there so few CI resources allocated to ceph-ansible that we can't
affoard testing all of this.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1764 from tchaikov/wip-ceph-pull-request-kill-tests

ceph-pull-requests: kill orphan processes after job is aborted

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-pull-requests: kill orphan processes after job is aborted

Fixes: https://tracker.ceph.com/issues/47830
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1763 from ceph/fix_ganesha_builds

nfs-ganesha: fix wrong url for setting REPO_URL

nfs-ganesha: fix wrong url for setting REPO_URL

7173ede7ec472b555c137093cb30e272ce1868cd was an attempt to fix the
nfs-ganesha job but it's using the wrong url. This commit fixes it.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1762 from ceph/fix_ganesha_builds

nfs-ganesha: fix devel builds

nfs-ganesha: fix devel builds

since recent changes in shaman, the query for setting `REPO_URL` must be
updated.
This should fix the builds that are currently broken.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1688 from ceph/wip-force-vagrant

ansible: New playbook to force reinstall vagrant on CentOS8

Merge pull request #1760 from ceph/new-c-v-maintainer

swap ceph-volume maintainers

ansible: Install software-properties-common on deb builders

Provides `add-apt-repository`

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1759 from dmick/wip-arm

prune-quay.py: Provision for aarch64 containers, manifest lists, minor improvements

swap ceph-volume maintainers

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

prune-quay.py: don't try to examine manifest-list tags

They don't have image_ids, so don't consider them for "same as"
comparisons

Signed-off-by: Dan Mick <dmick@redhat.com>

prune-quay.py: reinstate verbose log about form of tag

Signed-off-by: Dan Mick <dmick@redhat.com>

prune-quay.py: parse and hold possible arch value

currently only for debugging ease; may be useful in future

Signed-off-by: Dan Mick <dmick@redhat.com>

prune-quay.py: handle aarch64 images

prune-quay.py: limit shaman search to ceph project

Some of the searches were broad enough to find hits in other projects

Signed-off-by: Dan Mick <dmick@redhat.com>

build_utils.sh: Prefer python3 instead of python2 for virtualenvs

https://github.com/ceph/ceph-build/blob/master/scripts/build_utils.sh#L94-L105

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1758 from ceph/wip-tmpreaper-notimeout

ansible: Disable timeout on tmp{watch,reaper}

ansible: Disable timeout on tmp{watch,reaper}

From the man page:

```
       -T x, --runtime=x
              Execution of tmpreaper will aborted after x seconds; this is to prevent attacks that create many, many files.  By default the timeout is set to 55 seconds.  A value of 0 will disable this feature, which is not advised as this feature prevents possible
              race-conditions between different instances of tmpreaper.
```

So tmpreaper wasn't running for the past week since I set this up because there was so much to clean up that it'd time out.  These are Jenkins builders where the primary user already has sudo access so there's no real use to locking down tmpreaper.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1757 from ceph/ceph_ansible_nightly_pacific_centos_release

ceph_ansible_nightly: add pacific centos release

ceph_ansible_nightly: add pacific centos release

This adds Pacific release based on CentOS 8 like Octopus.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1756 from ceph/wip-win32-build

ceph-dev-new-build: cleanup root-owned mingw64 pbuilder environment

Reviewed-by: David Galloway <dgallowa@redhat.com>

ceph-dev-new-build: cleanup root-owned mingw64 pbuilder environment

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1754 from ceph/wip-win32-build-trigger

ceph-dev-new-trigger: build windows for pacific and master

Merge pull request #1755 from petrutlucian94/win_pbuilder

ceph-dev-new-build: cross-compile Windows targets using pbuilder

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-dev-new-build: cross-compile Windows targets using pbuilder

The Windows builds that are generated right now are unusable due to
known Mingw runtime bugs.

We need Ubuntu Focal or later, for which reason we're going to use
pbuilder.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #1753 from ceph/wip-win32-build

ceph-dev-new-build: tweak chacra binary and repo endpoints for windows

ceph-dev-new-build: use dynamic chacra url for windows build

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-dev-new-trigger: build windows for pacific and master

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-dev-new-build: tweak chacra binary and repo endpoints for windows

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1752 from ceph/wip-win32-build

ceph-dev-new-build: extract artifacts prior to attempting windows build

ceph-dev-new-build: extract artifacts prior to attempting windows build

Also fixed the chacra endpoints to be consistent between the setup and
build steps.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1751 from ceph/wip-win32-build

ceph-dev-new-build: fixed windows build environment variables

ceph-dev-new-build: fixed windows build environment variables

The environment variables should be placed before the command.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1722 from petrutlucian94/win32

ceph-dev-new: add Windows build

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1750 from ceph/wip-notcmalloc

Disable notcmalloc builds!

ceph-dev-new: add Windows build

We're going to do a Windows build using Mingw as part of the
ceph-dev-new job.

"windows" is added as a valid distro. The "validate_mingw" script
will ensure that we're running on top of a Linux distro that the
Windows build script supports (currently just Ubuntu and Suse
distros).

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Disable notcmalloc builds!

We no longer need notmcalloc packages.  Valgrind works now.

See https://github.com/ceph/ceph/pull/39561

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1749 from ceph/wip-defaults

Update ceph ARCH defaults

Update ceph ARCH defaults

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1747 from ceph/wip-noarmcrimson

ceph-dev: Disable arm64 notcmalloc and crimson builds

ceph-dev: Disable arm64 notcmalloc and crimson builds

Nobody will use them and crimson is failing anyway

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1746 from ceph/wip-always

always set facts

ansible: Always set_facts

Without these set_fact tasks having the always tag, a few conditionals (e.g., permanent|bool) and vars don't get set.  So if we're running `ansible-playbook --tags tmp`, for example, all the tasks get skipped because `permanent|bool` didn't get set automatically.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1745 from ceph/wip-tmpcleaner

ansible: Install tmpreaper/tmpwatch and clean /tmp daily

ansible: Install tmpreaper/tmpwatch and clean /tmp daily

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1744 from ceph/wip-make-ken-happy-again

s/ceph-dev-nightly/ceph-dev-cron

s/ceph-dev-nightly/ceph-dev-cron

Fixes: https://github.com/ceph/ceph-build/pull/1743#issuecomment-780840190
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1743 from ceph/wip-cephnightly

Switch to building ceph.git branches twice daily

ceph-dev-trigger: Disable job for now

We no longer want to build packages for every merge in active branches.  ceph-dev-nightly will build each active branch twice daily instead.

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-dev-nightly: Create job

Build active branches twice daily if changes are found

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1741 from ceph/remove_ca_add_osds

ceph-ansible: remove add_osds jobs

ceph-ansible: remove add_osds jobs

The add-osds.yml playbook was dropped as of 5.0

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1740 from ceph/quick_patch_cv

ceph-volume-prs: use ceph-ansible@stable-6.0 for pacific

ceph-volume-prs: use ceph-ansible@stable-6.0 for pacific

Since, ceph-ansible stable-6.0 has been branched for deploying pacific,
we should use this branch in ceph-volume testing.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1739 from ceph/ca_6.0

ceph-ansible: branch stable-6.0

ceph-ansible: branch stable-6.0

This commit brings the required modification to branch stable-6.0

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1738 from ceph/wip-reenable-arm

ceph-dev*: Enable dev builds for arm64

ceph-dev*: Enable dev builds for arm64

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1737 from ceph/wip-docs-easier

Stop building binaries for docs-only PRs

Merge pull request #1734 from dmick/wip-quay-fix-ref-delete

Fix prune-quay.py's deletion of busy (well-known) refs

ceph-pr-api: Don't run for docs-only PRs

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-pull-requests: Skip make check on a docs-only PR

Signed-off-by: David Galloway <dgallowa@redhat.com>

ceph-pr-commits: Move 'docs only' check to build_utils for reuse

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1736 from ceph/wip-notcmalloc-boost-valgrind

build: notcmalloc builds enable WITH_BOOST_VALGRIND

Reviewed-by: Kefu Chai <kchai@redhat.com>

build: notcmalloc builds enable WITH_BOOST_VALGRIND

this flag silences a false-positive from valgrind in boost::context:
```
<error>
  <unique>0x0</unique>
  <tid>124</tid>
  <kind>InvalidRead</kind>
  <what>Invalid read of size 8</what>
  <stack>
    <frame>
      <ip>0x52FF7E7</ip>
      <obj>/usr/lib64/libradosgw.so.2.0.0</obj>
      <fn>spawn::detail::continuation_context::resume()</fn>
    </frame>
  </stack>
  <auxwhat>Address 0x57f65728 is in a rw- anonymous segment</auxwhat>
</error>
```

Fixes: https://tracker.ceph.com/issues/48963
Signed-off-by: Casey Bodley <cbodley@redhat.com>

prune-quay.py: fix deletion of busy ref names (master etc.)

We must only delete the ref name if it refers to the same image
as the ref-shortsha1-el form of the name; if it's been pushed as a
new build of ref from a new sha1, it's no longer the image we
need to delete.  Verify the ref tag points to the same image as the
ref-shortsha1-el tag.

Signed-off-by: Dan Mick <dmick@redhat.com>

prune-quay.py: improve verbose logging

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1735 from dmick/wip-tcmu-runner-arm64

tcmu-runner: fix for arm64/aarch64

tcmu-runner: fix for arm64/aarch64

Add "available arch" arm64, and fix build_rpm script to use aarch64
Also limit aarch64 build to CentOS 8

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1733 from dmick/master

scripts/build_utils.sh: handle lowercase CentOS dnf repo names

Reviewed-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: handle lowercase CentOS dnf repo names

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1732 from ceph/wip-disable-leap15-for-master

disable leap15 for master and pacific

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>