ceph-ansible: add new scenario 'add_osds'

Adding new scenarios for "day 2 operation" playbook

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1152 from callithea/wip-ceph-dashboard-prs-python3-pkgs

ceph-dashboard-pull-requests: install python3 packages

Reviewed-by: Jos Collin <jcollin@redhat.com>

ceph-dashboard-pull-requests: install python3 packages

Signed-off-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #1161 from ceph/pipeline-sod

ceph-ansible-pipeline: fix wrong condition

ceph-ansible-pipeline: fix wrong condition

if grep report 0 ceph-osd was modified thus we exit

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1160 from ceph/pipeline

ceph-ansible-pipeline osd followup

ceph-ansible-pipeline osd followup

Not sure what's going on, this is mostly a commit to debug

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1159 from ceph/ansible-scenario

ceph-ansible-ci: do not run osd twice

ceph-ansible-ci: do not run osd twice

Do not play twice osds if they have been ack by the first task of the
pipeline.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1143 from ceph/ceph-container-build-ceph-base-push-imgs-arm64

add ceph-container-build-ceph-base-push-imgs-arm64

add ceph-container-build-ceph-base-push-imgs-arm64

Same as ceph-container-build-ceph-base-push-imgs but to build arm64
images.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1156 from ceph/guits-fix_collect_log

ceph-ansible: fix collect ceph config task

ceph-ansible: fix collect ceph config task

`ceph_conf_key_directory` doesn't exist in this playbook.
All our testing is done with '/etc/ceph', let's hardcode it here.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1155 from ceph/guits-syntax-check-ansible

ceph-ansible-pr: fix workspace in syntax check

ceph-ansible-pr: fix workspace in syntax check

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1154 from ceph/guits-syntax-check-ansible

ceph-ansible-syntax: fix path to pip

Merge branch 'master' into guits-syntax-check-ansible

ceph-ansible-syntax: fix path to pip

`$VENV` contains already '/bin/' pattern.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1153 from ceph/guits-syntax-check-ansible

ceph-ansible: install expected ansible version

ceph-ansible: install expected ansible version

install the ansible version corresponding to its respective ceph-ansible
ansible.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1151 from ceph/ci-master-typo

ceph-ansible fix job typo in pipeline

ceph-ansible fix job typo in pipeline

The job name is ceph-ansible-prs-luminous-lvm_osds_container

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1150 from ceph/ci-master

ceph-ansible pipeline branch improvement

ceph-ansible pipeline branch improvement

The code is not ideal and might be hard to read but the rationales are
the following:

* master ceph-ansible has fully adopted ceph-volume and we are about to
remove ceph-disk too
* this means we can not and should not run ceph-disk tests on master BUT
only on stable 3.0 and 3.1 branches.

Only lvm scenarios should play on ceph-ansible master.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1149 from tchaikov/wip-nautilus

ceph-dev-trigger: include nautilus branch

ceph-dev-trigger: include nautilus branch

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1148 from ceph/ceph-vol-cont

cpeh-ansible-prs: add lvm_batch scenario

cpeh-ansible-prs: add lvm_batch scenario

Add a new scenario

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1147 from ceph/guits-get_ceph_conf

ceph-ansible: fetch ceph config file on failure

ceph-ansible: fetch ceph config file on failure

Let's try to get ceph configuration file on failure, it can be useful at
some point.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1146 from ceph/ceph-volume-container

ceph-ansible: add more scenarios

ceph-ansible: add more scenarios

Add more scenarios for ceph-volume in container

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1145 from callithea/wip-ceph-dashboard-pull-requests-fix-script-exec

ceph-dashboard-pull-requests: Need to cd to the dashboard dir

ceph-dashboard-pull-requests: Need to cd to the dashboard dir before executing the e2e script

Signed-off-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #1115 from ceph/wip-sepia-future

sepia-fog-images: Don't hardcode distros

sepia-fog-images: Don't hardcode distros

This change ensures I don't have to add or remove if statements in
funSetProfiles function any time there's a new distro version released.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1142 from ceph/container-img

ceph-container-build-ceph-base-push-imgs: build with arg

ceph-container-build-ceph-base-push-imgs: build with arg

We need to pass the ARCH so the build can successfully run.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1140 from ceph/add-ceph-base

add new ceph-container-build-ceph-base-push-imgs job

add new ceph-container-build-ceph-base-push-imgs job

This job runs on a daily basis and is responsible for building ceph/base
versionned images.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1139 from tchaikov/dashboard-pr-sudo

ceph-dashboard-pull-requests: fix permission denied error

Reviewed-by: Laura Paduano <lpaduano@suse.com>

ceph-dashboard-pull-requests: fix permission denied error

as stdout redirect is performed by the shell, and shell is launched by
the "jenkins" user, so permission denied is expected when we are trying
to create a file under /etc/yum.repos.d/. in that case, we'll have
following failure:

/tmp/jenkins8899238112048417478.sh: line 14:
/etc/yum.repos.d/google-chrome.repo: Permission denied
Build step 'Execute shell' marked build as failure

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1138 from ceph/guits-fix_ca_pr_syntax

ceph-ansible: don't check for capital letter in meta directories

ceph-ansible: don't check for capital letter in meta directories

it's expected to have capital letters in roles/*/meta/main.yml so let's
exclude this path in git_diff_to_head function.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1137 from tchaikov/wip-lang-none

scripts/build_utils.sh: add lang=none option to sources.list

Reviewed-by: Laura Paduano <lpaduano@suse.com>

scripts/build_utils.sh: add lang=none option to sources.list

some mirrors do not contains the package description translations. when
apt tries to download the packages descriptions, it fails like:

E: Failed to fetch
http://mirror.yandex.ru/mirrors/launchpad/ubuntu-toolchain-r/dists/xenial/main/i18n/Translation-en
404  Not Found [IP: 213.180.204.183 80]
E: Some index files failed to download. They have been ignored, or old
ones used instead.

for more info, see sources.list(5) and apt.conf(5). the default value
of "lang" is set by Acquire::Languages, which is in turn "en" and
"environment" by default. we need to override them with "none".

Fixes #1136
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1135 from callithea/wip-ceph-dashboard-pull-requests-https

ceph-dashboard-pull-requests: switch to https for Chrome URLs

ceph-dashboard-pull-requests: switch to https for Chrome URLs

Signed-off-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #1134 from callithea/wip-ceph-dashboard-pull-request-fix

ceph-dashboard-pull-requests: fix another permission denied issue

ceph-dashboard-pull-requests: fix another permission denied issue
when using ``apt-key add``

Signed-off-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #1133 from callithea/wip-ceph-dashboard-pull-request-fix

ceph-dashboard-pull-requests: fix permission denied error

ceph-dashboard-pull-requests: fix permission denied error

Signed-off-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #1132 from ceph/cv-nightly-additions

ceph-volume-nightly: add more branches to batch tests

ceph-volume-nightly: add more branches to batch tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1086 from callithea/wip-ceph-dashboard-pull-requests

ceph-dashboard-pull-requests: Jenkins job for ceph dashboard frontend (e2e) tests

ceph-dashboard-pull-requests: added inital Jenkins job for dashboard frontend (e2e) tests

Signed-off-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #1131 from ceph/wip-debug-mutex

ceph-dev-[new-]setup/build/build: -DWITH_CEPH_DEBUG_MUTEX=ON for notcmalloc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge branch 'master' into wip-debug-mutex

Merge pull request #1123 from tchaikov/wip-decrease-n-jobs-to-fix-oom

scripts/build_utils.sh: increase mem per job to 2.2GB

scripts/build_utils.sh: increase mem per job to 2.2GB

it's observed that on our arm64 build host, using the existing
$total_mem/1600, n_build_jobs=37 . but we still have following error
when compiling Ceph:

virtual memory exhausted: Cannot allocate memory
make[3]: *** [src/mds/CMakeFiles/mds.dir/SnapClient.cc.o] Error 1

so increase the denominator to 2200 .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1130 from ceph/fix-signed-off-check

ceph-pr-commits: Fix check with many-commit PRs (3rd try)

ceph-dev-[new-]setup/build/build: -DWITH_CEPH_DEBUG_MUTEX=ON for notcmalloc

Enable the debug ceph::mutex in the notcmalloc build.  The normal build
will use the defaults, which (if it's a Release or RelWithDebug build)
will not include extra debug checks.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-pr-commits: Fix check with many-commit PRs

Relevant changes:
- All non-compliant commits are reported (previously, only the first
    wrong one was reported, stopping further processing).
- Signed-off-by regex put in place to check for
    <something@somewhere.tld> (including lt and gt symbols). This rule
    could be relaxed, as some commits in the past were sent without
    the enclosing <...>
- `extract_sha` function has been removed

Fixes: http://tracker.ceph.com/issues/36082
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1129 from ceph/revert-1128-fix-signed-off-check

Revert "ceph-pr-commits: Fix check with many-commit PRs (2nd try)"

Revert "ceph-pr-commits: Fix check with many-commit PRs (2nd try)"

Merge pull request #1128 from ceph/fix-signed-off-check

ceph-pr-commits: Fix check with many-commit PRs (2nd try)

ceph-pr-commits: Fix check with many-commit PRs

Relevant changes:
- All non-compliant commits are reported (previously, only the first
    wrong one was reported, stopping further processing).
- Signed-off-by regex put in place to check for
    <something@somewhere.tld> (including lt and gt symbols). This rule
    could be relaxed, as some commits in the past were sent without
    the enclosing <...>
- `extract_sha` function has been removed

Fixes: http://tracker.ceph.com/issues/36082
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1127 from ceph/revert-1125-fix-signed-off-check

Revert "ceph-pr-commits: Fix check with many-commit PRs"

Revert "ceph-pr-commits: Fix check with many-commit PRs"

Merge pull request #1125 from ceph/fix-signed-off-check

ceph-pr-commits: Fix check with many-commit PRs

ceph-pr-commits: Fix check with many-commit PRs

Relevant changes:
- All non-compliant commits are reported (previously, only the first
    wrong one was reported, stopping further processing).
- Signed-off-by regex put in place to check for
    <something@somewhere.tld> (including lt and gt symbols). This rule
    could be relaxed, as some commits in the past were sent without
    the enclosing <...>
- `extract_sha` function has been removed

Fixes: http://tracker.ceph.com/issues/36082
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1126 from ceph/wip-cap-retention

ceph-ansible-pipeline: Match days-to-keep with ceph-ansible-prs

ceph-ansible-pipeline: Match days-to-keep with ceph-ansible-prs

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1124 from ceph/cv-target-branch

ceph-volume-pr: add the TargetBranch param for manually triggering it

ceph-volume-pr: add the TargetBranch param for manually triggering it

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1122 from ceph/wip-container-ubuntu

ceph-container: Don't test Ubuntu

ceph-container: Don't test Ubuntu

No longer tested on Ubuntu: https://github.com/ceph/ceph-container/pull/1167

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1121 from ceph/log-retention

Change build discard options for ceph and ceph-volume jobs

Change build discard options for ceph and ceph-volume jobs

So that it isn't limited by the number, it is limited by days only. This
will allow builds to avoid 404'ing when the limit is reached, which is
unexpected if there are too many builds and not long enough time between
them (like 100 builds in one day, but limit is 20 before discarding)

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1118 from ceph/ceph-volume-tests

ceph-volume-nightly: add tests for batch with mixed device types

ceph-volume-nightly: add tests for batch with mixed device types

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #1119 from ceph/disable-ceph-medic

ceph-medic-tests: disable nightly tests

ceph-medic-tests: disable nightly tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1117 from ceph/ceph-volume-tests

ceph-volume-ansible-pr: include the batch mixed type tests

ceph-volume-ansible-pr: include the batch mixed type tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #1116 from tchaikov/wip-build-with-total-memory

ceph-*-build: evaluate num jobs using total mem not free mem

ceph-*-build: evaluate num jobs using total mem not free mem

there is chance that the system is under extremely high load when `vmstat`
is called. so the "free memory" can not reflect the available memroy
when building Ceph. so use "total memory" instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1114 from ceph/wip-centos-pip

ansible: Install python-pip from epel on centos slaves

ansible: Install python-pip from epel on centos slaves

I'm not sure why we're just now seeing this.  Perhaps it was manually
fixed on prado.ceph.com and a redeploy overwrote it?  Regardless
non-libvirt slaves are failing to join Jenkins currently.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1113 from ceph/group_vars_all

ceph-ansible-pr-syntax-check: group_vars operator

ceph-ansible-pr-syntax-check: group_vars operator

When defaults/main.yml is touched we will generate 2 group_vars files,
one for all.yml.sample and one for rhcs.yml.sample, this is making the
conditional never pass since there is one more value in the count.

Basically if defaults/main.yml and osd/defaults/main.yml are touched
this will result in 3 group_vars file but the count for
defaults/main.yml is 2.

In the end we want to fail if the number of defaults/main.yml is
greather than the number of group_vars files.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1112 from ceph/group_var

ceph-ansible-pr-syntax-check: fix typo

ceph-ansible-pr-syntax-check: fix typo

The directory we are looking for is called group_vars not group_var

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1105 from ceph/wip-iscsi-release

Add job for ceph-iscsi-* stable releases

Add job for ceph-iscsi-* stable releases

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1111 from ceph/ceph-volume-tests

ceph-volume-ansible-prs: enable tests on centos 7

ceph-volume-ansible-prs: enable tests on centos 7

Running the tests on Centos 7.4 hosts allows the vagrant vms
to boot properly.

See:
https://github.com/ceph/mita/pull/126/commits/35d53b13e177831306b0b401b0b5d0fafecf4b78

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #1103 from tchaikov/wip-less-make-j

ceph-pull-requests-arm64: set jobs number according to free memory

Merge pull request #1110 from ceph/wip-target-branch

ceph-volume-ansible-prs: require ghprbTargetBranch when manually triggering

ceph-volume-ansible-prs: require ghprbTargetBranch when manually triggering

Signed-off-by: Alfredo Deza <adeza@redhat.com>