Merge pull request #1549 from ceph/container-ipv6-bkp

backport: Container ipv6 bkp

Merge pull request #1550 from ceph/luminous-bkp

backport: ceph-common: add luminous to ceph_release_num

Merge pull request #1540 from ceph/piwi3910-master-bkp

backport: added support for Sandisk FusionIO devices

ceph-common: add luminous to ceph_release_num

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 471cdb0c3999b375f2891472d4645eea5f4d7051)
Signed-off-by: Sébastien Han <seb@redhat.com>

mon: do not hardcode ipv4

Problem: fail to deploy a containerized Ceph cluster with ipv6
Solution: do not hardcode ipv4 when bootstrapping the container.

Now use ip_version: ipv6 to get a containerized cluster deployed with
ipv6.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1451786
Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit c7aae7f965758eb08a806fc7be75d8075884b733)
Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1542 from ceph/monitor_interface-bkp

backport: monitor_interface: document need to use monitor_address when using IPv6

Merge pull request #1538 from ceph/quick_fix-bkp

backport: ignore error when doesn't exist

added support for Sandisk FusionIO devices

Signed-off-by: Pascal Watteel <pascal.watteel@emc.com>
(cherry picked from commit e4ef8bb87f9df22459908f71d8cc816b081d829a)
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1541 from ceph/control-path-bkp

backport: Control path bkp

Merge branch 'stable-2.2' into control-path-bkp

Merge pull request #1537 from ceph/ansible-ssh-control-path-bkp

backport: ansible: shorten default control_path

Merge pull request #1539 from ceph/docker-registry-bkp

backport: Docker registry bkp

Merge pull request #1521 from ceph/bpk-1512

backport: ansible: shorten default control_path

monitor_interface: document need to use monitor_address when using IPv6

Already documented in the Red Hat Ceph Storage 2 Installation Guide
for Red Hat Enterprise Linux, but not here

Signed-off-by: Florian Klink <flokli@flokli.de>
(cherry picked from commit 10b91661ceef7992354032030c7c2673a90d40f4)
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ansible: add a note in ansible.cfg

Comments inside this file must be set BEFORE the option. NOT after the
option, otherwise the comment will be interpreted as a value to that
option.

Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit ece9c14a3390321664cbb59ec9ea3f7984e71dba)
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ansible: fix control path line

If the comment is put after the line then it is interpreted so we need
to move it before and have a dedicated line.

Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit 9f2c21972debca9e0dc5bcc19bc5fce0f5c4c471)
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

group_vars: resync all.yml.sample with ceph-docker-common

Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit a80f07704cfe44bf1b271e76eb2101161c6814ec)
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

generate_group_vars_sample: ability to merge roles

Problem: the meta declaration just includes the role, it does nothing
with the group_vars. For ansible to use files defined in group_vars/ the
name of the file must match a host group. Like mons, osds, etc. There is
no group docker-common so the variables defined there are never used, as
proved by https://bugzilla.redhat.com/show_bug.cgi?id=1447179 and the
ansible documentation.

Solution: bring the ability to merge roles files. So now by default,
ceph-docker-common and ceph-common will go into all.yml.sample

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1447179
Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit a8c75c3bc9bd7211483f54cd98a99c95f69e7958)
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

generate_group_vars_sample: do not use expr

As mentionned in https://github.com/koalaman/shellcheck/wiki/SC2003
expr is antiquated.

Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit 53be0980be65170bb4f2acb3c87b0c1eff8c7acb)
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

generate_group_vars_sample.sh: fix script indent

Using indent of 2 on the entire script.

Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit 7fddcad8eae4632ff8e6007acd5d6b4f44416453)
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

rgw: resync group_vars

At some point, during a commit changing `roles/ceph-rgw/defaults/main.yml`
we forgot to run `generate_group_vars_sample.sh`.

Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit 21067453438ae5f87a25dbd00a9279ed9488e4fa)
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ignore error when key is not exist

(cherry picked from commit 027f9aaf09fd63a887a8d725b542930219f43707)
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ansible: shorten default control_path

Default ansible control_path option is too long, so we shorten it by
changing the ansible.cfg file.

For more info see: https://github.com/ansible/ansible/issues/11536
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1447569
Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit 0e931d2bce1030fcec9121a8dba70009cff83403)
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1530 from ceph/wip-rgw-testing-stable-2.2

rgw testing stable 2.2

Merge pull request #1523 from ceph/backport-pg-message

ceph-common: improve error message on restart osd daemon script

test: check if default bucket object quota is applied

Resolves: rhbz#1391500

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit db19984010fd75ece0ead7c49ab36ce3c707be15)

rgw: test functionality of conf vars and pool creation for tuning

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit 734a07f0b9feb1aba43b5ead7621ffe63401d6b3)

ceph-common: improve error message on restart osd daemon script

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b04d18988e7f4c3da332af8941550ba0af18365a)

Merge pull request #1509 from ceph/wip-stable-rgw-conf-var-move

rgw: move default bucket quota conf vars to global

ansible: shorten default control_path

Default ansible control_path option is too long, so we shorten it by
changing the ansible.cfg file.

For more info see: https://github.com/ansible/ansible/issues/11536
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1447569
Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit 0e931d2bce1030fcec9121a8dba70009cff83403)
Signed-off-by: Sébastien Han <seb@redhat.com>

ansible: add logging path

Ability to log ansible plays in a log file. Default location is
/var/log/ansible.log
Be sure the user running Ansible has permissions on the logfile.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1445065
Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit 6a69fa9495ffaf16d9aa5b862751ceb2c87bdc8f)
Signed-off-by: Sébastien Han <seb@redhat.com>

rgw: move default bucket quota conf vars to global

"rgw override bucket index max shards" and
"rgw bucket default quota max objects" were in the
client section of the ceph.conf and not being
applied, this commit moves them to global

Resolves: bz#1391500

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit 2aeb3a4957e17f6abace7c58cb795a4c0f50c9fd)

Merge pull request #1506 from ceph/backport-1505

backport: "remove the install-ansible.sh script #1505"

remove the install-ansible.sh script

I'm unsure why this is needed, users should be able to install ansible
on their own in the method they prefer.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 1dc0fab21ca3126bc06aa54af4daae87feedeeee)

Merge pull request #1491 from ceph/backport-1490

backport: "tests: allow for insecure docker registries when testing rhcs #1490"

tests: allow for insecure docker registries when testing rhcs

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 2fada9bd6b0b0099f191be875edeee23a2fe3daa)

Conflicts:
tests/functional/rhcs_setup.yml

Merge pull request #1489 from ceph/backport-1487

Backport: "tests: actually test jewel when we say we are #1487"

tests: switch to using key/value extra-vars instead of json

I continue to have issues with extra-vars as json. The latest issue
being that the ceph_docker_image_tag config option included in the json
was being ignored. I can't find the root cause, by using the key/value
format seems to work.

I've also removed several options here to simply the interface. We can
add those back if they become necessary.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit e384b2498755e5537595158ca498e03013e33673)

tests: actually test a jewel container for the jewel-* docker tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 1fcc3283d8891055ae394d03f3ded1f6a4c8ec64)

Merge pull request #1488 from ceph/backport-1427

tests: enable the rhcs-7-extras-nightly repo on test nodes

tests: enable the rhcs-7-extras-nightly repo on test nodes

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 35aab1274a5ec6ad696040965c87a87310bb019e)

Merge pull request #1486 from ceph/backport-1485

Backport: "tests: test all docker osds are up and in #1485"

tests: test that all docker OSDs are up and in

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 10d39e98dde5042d98d234887ead17ff7d278e55)

tests: adds a way to mark docker only tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 06b36fddf9689659b97c480b8b6644775fac907b)

Merge pull request #1478 from ceph/backport_1477

Backport: Common: Fix condition on osd restart handler

Common: Fix condition on osd restart handler

Fix the condition on the osd restart handler.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1474 from ceph/wip-2.2-rgw-tuning

rgw: set tuning parameters

rgw: set tuning parameters

Change civetweb_num_thread default to 100

Add capability to override number of pgs for
rgw pools.

Add ceph.conf vars to enable default bucket
object quota at users choosing into the ceph.conf.j2
template

Resolves: rhbz#1437173
Resolves: rhbz#1391500

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #1473 from ceph/backport-1464

backport: "purge-cluster: remove unneeded tasks #1464"

purge-cluster: check for any running ceph process after purge

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 5a3f95dfc1bc9bba7ad62999969997ce0a27da13)

purge-cluster: we don't support sysv or upstart anymore

Now that ceph-ansible only supports > jewel we don't need
to bother with sysv or upstart

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 26bdd59f5d445db7991eecd0d13f7c8615bc4812)

purge-cluster: do not need to check for running ceph processes

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 7ca2bddcce76dd505b169457fa8727b62b861756)

purge-cluster: no need to remove ceph.target

The package uninstalls will stop ceph.target

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit aac79df3b376afcb70b4e8888a72ad016079576b)

Merge pull request #1470 from ceph/backport-1466

backport-1466: mon: upgrade calamari when running rolling_update playbook

mon: upgrade calamari when running rolling_update playbook

Prior to this change, ansible was only checking for the existence of the
package, now if upgrade_ceph_packages is true this means we are
performing an upgrade.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1442016
Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit 84d96be197617ff8cdf8141e4a52657f172b2809)

Merge pull request #1467 from ceph/backport_restart_daemons

Common: Restore check_socket

Common: Restore check_socket

Restore the check_socket that was removed by `5bec62b`.
This commit also improves the logging in `restart_*_daemon.sh` scripts

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1450 from ceph/backport-1439

Backport: 'tests: fix ceph_rhcs setup #1439'

Merge pull request #1451 from ceph/doc-static-dirs

Doc static dirs

docs: force an empty _templates dir so sphinx is happy

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 66a9817e56d69b046425dc04a03869fb9a9d8369)
(cherry picked from commit df315fef897705c77bf12780d31ddf17376388fb)

docs: force an empty _static dir so sphinx is happy

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit aacbbf0efe802e9282793dfc8bf7b175596c5aad)
(cherry picked from commit d26ed99dcc1b930a3001ca1bc71f034b9da4f091)

tests: print contents of group_vars/all after modification

This is just nice to see in the test output so we know exactly what
configuration is going to be used.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 0e6d89b9db1b4677db42bf04c88547ebc00e1194)

ceph-common: is ceph_rchs is True do not include install_on_redhat.yml

That task includes logic for upstream installs that we do not want to
run when deploying RHCS.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 37d38b122bde02efc1bde0d61ceeee3999ebb7ff)

tests: set needed config in group_vars/all for rhcs testing

Instead of relying on environment variables and --extra-vars simply
modify the group_vars/all that ships with the specific testing scenario
to enable ceph_rchs testing.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 386afbec83b72dcc3a552682d4376d1bf6a99417)

Conflicts:
tests/functional/rhcs_setup.yml

tests: remove CEPH_STABLE environment variable

I'm removing this because when we use an 'rhcs' scenario then we attempt
to set CEPH_STABLE=false as an environment variable. The issue with that
is because the value is coming from an environment variable it is always
treated as a string and ansible treats that as a boolean True. I plan to
set the ceph_stable value with our rhcs_setup.yml playbook instead of
relying on ---extra-vars and environment variables.

Related ansible issue: https://github.com/ansible/ansible/issues/17193

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 63983fa0b728571518ecdeca249ef37506a00a64)

Merge pull request #1425 from ceph/bump-kraken

common: bump ceph version to kraken

common: bump ceph version to kraken

Kraken has been out for a couple of weeks now and the CI can test both
Kraken and Jewel.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1377 from ceph/ceph-mgr

mgr: add new role for ceph-mgr

test: add mgr bootstrap support

Signed-off-by: Sébastien Han <seb@redhat.com>

docker: fix monitors name

Prior to this change we were deploying a monitor using tis fqdn name but
we were checking its state and performing actions on it using its
shortname.

Signed-off-by: Sébastien Han <seb@redhat.com>

mgr: add new role for ceph-mgr

The Ceph Manager daemon (ceph-mgr) runs alongside monitor daemons, to
provide additional monitoring and interfaces to external monitoring and
management systems.

Only works as of the Kraken release.

Co-Authored-By: Guillaume Abrioux <gabrioux@redhat.com>
Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #1421 from ceph/tox-docs

tox-driven docs

Merge pull request #1422 from k0ste/newfeature

ceph-common: persist "disable_transparent_hugepage"

ceph-common: disable_transparent_hugepage now persist

docs: add missing sections/refs in scenarios.rst

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs: include testing conventions and testinfra in layout.rst

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs: fix typo in testing/index.rst

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs: include the tests and tox files in the glossary

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs: create a tests doc file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs: create a separate tox doc file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs: use glossary for the :doc: directive

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs: add the testing glossary in the toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs: normalize the glossary vs. contents files

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs: create a separate docs tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1412 from guits/fix_handlers

Common: Fix handlers that are not properly triggered.

Merge pull request #1416 from ceph/WingkaiHo-master

enable ceph-mon.target service after monitor start

Merge pull request #1418 from ceph/docs

Docs: an initial take on only test documentation

docs: initial structure for tests docs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs create a separate glossary file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs: the master doc is the glossary file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs: ignore build contents from git

Signed-off-by: Alfredo Deza <adeza@redhat.com>

docs: initial take on doc structure

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #1417 from ceph/docker-env-vars

tests: add docker environment variables to tox.ini

tests: add docker environment variables to tox.ini

This allows us to control the docker registry, image name and tag name
for the containers.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

enable ceph-mon.target service after monitor start

ceph-create-keys unit file was removed here:

* https://github.com/ceph/ceph/commit/8bcb4646b6b9846bb965cdec3ca2a21eb3b26bab
* https://github.com/ceph/ceph/commit/dc5fe8d415858358bd0baf5d8dce0a753f5e0cea

As a consequence the systemctl preset command now fails to run since the
unit does not exist anymore. Due to the redirection in /dev/null we
don't know what's happening.

Ultimately the mon unit doesn't get enabled and the mon service won't
start after reboot.
Removing the old/non-existent unit makes the command succeed now.

ceph fix: https://github.com/ceph/ceph/pull/14226

Signed-off-by: WingkaiHo <sanguosfiang@163.com>
Co-Authored-By: Sébastien Han <seb@redhat.com>

Common: Fix handlers that are not properly triggered.

Until now, only the first task were executed.
The idea here is to use `listen` statement to be able to notify multiple
handler and regroup all of them in `./handlers/main.yml` as notifying an
included handler task is not possible.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1415 from dahorak/osd-autodiscovery-fix-check-holders

osd autodiscovery mode: fix holders detection

osd autodiscovery mode: fix holders detection

Small fix for (probably copy&paste) issue from 42ffe6301.

Signed-off-by: Daniel Horak <dahorak@redhat.com>

Merge pull request #1414 from ceph/scenario-alias

tests: add an alias of cluster for the centos7_cluster scenario

tests: add an alias of cluster for the centos7_cluster scenario

When testing this downstream it makes more sense for this scenario to be
named just 'cluster' because have 'centos7' in the name is misleading.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>