Merge pull request #289 from ceph/wip-rhel73

Add RHEL7.3 support

Reviewed-by: Dan Mick <dmick@redhat.com>

Add RHEL7.3 support

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #288 from ceph/wip-nrpe-rhel

common: Install a few additional packages for NRPE

common: Install a few additional packages for NRPE

check_load nagios plugin needs to be installed on RHEL and CentOS

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #286 from ceph/wip-users-become

Use "become: true" for users role

Use "become: true" for users role

This is needed if the users role is called manually.  Typically it's
called via the common role and "become: true" is inherited.

But in the case of just needing to update update the teuthology_user's pubkeys with @all.pub, just calling the users role manually is quicker.  sudo access is obviously required to modify users and groups.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #285 from ceph/wip-beta2

common: Change method of detecting beta distro

common: Change method of detecting beta distro

It was recently discovered that Release Candidate ISOs of RHEL don't output "Beta" or
"Alpha" in `lsb_release` output.  Therefore, RHEL RC Distros were
registering with CDN and not putting beta_repos in place.

This new method registers every RHEL system with CDN, checks if there are
any available matching repos based on ansible_distribution_version, and
either:
  - Unregisters from CDN and runs beta_distros.yml if there are no CDN repos
  - Continues with setting up desired rhsm_repos

I also added some additional working to task descriptions to hopefully
claer up what's going on in this playbook since there are a lot of when
and if statements.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #283 from ceph/wip-common-updates

Updates to common role

Reviewed-by: Dan Mick <dmick@redhat.com>

common: Add nagios check_mem command

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: Create nagios_allowed_hosts var for lab-specific nagios IPs

Signed-off-by: David Galloway <dgallowa@redhat.com>

Add common.yml playbook file

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #282 from ceph/wip-vmlist

Add vmhosts to vmlist.py and run virsh in read-only mode

Reviewed-by: Dan Mick <dmick@redhat.com>

Add vmhosts to vmlist.py and run virsh in read-only mode

'virsh -r' allows vmlist.py to work with RHEV hypervisor hosts

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #280 from ceph/wip-ubuntu-groups

users: Leave managed_admin_users' other group memberships alone

Reviewed-by: Dan Mick <dmick@redhat.com>

users: Leave managed_admin_users' other group memberships alone

This change is mainly so the users role can be run on vps_hosts.

Previously, in order to update the ubuntu user's authorized_keys on
vps_hosts, the testnodes role had to be run which caused ubuntu to be
removed from the libvirtd group.

The ubuntu user is in managed_admin_users on vps_hosts so this will
ensure the user is added to sudo and left in the libvirtd group.

managed_users, on the other hand, we want to make sure are only in
groups we specify so the "Create all users without sudo access." task is
left as-is (without append: yes).

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #279 from ceph/wip-beta-repos

defaults: beta_repos must be a dict

defaults: beta_repos must be a dict

This causes ceph-cm-ansible to fail every teuthology job due to
the bad default: 'with_dict expects a dict'

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #278 from ceph/wip-beta-distros

Add support for testing RHEL beta distros using internal repos

cobbler: redhat-lsb-core package is required for ansible_lsb variables

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnodes: Add support for internal RHEL beta repos

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #277 from ceph/wip-teuthology

teuthology role updates

Provide a default for teuthology_yaml_extra

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #272 from ceph/wip-teuth-kill

teuthology: Create tkill group and grant tkill_users right to run `kill`

teuthology: Create test-admins group and grant test_admins right to run `kill`

Fixes: http://tracker.ceph.com/issues/16614
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #276 from ceph/wip-teuth-fixes

A couple fixes to teuthology role

Merge pull request #274 from ceph/wip-profile-tasks

Use the profile_tasks plugin

Use the profile_tasks plugin

This will cause ansible to display timing information for tasks,
allowing us to see what's slow and potentially improve speed.

I've been using this locally for a while now. It's nice.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #275 from ceph/wip-ssh-retries

Retry SSH connections five times

Merge pull request #273 from ceph/wip-strategy-free

Use strategy: free on some playbooks

Retry SSH connections five times

This may help with some of the problems we've been seeing.

Related to http://tracker.ceph.com/issues/16826
Inspired by
https://github.com/ansible/ansible/issues/13401#issuecomment-216768025

Signed-off-by: Zack Cerza <zack@redhat.com>

Use strategy: free on some playbooks

See http://docs.ansible.com/ansible/playbooks_strategies.html#strategies

This could potentially improve performance.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology: Ignore any errors when disabling apache2

Signed-off-by: David Galloway <dgallowa@redhat.com>

teuthology: Additional packages needed to run bootstrap now

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #271 from ceph/wip-no-fw

testnode: Disable firewalld and iptables regardless of OS version

testnode: Disable firewalld and iptables regardless of OS version

iptables was recently found installed and running on a RHEL7 system.
Previous testnode playbook runs wouldn't catch this since it shouldn't
be installed in the first place.  This change ensures firewalld and
iptables are stopped on all RPM-based distros.

Fixes: http://tracker.ceph.com/issues/16809
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #270 from ceph/wip-keys-retry

Add a retry when cloning the keys repo

Add a retry when cloning the keys repo

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #269 from ceph/wip-cobbler-gitrace

Use flock to avoid race conditions with git

Use flock to avoid race conditions with git

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #265 from ceph/wip-nagios-common

Move NRPE setup to common role

Merge pull request #267 from ceph/wip-no-gperftools-devel

testnode: drop gperftools-devel

Merge pull request #268 from ceph/wip-failure-log-debug

If a YAML error is hit, log the raw object

Reviewed-by: Dan Mick <dmick@redhat.com>

If a YAML error is hit, log the raw object

We are seeing: "RepresenterError: cannot represent an object: pcp"

Signed-off-by: Zack Cerza <zack@redhat.com>

testnode: drop gperftools-devel

Stop installing the gperftools-devel package on testnodes, for the
following reasons:

1. Ceph does not require gperftools-devel at runtime, so installing it
   unnecessarily just slows everything down at this point.

2. When we want to test newer builds of gperftools (say, from RHEL 7.3),
   if the corresponding "gperftools-devel" package is not available on
   the test node, then yum will fail the transaction.

(Additionally, gperftools-devel is no longer in EPEL 7, and it has moved to
RHEL 7 Optional in RHEL 7.2 (https://bugzilla.redhat.com/1213879,
https://access.redhat.com/errata/RHEA-2015:2293). The "epel" list is no
longer the correct list for this package on redhat_7 and centos_7.)

Merge pull request #266 from ceph/wip-timeout

use 120s instead of default 12s for cmd timeout

common: Update nrpe SELinux policy

This should've been done when smart.sh replaced smart.pl.
I just didn't notice smart.sh was getting denied by SELinux until I
started monitoring disks in Octo.  Evidently the new script requires
much more permission to run.

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: Create README

Signed-off-by: David Galloway <dgallowa@redhat.com>

cobbler: Install some ansible dependencies during kickstart

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: Combine some nrpe-selinux package installations

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: Always install disk_monitoring scripts when nagios tag is called

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: Move smartmontools to disk_monitoring task

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: Enable nrpe installation without epel repo on RHEL/CentOS

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: Move nrpe package install to common role

Signed-off-by: David Galloway <dgallowa@redhat.com>

Move NRPE setup to common role

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Configure firewalld (when enabled) for NRPE

Signed-off-by: David Galloway <dgallowa@redhat.com>

use 120s instead of default 12s for cmd timeout

Increase timeout from 12s to 120s for ansible cmds

Signed-off-by: Vasu Kulkarni vasu@redhat.com

Merge pull request #264 from ceph/wip-16615

Don't use a shallow copy of keys.git

Don't use a shallow copy of keys.git

http://tracker.ceph.com/issues/16615
Fixes: 16615
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #263 from ceph/wip-cpan

testnodes: Install perl-CPAN on yum systems

testnodes: Install perl-CPAN on yum systems

perl-CPAN is required to install Amazon::S3 using the 'cpan' command

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #262 from ceph/wip-become

More Ansible v2 fixes

Bracket bare variables for Ansible v2

Signed-off-by: David Galloway <dgallowa@redhat.com>

Add missing 'become' directive to a few more roles

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #261 from ceph/wip-rclocal-msg

cobbler: Output message indicating Ansible is running after firstboot

Reviewed-by: Dan Mick <dmick@redhat.com>

cobbler: Output message indicating Ansible is running after firstboot

Fixes: http://tracker.ceph.com/issues/14297
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #260 from ceph/wip-downstream-fixes

downstream_setup role fixes

downstream_setup: Ensure role is only run on RHEL or CentOS

Signed-off-by: David Galloway <dgallowa@redhat.com>

downstream_setup: Add 'become' privilege escalation directive

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #259 from ceph/wip-entitlements-bools

common: Fix boolean expressions

Fix boolean expressions

Signed-off-by: Zack Cerza <zack@redhat.com>

Revert "Use ansible_user instead of ansible_ssh_user"

This reverts commit 7cc43cf253fc0dceea381ff11ed1d3e546f70ff8.

Merge pull request #258 from ceph/wip-pip

Add missing pip.yml

Add missing pip.yml

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #255 from ceph/wip-ansible-2

DNM: Move to ansible 2

Merge pull request #257 from ceph/wip-disable-firewalld

testnodes: Disable firewalld service after reboot on RPM-based distros

Reviewed-by: Dan Mick <dmick@redhat.com>

testnodes: Disable firewalld service after reboot on RPM-based distros

Fixes: http://tracker.ceph.com/issues/16455
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #256 from ceph/wip-global-cpan

testnodes: Configure cpan and install Amazon::S3 on all OSes

Disable become for repo cloning (2.x method)

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove sudo role; replace with 'become' setting

The old method of setting the default behavior for become/sudo has
changed in 2.x; fortunately the new way is a bit more elegant

Signed-off-by: Zack Cerza <zack@redhat.com>

Quote variables used by with_items

Signed-off-by: Zack Cerza <zack@redhat.com>

cobbler: Install ansible from PyPI

The distros we're using don't ship ansible 2.

Signed-off-by: Zack Cerza <zack@redhat.com>

Locate templates properly

See https://github.com/ansible/ansible/issues/14161

Signed-off-by: Zack Cerza <zack@redhat.com>

Use ansible_user instead of ansible_ssh_user

The option was renamed in 2.0

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #249 from ceph/wip-gw-networking

Network configuration for gateway role

Reviewed-by: Dan Mick <dmick@redhat.com>

gateway: Add logrotate functionality

Signed-off-by: David Galloway <dgallowa@redhat.com>

gateway: Add README documentation for networking, firewall, fail2ban

Signed-off-by: David Galloway <dgallowa@redhat.com>

gateway: Add fail2ban support

Signed-off-by: David Galloway <dgallowa@redhat.com>

gateway: Configure firewalld

Signed-off-by: David Galloway <dgallowa@redhat.com>

gateway: Enable server network config

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #252 from ceph/wip-cblr-wget

cobbler: testnode post-install rc.local fixes

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Zack Cerza <zack@redhat.com>

testnodes: Configure cpan and install Amazon::S3 on all OSes

Fixes: http://tracker.ceph.com/issues/15316
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #254 from ceph/wip-clone-once

Only clone the keys repo once per execution

Reviewed-by: Dan Mick <dmick@redhat.com>

Only clone the keys repo once per execution

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #253 from ceph/wip-fast-keys

users: Greatly speed up ssh pubkey deployment

Reviewed-by: Dan Mick <dmick@redhat.com>

Put the keys repo in ~/.cache/src/ by default

To avoid multiple invocations by different users on the same host
stepping on each other.

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove unnecessary retry

Since each key isn't being fetched from a remote server any longer, we
can drop the retries.

Signed-off-by: Zack Cerza <zack@redhat.com>

Speed up key deployment by using the git repo

Instead of downloading each key over HTTPS from github.com, we can
simply clone the entire repo (with depth 1) and lookup each key using
the username.

On my laptop, execution time went from 2m49s to 29s.

Signed-off-by: Zack Cerza <zack@redhat.com>

users: Add defaults for keys_repo

Signed-off-by: Zack Cerza <zack@redhat.com>

users: split tasks/main.yml into separate files

Signed-off-by: Zack Cerza <zack@redhat.com>