cobbler: Set hostname using DHCP every boot on testnodes

This is for use with FOG.  When we create an OS image, the image
inherits the testnode's hostname that was used to create the image.

This rc.local snippet will make sure the testnode's hostname is set to
whatever the IP's PTR record is on every boot.

`rclocal_nameserver` should be set in the inventory groupvars.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #340 from ceph/wip-find-fail

testnode: Avoid job failure due to fs loop

testnode: Avoid job failure due to fs loop

Example:

failed: [smithi205.front.sepia.ceph.com] (item=/var/run/) => {"changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004976", "end": "2017-09-29 18:42:23.050412", "failed": true, "failed_when_result": true, "item": "/var/run/", "rc": 1, "start": "2017-09-29 18:42:23.045436", "stderr": "find: File system loop detected; '/var/run/rpc_pipefs/gssd' is part of the same file system loop as '/var/run/rpc_pipefs'.", "stderr_lines": ["find: File system loop detected; '/var/run/rpc_pipefs/gssd' is part of the same file system loop as '/var/run/rpc_pipefs'."], "stdout": "", "stdout_lines": []}

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #345 from ceph/wip-21989

testnode: Be super aggressive with wiping LVM data

testnode: Be super aggressive with wiping LVM data

Fixes: http://tracker.ceph.com/issues/21989
Fixes: http://tracker.ceph.com/issues/21935 (hopefully)
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #344 from ceph/wip-named-ddos

nameserver: Double max amount of concurrent connections

nameserver: Double max amount of concurrent connections

I observed an unintentional DoS on ns1.front last night right as most of
the nightly scheduled jobs started up.  Lots of "nf_conntrack: table
full, dropping packet" messages in the syslog.

Doubling it should be safe.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #343 from ceph/wip-shuffle-tasks

testnode: Shuffle tasks around to make sure packages install first

testnode: Shuffle tasks around to make sure packages install first

I moved lvm.conf in https://github.com/ceph/ceph-cm-ansible/pull/342
because I wanted all the disk management tasks clustered together.  I
failed to take into account the fact that the lvm2 package might not be
installed yet (like on OVH nodes).

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #342 from ceph/wip-var-lib-ceph

testnode: Mount /var/lib/ceph to NVMe partition on smithi

testnode: Mount /var/lib/ceph to NVMe partition on smithi

Fixes: http://tracker.ceph.com/issues/20910
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #341 from ceph/wip-dmsetup

testnode: Update dependencies for zap tasks

testnode: Update dependencies for zap tasks

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #339 from ceph/wip-lvm

Support LVM on testnodes

testnode: Docs for lvm tasks

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Support logical volume management

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #338 from ceph/wip-smithi-part

Re-do how we partition drives

testnode: Add some missing tags

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Add README for partition_dict

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Erase /scratch_devs each run in case the devices changed

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Use new drives_to_partition dict to partition drives

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Create gpt instead of msdos table.

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: We repartition on every job after zapping

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #337 from ceph/wip-apt-pinning

testnode: Update apt pinning URL

testnode: Update apt pinning URL

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #336 from ceph/wip-ntp-install

cobbler: Install ntp package on RHEL/CentOS during kickstart

cobbler: Install ntp package on RHEL/CentOS during kickstart

Apparently doesn't come stock with CentOS 7.4 which was causing the
Cobbler ansible-playbook post-install run to fail.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #335 from ceph/wip-pub-ssl

public_facing: Finally get letsencrypt automated renewal  working

public_facing: Finally get letsencrypt automated renewal  working

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #334 from ceph/wip-selinux-modules

Make selinux policy module checking bulletproof

common: Check for *any* empty selinux policy module files

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: reinstall pcpupstream selinux policy module if corrupt

Makes sure policy module is latest and not empty

Fixes: http://tracker.ceph.com/issues/21403
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #333 from ceph/wip-centos74

cobbler: Add CentOS 7.4

cobbler: Add CentOS 7.4

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #326 from ceph/wip-20623

testnodes: Clean up leftover Ceph artifacts from previous jobs

Merge pull request #331 from ceph/wip-cobbler-script

tools: Script to simplify Cobbler user creation

testnodes: Clean up leftover Ceph artifacts from previous jobs

Fixes: http://tracker.ceph.com/issues/20623
Signed-off-by: David Galloway <dgallowa@redhat.com>

tools: Script to simplify Cobbler user creation

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #330 from ceph/wip-sudo-nvme

common: Grant nrpe user access to `sudo nvme`

common: Grant nrpe user access to `sudo nvme`

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #329 from ceph/wip-krbd-lvm-conf

testnode: edit lvm.conf on both ubuntu and centos

testnode: edit lvm.conf on both ubuntu and centos

This modification is needed for krbd xfstests (e.g. generic/081), so
factor out lvm.conf snippet from setup-ubuntu.yml into tasks/.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

testnode: correct krbd max_partitions value in lvm.conf

1024 is clearly a made up number.  16 reflects the current reality,
more or less:

  "Under single_major scheme introduced in kernel 3.14, rbd devices
  support up to 16 partitions.  In pre-single_major mode, rbd driver
  takes up the whole major."

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #327 from ceph/wip-20456

teuthology: minor README update

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #318 from ceph/wip-smart-nvme-binary

Support NVMe drive monitoring via Nagios

Reviewed-by: Dan Mick <dmick@redhat.com>

teuthology: minor README update

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #325 from ceph/wip-nrpe-pidfile

common: Change path to nrpe pid_file on RedHat/CentOS

common: Change path to nrpe pid_file on RedHat/CentOS

I can't find the exact commit in nrpe that caused this to change but the
path has changed and the nrpe service fails to start on CentOS/RHEL now.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #324 from ceph/wip-zap-fix

testnode: Try wipefs without --force if flag is missing

testnode: Try wipefs without --force if flag is missing

Fixes bug introduced in https://github.com/ceph/ceph-cm-ansible/pull/323

The version of wipefs shipped with Trusty lacks the --force flag

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #323 from ceph/wip-20533

Updates to testnode disk zapping

testnode: Retry zapping disk if sgdisk fails the first time

I've seen `sgdisk -Z` fail the first time on RMA replacement drives from HGST but succeed on re-running it.

@alfredodeza recommended we add a retry.  Can't hurt.

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Shred any leftover ceph osd metadata on non-root partitions

Fixes: http://tracker.ceph.com/issues/20533
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #322 from ceph/wip-www

public_facing: Create separate task yml for letsencrypt/nginx/ipv6 hosts

public_facing: Create separate task yml for letsencrypt/nginx/ipv6 hosts

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #321 from ceph/wip-download

public_facing: Changes to download.ceph.com crontab

public_facing: Changes to download.ceph.com crontab

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #320 from ceph/wip-rhel74

cobbler: Add RHEL 7.4

Merge pull request #319 from ceph/wip-18126

Repo changes for Ubuntu testnodes

testnode: Only install custom fastcgi repo on Trusty testnodes

Now that we're running apt/repos.yml tasks on Xenial, we can move
the fastcgi repo to ubuntu_14.yml since it's exclusive to that distro
version.

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Permanently disable/remove custom blkin repo

See https://github.com/ceph/ceph-cm-ansible/pull/156

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Perform repo tasks on all Ubuntu testnodes

Just realized apt/repos.yml hasn't been getting run on Xenial testnodes.
There are some important tasks in there that should be run (e.g., Removing custom repos from
/etc/apt/sources.list.d).

We aren't using apt-mirror for Xenial so I've limited writing the master
sources.list to Trusty and earlier.

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Add custom repo to Xenial to provide newer version of valgrind

Fixes: http://tracker.ceph.com/issues/18126#note-11
Signed-off-by: David Galloway <dgallowa@redhat.com>

cobbler: Add RHEL 7.4

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #317 from ceph/wip-13522

Update libgoogle-perftools4 on Xenial

Update libgoogle-perftools4 on Xenial

Fixes: http://tracker.ceph.com/issues/13522#note-51
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #316 from ceph/wip-retry-unregister

common: Retry unregister beta distro from CDN

common: Retry unregister beta distro from CDN

Seeing intermittent `subscription-manager unregister` failures in Octo.
Since the register task has retries, I see no reason unregistering
can't.

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: Add support for NVMe drives in smart.sh

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: Install nvme-cli for use with smart.sh

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: Don't return WARN if previous disk is CRIT in smart.sh

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #314 from ceph/wip-www-cron

public_facing: Add cronjobs for www.ceph.com

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #315 from ceph/wip-new-ansible

common: Workaround for ansible bug conditional evaluation

Reviewed-by: Dan Mick <dmick@redhat.com>

common: Workaround for new ansible conditional evaluation bug

Prior to ansible v2.2.2.0, if the first conditional was false, the second
conditional wasn't checked.

This change makes sure nrpe-selinux.yml is only included on RHEL/CentOS
which are the only OSes that will have stdout for the selinux_status
var.

Signed-off-by: David Galloway <dgallowa@redhat.com>

public_facing: Add cronjobs for www.ceph.com

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #313 from ceph/wip-fix-312

common: Fix a bug in PR #312

Reviewed-by: Dan Mick <dmick@redhat.com>

common: Fix a bug in PR #312

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #312 from ceph/wip-15439

common: Clean apt cache before updating it

Reviewed-by: Dan Mick <dmick@redhat.com>

common: Clean apt cache before updating it

This is the first apt transaction that gets run in the common role.

Fixes: http://tracker.ceph.com/issues/15439
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #307 from ceph/wip-public-facing

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #311 from ceph/wip-vps-replace

tools: Replace mira091 w mira005 and add missing hv03

Reviewed-by: Dan Mick <dmick@redhat.com>

tools: Replace mira091 w mira005 and add missing hv03

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #310 from ceph/wip-19126

common: Reinstall selinux-policy-targeted if needed

Reviewed-by: Dan Mick <dmick@redhat.com>

common: Reinstall selinux-policy-targeted if needed

This ensures module file integrity so we can build the nrpe selinux
module in the proceeding tasks.

Fixes: http://tracker.ceph.com/issues/19126
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #309 from ceph/wip-19126

common: Delete leftover SELinux policy modules

Reviewed-by: Dan Mick <dmick@redhat.com>

common: Delete leftover SELinux policy modules

Jobs were occasionally failing due to
/etc/selinux/targeted/active/modules/400/mod_fastcgi/lang_ext being
corrupt.  mod_fastcgi is removed and reinstalled with every
testnodes.yml run anyway so this just makes sure valid SELinux policy
modules are put in place.

Fixes: http://tracker.ceph.com/issues/19126
Signed-off-by: David Galloway <dgallowa@redhat.com>

public_facing: Disable PasswordAuthentication in sshd config

Signed-off-by: David Galloway <dgallowa@redhat.com>

public_facing: Add status.sepia.ceph.com-specific tasks

Signed-off-by: David Galloway <dgallowa@redhat.com>

public_facing: Add download.ceph.com-specific tasks

Signed-off-by: David Galloway <dgallowa@redhat.com>

public_facing: Add fail2ban support

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #308 from dmick/master

Reduce 'internal_poll_interval' to stem the CPU usage tide

Reduce 'internal_poll_interval' to stem the CPU usage tide

See https://github.com/ansible/ansible/pull/16560 and
https://github.com/ansible/ansible/issues/14219

Signed-off-by: Dan Mick <dan.mick@redhat.com>

public_facing: Enable ufw support

Signed-off-by: David Galloway <dgallowa@redhat.com>

public_facing: Create base role config

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #299 from ceph/wip-nsupdate

Add role for nsupdate_web

Add role for nsupdate_web

See https://github.com/zmc/nsupdate-web/

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #306 from ceph/wip-lsbrelease

nameserver: Install redhat-lsb-core for `lsb_release`

Reviewed-by: Dan Mick <dmick@redhat.com>

nameserver: Install redhat-lsb-core for `lsb_release`

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #305 from dmick/master

testnode/tasks/ntp: remove '/etc/network/if-up.d/ntpdate' on Xenial

testnode/tasks/ntp: remove '/etc/network/if-up.d/ntpdate' on Xenial

It's buggy (see comment in code) and we don't need it, because
teuthology does what it would do anyway.  Supposed to be fixed in
a later version of the Debian package which is in Yakkety.

Fixes: http://tracker.ceph.com/issues/18837
Signed-off-by: Dan Mick <dan.mick@redhat.com>