testnode: Be super aggressive with wiping LVM data

Fixes: http://tracker.ceph.com/issues/21989
Fixes: http://tracker.ceph.com/issues/21935 (hopefully)
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #344 from ceph/wip-named-ddos

nameserver: Double max amount of concurrent connections

nameserver: Double max amount of concurrent connections

I observed an unintentional DoS on ns1.front last night right as most of
the nightly scheduled jobs started up.  Lots of "nf_conntrack: table
full, dropping packet" messages in the syslog.

Doubling it should be safe.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #343 from ceph/wip-shuffle-tasks

testnode: Shuffle tasks around to make sure packages install first

testnode: Shuffle tasks around to make sure packages install first

I moved lvm.conf in https://github.com/ceph/ceph-cm-ansible/pull/342
because I wanted all the disk management tasks clustered together.  I
failed to take into account the fact that the lvm2 package might not be
installed yet (like on OVH nodes).

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #342 from ceph/wip-var-lib-ceph

testnode: Mount /var/lib/ceph to NVMe partition on smithi

testnode: Mount /var/lib/ceph to NVMe partition on smithi

Fixes: http://tracker.ceph.com/issues/20910
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #341 from ceph/wip-dmsetup

testnode: Update dependencies for zap tasks

testnode: Update dependencies for zap tasks

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #339 from ceph/wip-lvm

Support LVM on testnodes

testnode: Docs for lvm tasks

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Support logical volume management

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #338 from ceph/wip-smithi-part

Re-do how we partition drives

testnode: Add some missing tags

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Add README for partition_dict

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Erase /scratch_devs each run in case the devices changed

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Use new drives_to_partition dict to partition drives

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Create gpt instead of msdos table.

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: We repartition on every job after zapping

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #337 from ceph/wip-apt-pinning

testnode: Update apt pinning URL

testnode: Update apt pinning URL

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #336 from ceph/wip-ntp-install

cobbler: Install ntp package on RHEL/CentOS during kickstart

cobbler: Install ntp package on RHEL/CentOS during kickstart

Apparently doesn't come stock with CentOS 7.4 which was causing the
Cobbler ansible-playbook post-install run to fail.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #335 from ceph/wip-pub-ssl

public_facing: Finally get letsencrypt automated renewal  working

public_facing: Finally get letsencrypt automated renewal  working

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #334 from ceph/wip-selinux-modules

Make selinux policy module checking bulletproof

common: Check for *any* empty selinux policy module files

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: reinstall pcpupstream selinux policy module if corrupt

Makes sure policy module is latest and not empty

Fixes: http://tracker.ceph.com/issues/21403
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #333 from ceph/wip-centos74

cobbler: Add CentOS 7.4

cobbler: Add CentOS 7.4

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #326 from ceph/wip-20623

testnodes: Clean up leftover Ceph artifacts from previous jobs

Merge pull request #331 from ceph/wip-cobbler-script

tools: Script to simplify Cobbler user creation

testnodes: Clean up leftover Ceph artifacts from previous jobs

Fixes: http://tracker.ceph.com/issues/20623
Signed-off-by: David Galloway <dgallowa@redhat.com>

tools: Script to simplify Cobbler user creation

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #330 from ceph/wip-sudo-nvme

common: Grant nrpe user access to `sudo nvme`

common: Grant nrpe user access to `sudo nvme`

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #329 from ceph/wip-krbd-lvm-conf

testnode: edit lvm.conf on both ubuntu and centos

testnode: edit lvm.conf on both ubuntu and centos

This modification is needed for krbd xfstests (e.g. generic/081), so
factor out lvm.conf snippet from setup-ubuntu.yml into tasks/.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

testnode: correct krbd max_partitions value in lvm.conf

1024 is clearly a made up number.  16 reflects the current reality,
more or less:

  "Under single_major scheme introduced in kernel 3.14, rbd devices
  support up to 16 partitions.  In pre-single_major mode, rbd driver
  takes up the whole major."

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #327 from ceph/wip-20456

teuthology: minor README update

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #318 from ceph/wip-smart-nvme-binary

Support NVMe drive monitoring via Nagios

Reviewed-by: Dan Mick <dmick@redhat.com>

teuthology: minor README update

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #325 from ceph/wip-nrpe-pidfile

common: Change path to nrpe pid_file on RedHat/CentOS

common: Change path to nrpe pid_file on RedHat/CentOS

I can't find the exact commit in nrpe that caused this to change but the
path has changed and the nrpe service fails to start on CentOS/RHEL now.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #324 from ceph/wip-zap-fix

testnode: Try wipefs without --force if flag is missing

testnode: Try wipefs without --force if flag is missing

Fixes bug introduced in https://github.com/ceph/ceph-cm-ansible/pull/323

The version of wipefs shipped with Trusty lacks the --force flag

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #323 from ceph/wip-20533

Updates to testnode disk zapping

testnode: Retry zapping disk if sgdisk fails the first time

I've seen `sgdisk -Z` fail the first time on RMA replacement drives from HGST but succeed on re-running it.

@alfredodeza recommended we add a retry.  Can't hurt.

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Shred any leftover ceph osd metadata on non-root partitions

Fixes: http://tracker.ceph.com/issues/20533
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #322 from ceph/wip-www

public_facing: Create separate task yml for letsencrypt/nginx/ipv6 hosts

public_facing: Create separate task yml for letsencrypt/nginx/ipv6 hosts

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #321 from ceph/wip-download

public_facing: Changes to download.ceph.com crontab

public_facing: Changes to download.ceph.com crontab

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #320 from ceph/wip-rhel74

cobbler: Add RHEL 7.4

Merge pull request #319 from ceph/wip-18126

Repo changes for Ubuntu testnodes

testnode: Only install custom fastcgi repo on Trusty testnodes

Now that we're running apt/repos.yml tasks on Xenial, we can move
the fastcgi repo to ubuntu_14.yml since it's exclusive to that distro
version.

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Permanently disable/remove custom blkin repo

See https://github.com/ceph/ceph-cm-ansible/pull/156

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Perform repo tasks on all Ubuntu testnodes

Just realized apt/repos.yml hasn't been getting run on Xenial testnodes.
There are some important tasks in there that should be run (e.g., Removing custom repos from
/etc/apt/sources.list.d).

We aren't using apt-mirror for Xenial so I've limited writing the master
sources.list to Trusty and earlier.

Signed-off-by: David Galloway <dgallowa@redhat.com>

testnode: Add custom repo to Xenial to provide newer version of valgrind

Fixes: http://tracker.ceph.com/issues/18126#note-11
Signed-off-by: David Galloway <dgallowa@redhat.com>

cobbler: Add RHEL 7.4

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #317 from ceph/wip-13522

Update libgoogle-perftools4 on Xenial

Update libgoogle-perftools4 on Xenial

Fixes: http://tracker.ceph.com/issues/13522#note-51
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #316 from ceph/wip-retry-unregister

common: Retry unregister beta distro from CDN

common: Retry unregister beta distro from CDN

Seeing intermittent `subscription-manager unregister` failures in Octo.
Since the register task has retries, I see no reason unregistering
can't.

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: Add support for NVMe drives in smart.sh

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: Install nvme-cli for use with smart.sh

Signed-off-by: David Galloway <dgallowa@redhat.com>

common: Don't return WARN if previous disk is CRIT in smart.sh

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #314 from ceph/wip-www-cron

public_facing: Add cronjobs for www.ceph.com

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #315 from ceph/wip-new-ansible

common: Workaround for ansible bug conditional evaluation

Reviewed-by: Dan Mick <dmick@redhat.com>

common: Workaround for new ansible conditional evaluation bug

Prior to ansible v2.2.2.0, if the first conditional was false, the second
conditional wasn't checked.

This change makes sure nrpe-selinux.yml is only included on RHEL/CentOS
which are the only OSes that will have stdout for the selinux_status
var.

Signed-off-by: David Galloway <dgallowa@redhat.com>

public_facing: Add cronjobs for www.ceph.com

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #313 from ceph/wip-fix-312

common: Fix a bug in PR #312

Reviewed-by: Dan Mick <dmick@redhat.com>

common: Fix a bug in PR #312

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #312 from ceph/wip-15439

common: Clean apt cache before updating it

Reviewed-by: Dan Mick <dmick@redhat.com>

common: Clean apt cache before updating it

This is the first apt transaction that gets run in the common role.

Fixes: http://tracker.ceph.com/issues/15439
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #307 from ceph/wip-public-facing

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #311 from ceph/wip-vps-replace

tools: Replace mira091 w mira005 and add missing hv03

Reviewed-by: Dan Mick <dmick@redhat.com>

tools: Replace mira091 w mira005 and add missing hv03

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #310 from ceph/wip-19126

common: Reinstall selinux-policy-targeted if needed

Reviewed-by: Dan Mick <dmick@redhat.com>

common: Reinstall selinux-policy-targeted if needed

This ensures module file integrity so we can build the nrpe selinux
module in the proceeding tasks.

Fixes: http://tracker.ceph.com/issues/19126
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #309 from ceph/wip-19126

common: Delete leftover SELinux policy modules

Reviewed-by: Dan Mick <dmick@redhat.com>

common: Delete leftover SELinux policy modules

Jobs were occasionally failing due to
/etc/selinux/targeted/active/modules/400/mod_fastcgi/lang_ext being
corrupt.  mod_fastcgi is removed and reinstalled with every
testnodes.yml run anyway so this just makes sure valid SELinux policy
modules are put in place.

Fixes: http://tracker.ceph.com/issues/19126
Signed-off-by: David Galloway <dgallowa@redhat.com>

public_facing: Disable PasswordAuthentication in sshd config

Signed-off-by: David Galloway <dgallowa@redhat.com>

public_facing: Add status.sepia.ceph.com-specific tasks

Signed-off-by: David Galloway <dgallowa@redhat.com>

public_facing: Add download.ceph.com-specific tasks

Signed-off-by: David Galloway <dgallowa@redhat.com>

public_facing: Add fail2ban support

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #308 from dmick/master

Reduce 'internal_poll_interval' to stem the CPU usage tide

Reduce 'internal_poll_interval' to stem the CPU usage tide

See https://github.com/ansible/ansible/pull/16560 and
https://github.com/ansible/ansible/issues/14219

Signed-off-by: Dan Mick <dan.mick@redhat.com>

public_facing: Enable ufw support

Signed-off-by: David Galloway <dgallowa@redhat.com>

public_facing: Create base role config

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #299 from ceph/wip-nsupdate

Add role for nsupdate_web

Add role for nsupdate_web

See https://github.com/zmc/nsupdate-web/

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #306 from ceph/wip-lsbrelease

nameserver: Install redhat-lsb-core for `lsb_release`

Reviewed-by: Dan Mick <dmick@redhat.com>

nameserver: Install redhat-lsb-core for `lsb_release`

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #305 from dmick/master

testnode/tasks/ntp: remove '/etc/network/if-up.d/ntpdate' on Xenial

testnode/tasks/ntp: remove '/etc/network/if-up.d/ntpdate' on Xenial

It's buggy (see comment in code) and we don't need it, because
teuthology does what it would do anyway.  Supposed to be fixed in
a later version of the Debian package which is in Yakkety.

Fixes: http://tracker.ceph.com/issues/18837
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #281 from rjfd/wip-opensuse-upstream-pr

Support for running teuthology suites in OpenSUSE Leap images

Merge pull request #303 from ceph/wip-teuth-mime

teuthology: Correct MIME type spec for log files

teuthology: Correct MIME type spec for log files

Gzipped files were being served as text/plain. Instead of applying text/plain to everythincg, extend the default MIME configuration to serve *.log as text/plain.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #302 from ceph/wip-named-chown

nameserver: Fix permissions on /var/named/zones