Merge pull request #849 from ceph/wtf-dnf-bin

prep-fog-capture: Might want to install the script we need

prep-fog-capture: Might want to install the script we need

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #843 from ceph/wip-56434

long_running_cluster: Enable storing coredumps

Merge pull request #848 from ceph/wip-37728

testnode: Provide option to enable firewalld

testnode: Provide option to enable firewalld

Fixes: https://tracker.ceph.com/issues/37728
Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #845 from ceph/wip-63567

testnode: Replace apt_key module

Merge pull request #847 from ceph/teuth-journald

teuthology: Configure week-long journald

Merge pull request #846 from ceph/minimal-responses

nameserver: Disable minimal-responses

teuthology: Configure week-long journald

Fixes: https://tracker.ceph.com/issues/70634
Signed-off-by: David Galloway <david.galloway@ibm.com>

nameserver: Disable minimal-responses

Just upgraded vpn-pub to el9.

BIND 9.16 (EL9) changed the default behavior for minimal-responses,
causing authority and additional (glue) sections to be omitted from
responses. Set minimal-responses no explicitly to restore consistent
behavior across all nameservers regardless of BIND version.

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #844 from ceph/wip-63562

users: Change home dir path for teuthology host

testnode: Replace apt_key module

As of this writing, apt_repos and common_apt_repos are actually unused/empty but
the functionality is still useful.

When a chacra repo is added during a teuthology job, `[trusted=yes]` is added to
the repo file so no worries there.

```
curl -s https://2.chacra.ceph.com/repos/ceph/tentacle/64721f2bfd1c725c9d5f65983a99bac4d01e22d4/ubuntu/jammy/flavors/default/repo
deb [trusted=yes] https://2.chacra.ceph.com/r/ceph/tentacle/64721f2bfd1c725c9d5f65983a99bac4d01e22d4/ubuntu/jammy/flavors/default/ jammy main
```

Fixes: https://tracker.ceph.com/issues/63567
Signed-off-by: David Galloway <david.galloway@ibm.com>

users: Change home dir path for teuthology host

The home dirs live at the /tank/home mountpoint.  This change will default to /home unless the host we're running the role against is in the teuthology ansible group.

Fixes: https://tracker.ceph.com/issues/63562
Signed-off-by: David Galloway <david.galloway@ibm.com>

long_running_cluster: Enable storing coredumps

Fixes: https://tracker.ceph.com/issues/56434
Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #782 from ceph/wip-remove-old-grafana-repo-and-gpg-key

Add cleanup tasks to resolve Grafana apt repository conflict

Add cleanup tasks to resolve Grafana APT repository conflict
The new tasks remove the conflicting key file and duplicate repository entries before adding the standardized Grafana repository, ensuring a clean APT configuration

Signed-off-by: Adam Kraitman <akraitma@li-8b09b2cc-35b7-11b2-a85c-cd1dbade58f9.ibm.com>

Merge pull request #842 from ceph/braggi-disk

jenkins-builder-disk: Fix adami/braggi disk selection

Merge pull request #841 from ceph/bath-disk

jenkins-builder-disk: Missing disk selection for bath

jenkins-builder-disk: Fix adami/braggi disk selection

Signed-off-by: David Galloway <david.galloway@ibm.com>

jenkins-builder-disk: Missing disk selection for bath

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #839 from ceph/wip-mimir-external-url

Updated README with instructions on how to use the external Mimir URL for public_facing hosts and Moved README.md to the correct folder location

document agent_mimir_external_url + usage for external Mimir

Signed-off-by: Adam Kraitman <akraitma@li-8b09b2cc-35b7-11b2-a85c-cd1dbade58f9.ibm.com>

Merge pull request #838 from ceph/epel-mirror

Use epel mirror

common: Use internal epel mirror

Signed-off-by: David Galloway <david.galloway@ibm.com>

common: Remove epel-testing repo

The repo's already disabled by default and the string 'epel-testing' does not appear anywhere in ceph.git or teuthology.git.

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #837 from ceph/wip-update-grafana-agent

roles/grafana_agent: major cleanup and cross-platform fixes

- Update default Mimir endpoint
- Improve port 9090 conflict detection (now fails clearly on all distros)
- Fix node_exporter installation on RedHat (EPEL + correct package)
- Refactor grafana-agent.yaml.j2 to use clean static scrape config + basic_auth
- Add WAL directory management
- Add comprehensive README.md

Signed-off-by: Adam Kraitman <akraitma@li-8b09b2cc-35b7-11b2-a85c-cd1dbade58f9.ibm.com>

Merge pull request #836 from ceph/1001

maas: Use 1001 for cm user

maas: Use 1001 for cm user

teuthology's ubuntu user expects UID 1000

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #788 from ceph/wip-maas-apt

Create role to deploy and configure MAAS

Merge pull request #835 from ceph/short

prep-fog-capture: Set short hostname on boot

prep-fog-capture: Set short hostname on boot

Setting FQDN via `hostname` is not best practice

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #834 from ceph/fix-nameserver-for-deb

Fix nameserver for deb

nameserver: Fix vars for Debian

Signed-off-by: David Galloway <david.galloway@ibm.com>

nameserver: named.conf is in /etc/bind on Debian

Signed-off-by: Dan Mick <dmick@ibm.com>

nameserver: don't create /etc/<X>/named if it doesn't exist

It should exist.  This would have caught the error that
X is sysconfig on el systems and default on deb systems.

Signed-off-by: Dan Mick <dmick@ibm.com>

roles/nameserver: allow defaults file to vary

deb: /etc/default/named
el: /etc/sysconfig/named

Signed-off-by: Dan Mick <dmick@ibm.com>

Merge pull request #832 from ceph/ansible_user

testnode: Add iozone to path

testnode: Ensure iozone is in path

Signed-off-by: David Galloway <david.galloway@ibm.com>

testnode: Default ansible_ssh_user to cm_user, then nothing

Ansible, in their infinite wisdom, have deprecated yet another magic variable breaking backwards compatibility.

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #831 from dmick/rocky10-iozone

testnode: add iozone to rocky_10.yml

testnode: add iozone to rocky_10.yml

I built the RPM and installed it on the apt-mirror service
(on the POK lab Openshift)

Fixes: https://tracker.ceph.com/issues/75381
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Enhance install_grub script for ARM deployment and RHEL 10 support

Merge pull request #821 from ceph/prepfog-v2

prep-fog-capture: Refactor, split, re-add cobbler-provided hacks

Merge pull request #830 from ceph/nameserver-apt

Nameserver role updates

Merge pull request #828 from ceph/builder-disk-uuid

jenkins-builder-disk: Use UUID instead of disk name

nameserver: Optionally disable DNSSEC

Signed-off-by: David Galloway <david.galloway@ibm.com>

nameserver: Disable ipv6 for now

Signed-off-by: David Galloway <david.galloway@ibm.com>

nameserver: Support Ubuntu

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #829 from ceph/force-ntp

tools: Playbook to force time sync

Merge pull request #827 from ceph/fix-forward

nameserver: Match A records to reverse[] in named_domains

Merge pull request #826 from ceph/fix-ntp

ntp-server: Overhaul role

jenkins-builder-disk: Support disk flipping on adami too

Signed-off-by: David Galloway <david.galloway@ibm.com>

tools: Playbook to force time sync

Useful during our emergency bring-up of the POK lab.

We were using the RHV hypervisor hosts as the NTP servers but who knows when they will be accessible again.  For now, NTP server is on soko02

Signed-off-by: David Galloway <david.galloway@ibm.com>

jenkins-builder-disk: Use UUID instead of disk name

Avoids disk renaming tomfoolery during boot

Signed-off-by: David Galloway <david.galloway@ibm.com>

ntp-server: Overhaul role

 - Get rid of netaddr. Modern ntpd supports CIDRs
 - Make OS agnostic
 - Support ufw, don't blow up if no firewall in use

Signed-off-by: David Galloway <david.galloway@ibm.com>

nameserver: Match A records to reverse[] in named_domains

This will let multiple subnets/domains use the same ipvar

For example, bath and infra-compute can both have a if_25Gb_ip
```
bath14.front.sepia.ceph.com if_25Gb_ip=10.20.192.44
infra-compute-01.os.sepia.ceph.com if_25Gb_1_ip=172.16.53.11

  front.sepia.ceph.com:
    forward: front.sepia.ceph.com
    ipvars:
      - ip
      - if_25Gb_1_ip
    dynamic: true
    ddns_hostname_prefixes:
      - ovh
    reverse:
      - 10.20.192
      - 10.20.193
  os.sepia.ceph.com:
    forward: os.sepia.ceph.com
    ipvar: if_25Gb_1_ip
    dynamic: false
    reverse:
      - 172.16.53
      - 172.16.54

```

`bath14 IN A` will get written to front.sepia.ceph.com
`infra-compute-01 IN A` will get written to os.sepia.ceph.com

Signed-off-by: David Galloway <david.galloway@ibm.com>

Refactor MAAS configuration tasks to use ansible.builtin.stat instead of shell commands for improved reliability and clarity

Merge pull request #801 from ceph/playbook-sync

Sync playbooks from David's repo

Sync playbooks from other @David's repo

amend

Removed updated tools/jenkins-builder-disk.yml

prep-fog-capture: Refactor, split, re-add cobbler-provided hacks

- Using systemd instead of rc.local:
  - Configure netplan on boot for Ubuntu
  - Configure NetworkManager on boot for CentOS/Rocky
  - Set hostname
  - Update packages and reboot

Signed-off-by: David Galloway <david.galloway@ibm.com>

testnode: Stop rewriting /etc/hosts

We handle this in prep-fog-capture now by writing /etc/hosts from scratch.

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #825 from ceph/bindutils

testnode: bind-utils for dig

testnode: bind-utils for dig

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #824 from ceph/deps

testnode: semanage and bzip2 for el9/10

testnode: semanage and bzip2 for el9/10

Fixes: https://tracker.ceph.com/issues/74570
Fixes: https://tracker.ceph.com/issues/71624
Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #820 from deepssin/centos9-perl-test-harness

Fix pjd failure: prove - command not available

Fix pjd failure: prove - command not available

Signed-off-by: deepssin <deepssin@redhat.com>

Merge pull request #819 from ceph/c9-fuse

testnode: fuse and net-tools for CentOS 9

testnode: fuse and net-tools for CentOS 9

centos_9_stream.yml isn't actually pulled in

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #817 from ceph/apt-https

common: s/http/https in apt sources as outbound port 80 is blocked in…

Merge pull request #818 from ceph/openssl-rocky

testnode: Install openssl on Rocky

testnode: Install openssl on Rocky

Fixes: https://tracker.ceph.com/issues/74766
Signed-off-by: David Galloway <david.galloway@ibm.com>

common: s/http/https in apt sources as outbound port 80 is blocked in Sepia

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #816 from ceph/rocky-packages

Let's fix some teuthology tests

testnode: Fuse for centos9

Fixes: https://tracker.ceph.com/issues/73630
Signed-off-by: David Galloway <david.galloway@ibm.com>

testnode: net-tools for centos9

Fixes: https://tracker.ceph.com/issues/74589
Signed-off-by: David Galloway <david.galloway@ibm.com>

testnode: Additional missing Rocky packages

Fixes: https://tracker.ceph.com/issues/73630
Fixes: https://tracker.ceph.com/issues/74572
Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #815 from ceph/wget-r10

testnode: Install wget on Rocky 10

testnode: Install wget on Rocky 10

Same as in commit ab478050c523 ("testnode: Install wget on CentOS 9").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #814 from ceph/define-rocky-mirrors

Define rocky repo mirrors

testnode: Specify a few mirrors for Rocky 10

As was the case with EPEL mirrors, hitting the mirrorlist.rockylinux.org endpoint frequently fails and causes jobs to die.  We'll just specify a few mirrors manually.

DNF's behavior is to try them in order when multiple baseurls are defined.

Fixes: https://tracker.ceph.com/issues/74620
Signed-off-by: David Galloway <david.galloway@ibm.com>

testnode: Support for yum_repos_to_remove

A follow up commit will put specific repo files we want back in place.

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #813 from ceph/retry-mirrorlists

testnode: Try to work around flaky rocky mirrorlists

testnode: Try to work around flaky rocky mirrorlists

```
[root@trial052 ~]# cat /etc/yum.repos.d/rocky.repo

[baseos]
name=Rocky Linux $releasever - BaseOS
mirrorlist=https://mirrors.rockylinux.org/mirrorlist?arch=$basearch&repo=BaseOS-$releasever$rltype
gpgcheck=1
enabled=1
countme=1
metadata_expire=6h
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Rocky-10
```

Fixes: https://tracker.ceph.com/issues/74563
Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #811 from ceph/wget-c9

testnode: Install wget on CentOS 9

testnode: Install wget on CentOS 9

We were doing this in kickstart via cobbler.

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #809 from ceph/fog-fixes

Fog fixes

Update the documentation in the README file to include some missing things and things that were added with the latest tasks.

Signed-off-by: Fernando <fernando.alcocer.ochoa@ibm.com>

Merge pull request #810 from ceph/https

testnode: More s/http/https

testnode: More s/http/https

Signed-off-by: David Galloway <david.galloway@ibm.com>

prep-fog-capture: systemd unit to generate host ssh keys

Ubuntu does not automatically regenerate SSH host keys after image-based
deployments. When keys are removed prior to FOG capture, sshd fails to
start on redeploy.

Add a one-shot systemd service to regenerate host keys on first boot and
disable itself after running.

Signed-off-by: David Galloway <david.galloway@ibm.com>

prep-fog-capture: install ntp if missing

Signed-off-by: David Galloway <david.galloway@ibm.com>

fog-server: Clone as fog user

Avoids 'dubious ownership' git error

Signed-off-by: David Galloway <david.galloway@ibm.com>

fog-server: primary branch is now 'stable'

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #808 from dmick/teuthology-conserver-client

teuthology: add conserver-client as package dependency

teuthology: add conserver-client as package dependency

left aside leap because I believe that code is dead and I don't
have systems to verify the package name

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #806 from deepssin/update_lab_extras

testnode: Switch lab-extras repository URLs to HTTPS

testnode: Switch lab-extras repository URLs to HTTPS

Signed-off-by: deepssin <deepssin@redhat.com>

Merge pull request #805 from ceph/zap-again

Refactor Zap again