--- /dev/null
+<<<<<<< HEAD
+# Builder Reimage Pipeline
+
+## Overview
+
+This pipeline is used to reimage Jenkins builder nodes and then perform post-reimage configuration using Ansible.
+
+It supports one or more target nodes and can process multiple nodes in parallel. The pipeline derives the target operating system from builder labels defined in the `ceph-sepia-secrets` inventory instead of relying on manually supplied OS input. After reimage, it prepares the execution environment, clones the required repositories, and runs a sequence of Ansible playbooks to configure the builder.
+
+The pipeline is designed so that it can be reproduced with the correct Jenkins configuration, required credentials, repository access, and workspace dependencies.
+
+---
+
+## Purpose
+
+The pipeline is responsible for the following:
+
+- reimage one or more Jenkins builder nodes using the MaaS-based reimage utility
+- detect the target OS from the source-of-truth inventory labels
+- prepare workspace-local dependencies
+- clone or update the required repositories
+- run post-reimage Ansible playbooks
+- support parallel execution for multiple nodes
+- capture and archive Ansible logs for troubleshooting
+
+---
+
+## Source of truth for OS detection
+
+The operating system is not passed manually as a Jenkins parameter.
+
+Instead, the pipeline reads:
+
+```text
+ceph-sepia-secrets/ansible/inventory/group_vars/jenkins_builders.yml
+```
+From that file, it reads the `jenkins_labels` mapping and extracts:
+
+- `installed-os-*` to determine the target operating system
+- Optional feature labels such as libvirt
+
+Example label string:
+```
+braggi noble small huge sepia x86_64 installed-os-noble libvirt
+```
+From the above, the pipeline derives:
+
+- OS = noble
+- libvirt = true
+
+## High level workflow
+For each node provided in the MACHINE parameter, the pipeline performs the following steps:
+
+1. Checkout the repository containing the pipeline and helper scripts
+2. Prepare the node-specific workspace and execution environment
+3. Clone or update the required repositories
+4. Read the builder inventory file and detect the OS and feature labels
+5. Reimage the node using the detected OS
+6. Run post-reimage Ansible playbooks
+7. Archive node-specific Ansible logs
+
+If multiple nodes are passed, each node is processed in parallel.
+
+## Jenkins job parameters
+
+### ***MACHINE***
+Comma-separated list of target builder short names.
+
+Examples:
+```
+braggi01
+braggi01,braggi16
+```
+
+### ***ACTION***
+Supported values:
+- reimage
+- reimage-all
+- wait_online
+
+Current execution primarily uses the reimage flow.
+
+### ***SKIP_REIMAGE***
+
+If set to true, the MaaS reimage step is skipped and only the post-reimage Ansible tasks are run.
+
+## Pipeline configuration
+
+***Jenkins pipeline***
+```
+builder-reimage/build/Jenkinsfile
+```
+Defines the pipeline stages, parallel execution, OS detection, reimage, Ansible execution, and log archiving.
+
+***Environment preparation***
+
+```
+builder-reimage/build/prepare_env.sh
+```
+Clones or updates required repositories, creates the virtual environment, installs Ansible if needed, and prepares workspace-local Ansible configuration.
+
+***Ansible runner***
+```
+builder-reimage/build/ansible_runner.sh
+```
+Runs the post-reimage Ansible playbooks, handles retries, writes logs, and prints the final execution summary.
+
+***OS and feature detection***
+```
+builder-reimage/build/get_node_os.py
+```
+Reads jenkins_builders.yml and extracts the installed OS and relevant labels such as libvirt.
+
+***Reimage utility***
+```
+builder-reimage/build/Jenkins_builder-reimage.py
+```
+Performs the MaaS-based release and deploy operations.
+
+## Jenkins credentials used
+
+The pipeline depends on the following Jenkins credentials.
+
+### ***maas-api-key***
+Used by the reimage utility to interact with MaaS.
+
+### ***builder-api-token***
+Passed to the builder playbook.
+
+### ***ansible-vault-pass***
+Used to unlock Ansible Vault protected data.
+
+### ***ansible-ssh-key***
+Used to clone and update private Git repositories over SSH.
+
+## Required repositories
+The pipeline uses these repositories:
+
+```
+git@github.com:ceph/ceph-build.git
+git@github.com:ceph/ceph-cm-ansible.git
+git@github.com:ceph/ceph-sepia-secrets.git
+```
+
+## Ansible playbooks executed
+The pipeline currently runs these playbooks in order:
+
+```
+ansible_managed.yml
+users.yml
+tools/jenkins-builder-disk.yml
+examples/builder.yml
+```
+
+ansible_runner.sh retries each playbook up to three times and prints a final summary at the end.
+
+## Reproducing this pipeline
+To reproduce this pipeline, the following are required:
+
+- Jenkins pipeline job configured with the JJB definition `builder-reimage/config/definitions/builder-reimage.yml`
+- Jenkins agent with the teuthology label
+- Jenkins credentials:
+
+ - maas-api-key
+ - builder-api-token
+ - ansible-vault-pass
+ - ansible-ssh-key
+
+
+- access to the required GitHub repositories
+- access to MaaS
+- Python 3, virtual environment support, Git, SSH, and Ansible installation capability on the Jenkins agent
+
+## Notes
+
+- OS selection is inventory-driven and no longer passed manually.
+- The pipeline supports parallel execution for multiple nodes.
+- Logs are written under ci-work-<node>/ansible-logs/ and archived by Jenkins.
+- Optional labels such as libvirt can be passed through to Ansible when needed.
+=======
+# Builder Reimage
+
+Builder Reimage is a lightweight automation utility for interacting with **MAAS (Metal as a Service)**.
+It allows you to list, query, check status, and redeploy machines efficiently, with or without specifying an operating system.
+
+This script simplifies MAAS machine management tasks and helps maintain consistent environment setups through a command-line interface.
+
+---
+
+## Features
+- List all managed MAAS machines
+- Query machine details
+- Check deployment or power status
+- Redeploy single or multiple machines
+- Redeploy with or without specifying an OS version
+
+---
+
+## Requirements
+- Python 3.8 or higher
+- MAAS CLI installed and configured
+- Valid MAAS API credentials
+
+---
+
+## Cloning the Repository
+
+To get started, clone the repository and navigate into the project directory:
+
+```bash
+git clone https://github.com/jitendrasahu1803/builder-reimage.git
+
+cd builder-reimage
+```
+
+## Setup your environment to use the tool
+
+Run bootstrap script:
+```bash
+bash bootstrap_env.sh
+```
+Active your virtual environment:
+```bash
+source .venv/bin/activate
+```
+Edit ```bash maas.conf``` file and update it with the correct MAAS URL (Sepia/Tucson):
+```bash
+cat maas.conf
+[maas]
+maas_url=http://<Enter_MAAS_URL>/MAAS
+```
+To setup your credential to connect with MAAS API, run (Replace <YOUR_MAAS_API_KEY> with your actual MAAS API key):
+```bash
+echo -n "<YOUR_MAAS_API_KEY>" | python3 -c 'import sys; from cryptography.fernet import Fernet; key=Fernet.generate_key(); open("maas_api.key","wb").write(key); data=sys.stdin.buffer.read(); open("maas_api_key.encrypted","wb").write(Fernet(key).encrypt(data)); print("wrote: maas_api.key and maas_api_key.encrypted")'
+```
+---
+
+## Usage
+Activate your virtual environment (if not already active), then run:
+```bash
+python3 maas-reimage.py [OPTIONS]
+```
+
+## Examples
+List all machines
+```bash
+--action list
+```
+Query details for one machine
+```bash
+--action query --machine node01
+```
+Check machine status
+```bash
+--action status --machine node01
+```
+Deploy a machine with a specific OS:
+```bash
+--action deploy --machine node01 --os 9.6
+```
+Reimage one machine (same OS)
+```bash
+--action reimage --machine node01
+```
+Reimage one machine with a specific OS
+```bash
+--action reimage --machine node01 --os jammy
+```
+Redeploy all machines
+```bash
+--action reimage-all
+```
+Redeploy all machines with a specific OS
+```bash
+--action reimage-all --os jammy
+```
+Find last deployed machine
+```bash
+--action last-deployed
+```
+
+## Additional option of (--owner)
+
+When you deploy or reimage a machine, it is automatically tagged with the default owner ```jitendra```. If you want to assign a different owner, you can specify it using the option shown below.
+
+Reimage a machine with a custom owner tag:
+```bash
+--action reimage --machine node01 --owner "<owner_name>"
+```
+
+This option works for deploy, reimage, reimage_all, and all other relevant commands.
+
+---
+
+## Author
+
+Jitendra Sahu
+
+GitHub: jitendrasahu1803
+
+## License
+
+This project is licensed under the MIT License.
+>>>>>>> 5c056f6f (Add builder reimage Jenkins pipelines and configs)
--- /dev/null
+#!/usr/bin/env python3
+"""
+MAAS Reimage Automation Script (Enhanced)
+
+Supports:
+- Single machine
+- Multiple machines: host1,host2
+- Per-machine OS: host1:jammy,host2:centos9
+- Mixed mode: host1,host2:jammy
+"""
+
+import asyncio
+import argparse
+import sys
+import time
+from datetime import datetime
+import os
+
+from maas.client import connect
+from aiohttp.client_exceptions import (
+ ClientConnectorError,
+ ServerDisconnectedError,
+ ClientResponseError,
+ ClientOSError
+)
+
+# -------------------------------------------------------------------------
+# Global Defaults
+# -------------------------------------------------------------------------
+LOG_FILE_DEFAULT = "maas_reimage.log"
+STATUS_WAIT_TIMEOUT = 1200
+DEFAULT_OWNER = "jitendra"
+
+MAAS_URL = "http://soko02.front.sepia.ceph.com:5240/MAAS"
+MAAS_API_KEY = os.environ.get("MAAS_API_KEY")
+
+if not MAAS_API_KEY:
+ print("Error: MAAS_API_KEY env variable not set")
+ sys.exit(1)
+
+# -------------------------------------------------------------------------
+# Helpers
+# -------------------------------------------------------------------------
+def log(msg, log_file=None):
+ print(msg)
+ if log_file:
+ with open(log_file, "a") as f:
+ f.write(f"{datetime.now().isoformat()} - {msg}\n")
+
+def get_normalized_status(machine):
+ return str(
+ getattr(machine, "status_name", "") or
+ getattr(getattr(machine, "status", None), "name", "")
+ ).lower()
+
+# NEW: parse machine + OS mapping
+def parse_machine_list(machine_arg, default_os=None):
+ result = []
+
+ items = [x.strip() for x in machine_arg.split(",") if x.strip()]
+
+ for item in items:
+ if ":" in item:
+ host, os_val = item.split(":", 1)
+ result.append((host.strip(), os_val.strip()))
+ else:
+ result.append((item.strip(), default_os))
+
+ return result
+
+# -------------------------------------------------------------------------
+# Connect
+# -------------------------------------------------------------------------
+async def connect_maas():
+ try:
+ return await connect(MAAS_URL, apikey=MAAS_API_KEY)
+ except Exception as e:
+ print(f"Connection failed: {e}")
+ sys.exit(1)
+
+async def get_cached_machines(client):
+ if getattr(client, "_machines_cache", None) is None:
+ client._machines_cache = await client.machines.list()
+ return client._machines_cache
+
+# -------------------------------------------------------------------------
+# Wait for status
+# -------------------------------------------------------------------------
+async def wait_for_status(client, system_id, expected):
+ expected = expected.lower()
+ start = time.time()
+
+ while time.time() - start < STATUS_WAIT_TIMEOUT:
+ m = await client.machines.get(system_id=system_id)
+ if get_normalized_status(m) == expected:
+ return True
+ await asyncio.sleep(5)
+
+ print(f"Timeout waiting for {system_id}")
+ return False
+
+# -------------------------------------------------------------------------
+# Deploy
+# -------------------------------------------------------------------------
+async def deploy_machine(client, hostname, os_release, log_file):
+ machines = await get_cached_machines(client)
+ m = next((x for x in machines if x.hostname == hostname), None)
+
+ if not m:
+ log(f"[ERROR] {hostname} not found", log_file)
+ return False
+
+ m = await client.machines.get(system_id=m.system_id)
+
+ status = get_normalized_status(m)
+
+ if status == "deployed":
+ log(f"[INFO] {hostname} already deployed", log_file)
+ return True
+
+ os_to_use = os_release or getattr(m, "distro_series", "focal")
+
+ log(f"[ACTION] Deploying {hostname} with {os_to_use}", log_file)
+
+ try:
+ await m.deploy(distro_series=os_to_use)
+ except Exception as e:
+ log(f"[ERROR] Deploy failed: {e}", log_file)
+ return False
+
+ if await wait_for_status(client, m.system_id, "deployed"):
+ log(f"[SUCCESS] {hostname} deployed", log_file)
+ return True
+
+ return False
+
+# -------------------------------------------------------------------------
+# Release
+# -------------------------------------------------------------------------
+async def release_machine(machine):
+ if get_normalized_status(machine) == "deployed":
+ await machine.release()
+
+# -------------------------------------------------------------------------
+# Reimage
+# -------------------------------------------------------------------------
+async def reimage_machine(client, hostname, os_release, log_file):
+ machines = await get_cached_machines(client)
+ m = next((x for x in machines if x.hostname == hostname), None)
+
+ if not m:
+ log(f"[ERROR] {hostname} not found", log_file)
+ return False
+
+ m = await client.machines.get(system_id=m.system_id)
+
+ status = get_normalized_status(m)
+
+ if status == "deployed":
+ log(f"[ACTION] Releasing {hostname}", log_file)
+ await release_machine(m)
+ if not await wait_for_status(client, m.system_id, "ready"):
+ return False
+
+ return await deploy_machine(client, hostname, os_release, log_file)
+
+# -------------------------------------------------------------------------
+# Main
+# -------------------------------------------------------------------------
+async def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--action", required=True)
+ parser.add_argument("--machine")
+ parser.add_argument("--os")
+ parser.add_argument("--log-file")
+
+ args = parser.parse_args()
+ log_file = args.log_file or LOG_FILE_DEFAULT
+
+ client = await connect_maas()
+ client._machines_cache = await client.machines.list()
+
+ # UPDATED: multi-machine support
+ if args.action == "reimage":
+ if not args.machine:
+ print("Missing --machine")
+ return
+
+ machine_list = parse_machine_list(args.machine, args.os)
+
+ success = 0
+ failed = 0
+
+ for hostname, os_val in machine_list:
+ log(f"\n[START] {hostname} (OS={os_val})", log_file)
+ try:
+ result = await reimage_machine(client, hostname, os_val, log_file)
+ if result:
+ success += 1
+ else:
+ failed += 1
+ except Exception as e:
+ log(f"[ERROR] {hostname}: {e}", log_file)
+ failed += 1
+
+ log("\n===== SUMMARY =====", log_file)
+ log(f"Total: {len(machine_list)}", log_file)
+ log(f"Success: {success}", log_file)
+ log(f"Failed: {failed}", log_file)
+
+# -------------------------------------------------------------------------
+
+if __name__ == "__main__":
+ try:
+ asyncio.run(main())
+ except KeyboardInterrupt:
+ print("Interrupted by user.")
+ except Exception as e:
+ print(f"Fatal error: {e}")
--- /dev/null
+pipeline {
+ agent { label 'teuthology' }
+
+ environment {
+ VENV_DIR = ".venv"
+ DOMAIN = "front.sepia.ceph.com"
+ }
+
+ parameters {
+ booleanParam(name: 'SKIP_REIMAGE', defaultValue: false)
+ choice(name: 'ACTION', choices: ['reimage','reimage-all','wait_online'])
+ string(
+ name: 'MACHINE',
+ defaultValue: '',
+ description: 'node1 or node1,node2'
+ )
+ }
+
+ stages {
+
+ stage('Parallel Execution') {
+ steps {
+ script {
+ if (!params.MACHINE?.trim()) {
+ error("MACHINE parameter is required")
+ }
+
+ def machines = params.MACHINE.split(',').collect { it.trim() }
+ def branches = [:]
+
+ for (int i = 0; i < machines.size(); i++) {
+ def machine = machines[i]
+
+ branches["Node-${machine}"] = {
+ node('teuthology') {
+
+ stage("Checkout-${machine}") {
+ checkout scm
+ }
+
+ def WORK_DIR = "${env.WORKSPACE}/ci-work-${machine}"
+ def TARGET_FQDN = "${machine}.${env.DOMAIN}"
+ def INVENTORY_FILE = "${WORK_DIR}/repos/secret-repo/ansible/inventory/group_vars/jenkins_builders.yml"
+
+ stage("Prepare-${machine}") {
+ withCredentials([
+ sshUserPrivateKey(credentialsId: 'ansible-ssh-key', keyFileVariable: 'SSH_KEY')
+ ]) {
+ sh """#!/bin/bash
+ set -euo pipefail
+
+ mkdir -p "${WORK_DIR}"
+
+ cp "${SSH_KEY}" /tmp/jenkins_git_key
+ chmod 600 /tmp/jenkins_git_key
+
+ bash builder-reimage/build/prepare_env.sh \
+ "${TARGET_FQDN}" \
+ "${WORK_DIR}" \
+ true \
+ git@github.com:ceph/ceph-build.git \
+ git@github.com:ceph/ceph-cm-ansible.git \
+ git@github.com:ceph/ceph-sepia-secrets.git
+ """
+ }
+ }
+
+ stage("Sync Labels-${machine}") {
+ withCredentials([
+ usernamePassword(
+ credentialsId: 'jenkins-api-token',
+ usernameVariable: 'JOB_BUILDER_USER',
+ passwordVariable: 'JOB_BUILDER_PASS'
+ )
+ ]) {
+ withEnv([
+ "SYNC_MACHINE=${machine}",
+ "SYNC_TARGET_FQDN=${TARGET_FQDN}",
+ "SYNC_INVENTORY_FILE=${INVENTORY_FILE}"
+ ]) {
+ sh '''#!/bin/bash
+ set -euo pipefail
+
+ echo "[INFO] Syncing Jenkins labels for $SYNC_MACHINE"
+ echo "JENKINS_URL is: $JENKINS_URL"
+
+ python3 builder-reimage/build/sync_labels.py \
+ --file "$SYNC_INVENTORY_FILE" \
+ --jenkins_url "$JENKINS_URL" \
+ --user "$JOB_BUILDER_USER" \
+ --token "$JOB_BUILDER_PASS" \
+ --node "$SYNC_TARGET_FQDN"
+ '''
+ }
+ }
+ }
+ stage("Detect OS-${machine}") {
+ def DETECTED_OS = sh(
+ script: """#!/bin/bash
+ python3 builder-reimage/build/get_node_os.py \
+ "${INVENTORY_FILE}" \
+ "${machine}"
+ """,
+ returnStdout: true
+ ).trim()
+
+ echo "Detected OS for ${machine} is ${DETECTED_OS}"
+
+ if (!DETECTED_OS) {
+ error("OS detection failed for ${machine}")
+ }
+
+ def parts = DETECTED_OS.split(",")
+
+ env.DETECTED_OS = parts[0]
+ env.LIBVIRT_ENABLED = parts.size() > 1 ? parts[1] : "false"
+
+ echo "Detected OS: ${env.DETECTED_OS}"
+ echo "Libvirt enabled: ${env.LIBVIRT_ENABLED}"
+
+ }
+
+ stage("Reimage-${machine}") {
+ if (!params.SKIP_REIMAGE) {
+ withCredentials([string(credentialsId: 'maas-api-key', variable: 'MAAS_API_KEY')]) {
+ sh """#!/bin/bash
+ set -euo pipefail
+
+ REPO_DIR="\$WORKSPACE/builder-reimage/build"
+ VENV_PATH="\$REPO_DIR/.venv"
+
+ cd "\$REPO_DIR"
+
+ if [ ! -d "\$VENV_PATH" ]; then
+ echo "Creating virtual environment"
+ python3 -m venv "\$VENV_PATH"
+ source "\$VENV_PATH/bin/activate"
+ pip install --upgrade pip
+ pip install -r requirements.txt
+ else
+ source "\$VENV_PATH/bin/activate"
+ fi
+
+ echo "Starting reimage for ${machine} with OS ${env.DETECTED_OS}"
+
+ python Jenkins_builder-reimage.py \
+ --action reimage \
+ --machine ${machine} \
+ --os ${env.DETECTED_OS}
+ """
+ }
+ }
+ }
+
+ stage("Ansible-${machine}") {
+ withCredentials([
+ string(credentialsId: 'builder-api-token', variable: 'BUILDER_TOKEN'),
+ string(credentialsId: 'ansible-vault-pass', variable: 'VAULT_PASS'),
+ sshUserPrivateKey(credentialsId: 'ansible-ssh-key', keyFileVariable: 'SSH_KEY')
+ ]) {
+
+ sh """#!/bin/bash
+ set -euo pipefail
+
+ REPO_DIR="\$WORKSPACE/builder-reimage"
+ BUILD_DIR="\$REPO_DIR/build"
+ WORK_DIR="${WORK_DIR}"
+ TARGET_FQDN="${TARGET_FQDN}"
+
+ echo "[INFO] Running Ansible for \$TARGET_FQDN"
+
+ mkdir -p "\$WORK_DIR"
+
+ cp "${SSH_KEY}" /tmp/jenkins_git_key
+ chmod 600 /tmp/jenkins_git_key
+
+ export GIT_SSH_COMMAND="ssh -i /tmp/jenkins_git_key -o StrictHostKeyChecking=no"
+
+ VAULT_FILE="/home/jenkins-build/.vault_pass.txt"
+ echo "\$VAULT_PASS" > "\$VAULT_FILE"
+
+ cd "\$BUILD_DIR"
+
+ bash prepare_env.sh "\$TARGET_FQDN" "\$WORK_DIR" true \
+ git@github.com:ceph/ceph-build.git \
+ git@github.com:ceph/ceph-cm-ansible.git \
+ git@github.com:ceph/ceph-sepia-secrets.git
+
+ bash ansible_runner.sh \
+ "\$TARGET_FQDN" \
+ "\$WORK_DIR" \
+ "${env.VENV_DIR}" \
+ "${env.DETECTED_OS}" \
+ "\$BUILDER_TOKEN" \
+ "${env.LIBVIRT_ENABLED}"
+ """
+
+ archiveArtifacts artifacts: "ci-work-${machine}/ansible-logs/*.log", allowEmptyArchive: true
+ }
+ }
+ }
+ }
+ }
+
+ branches.failFast = false
+ parallel branches
+ }
+ }
+ }
+ }
+
+ post {
+ always {
+ sh 'rm -f /home/jenkins-build/.vault_pass.txt || true'
+ echo 'Build finished'
+ }
+
+ failure {
+ echo 'One or more nodes failed'
+ }
+
+ success {
+ echo 'All nodes completed successfully'
+ }
+ }
+}
--- /dev/null
+#!/usr/bin/env bash
+set -euo pipefail
+
+# ansible_runner.sh
+#
+# Usage:
+# ansible_runner.sh <target_fqdn> <work_dir> <venv_dir> <os> <builder_token> <libvirt_flag>
+#
+# libvirt_flag:
+# "true" -> enable libvirt-specific configuration
+# "false" -> default behavior
+
+
+# Use default Ansible output format
+export ANSIBLE_STDOUT_CALLBACK=default
+
+# Disable creation of retry files
+export ANSIBLE_RETRY_FILES_ENABLED=False
+
+# Fully qualified domain name of the target node
+TARGET_FQDN="$1"
+
+# Per-node workspace directory used for execution
+WORK_DIR="$2"
+
+# Virtual environment directory name
+VENV_DIR="$3"
+
+# Detected OS value (converted to lowercase)
+OS_VALUE="${4,,}"
+
+# Token used by builder playbook for authentication
+BUILDER_TOKEN="$5"
+
+# Libvirt flag passed from pipeline (normalized to true/false)
+LIBVIRT=$(echo "${6:-false}" | tr '[:upper:]' '[:lower:]' | xargs)
+
+# Debug logs for libvirt flag
+echo "[ansible_runner][DEBUG] LIBVIRT raw value: '${6:-unset}'"
+echo "[ansible_runner][DEBUG] LIBVIRT normalized: '${LIBVIRT}'"
+
+# Path to Ansible repository (playbooks, roles)
+ANSIBLE_DIR="${WORK_DIR}/repos/ansible"
+
+# Path to main repository containing builder playbook
+MAIN_DIR="${WORK_DIR}/repos/main"
+
+# Path to store log files
+LOG_DIR="${WORK_DIR}/ansible-logs"
+
+# failure tracking
+FAILED_PLAYBOOKS=()
+FAILED_LOGS=()
+
+
+# Ensure secrets path exists (independent of prepare_env.sh)
+
+SECRETS_PATH="${WORK_DIR}/repos/secret-repo/ansible/secrets"
+
+# Backward compatibility for playbook expecting this env var
+export ANSIBLE_SECRETS_PATH="${ANSIBLE_SECRETS_PATH:-${SECRETS_PATH}}"
+
+echo "[ansible_runner] ANSIBLE_SECRETS_PATH=${ANSIBLE_SECRETS_PATH}"
+
+
+# consume paths exported by prepare_env.sh
+INVENTORY_PATH="${INVENTORY_PATH:-${WORK_DIR}/repos/secret-repo/ansible/inventory}"
+SECRETS_PATH="${SECRETS_PATH:-${WORK_DIR}/repos/secret-repo/ansible/secrets}"
+
+# Vault comes from this file (created by Jenkinsfile)
+VAULT_FILE="/home/jenkins-build/.vault_pass.txt"
+VAULT_ARG="--vault-password-file=${VAULT_FILE}"
+
+mkdir -p "${LOG_DIR}"
+
+# Activate virtualenv if present
+if [[ -f "${WORK_DIR}/${VENV_DIR}/bin/activate" ]]; then
+ echo "[ansible_runner] Activating venv: ${WORK_DIR}/${VENV_DIR}"
+ # shellcheck disable=SC1090
+ source "${WORK_DIR}/${VENV_DIR}/bin/activate"
+else
+ echo "[ansible_runner] WARNING: venv not found at ${WORK_DIR}/${VENV_DIR}. Continuing without venv."
+fi
+
+# Determine SSH user based on OS
+if [[ "$OS_VALUE" =~ (rhel|centos|rocky|almai|9-stream|rhel10|centos70|8) ]]; then
+ SSH_USER="cloud-user"
+else
+ SSH_USER="ubuntu"
+fi
+
+echo "[ansible_runner] SSH user selected = ${SSH_USER}"
+echo "[ansible_runner] Using inventory = ${INVENTORY_PATH}"
+echo "[ansible_runner] Using secrets = ${SECRETS_PATH}"
+
+# Admin user JSON builder
+ADMIN_USERS=(
+ "akraitma"
+ "dgalloway"
+ "dmick"
+ "falcocer"
+ "jitendra"
+ "zack"
+)
+
+build_admin_users_json() {
+ local json='{"managed_admin_users":['
+ for u in "${ADMIN_USERS[@]}"; do
+ json+="{\"name\":\"${u}\"},"
+ done
+ json="${json%,}]}"
+ echo "${json}"
+}
+
+# Playbook runner with retries (3 attempts)
+run_playbook() {
+ local pb_name="$1"
+ local cmd="$2"
+ local logfile="${LOG_DIR}/${TARGET_FQDN}-${pb_name}.log"
+
+ echo "[ansible_runner] Running ${pb_name} -> ${logfile}"
+
+ attempts=0
+ max_attempts=3
+
+ until [[ $attempts -ge $max_attempts ]]; do
+ attempts=$((attempts + 1))
+
+ eval "${cmd}" 2>&1 | tee "${logfile}"
+ rc=${PIPESTATUS[0]}
+
+ if [[ $rc -eq 0 ]]; then
+ echo "[ansible_runner] ${pb_name} succeeded on attempt ${attempts}"
+ return 0
+ fi
+
+ echo "[ansible_runner] ${pb_name} failed (rc=${rc}), attempt ${attempts}/${max_attempts}"
+
+ if [[ $attempts -lt $max_attempts ]]; then
+ sleep $((attempts * 10))
+ else
+ echo "[ansible_runner] Max attempts reached for ${pb_name}"
+
+ # NEW: Track failure instead of exiting
+ FAILED_PLAYBOOKS+=("${pb_name}")
+ FAILED_LOGS+=("${logfile}")
+
+ return 1
+ fi
+ done
+}
+
+##############################################
+# PLAYBOOK 1 — ansible_managed.yml
+##############################################
+(
+ cd "${ANSIBLE_DIR}"
+
+ CMD="ANSIBLE_HOST_KEY_CHECKING=False ANSIBLE_STDOUT_CALLBACK=json ansible-playbook ansible_managed.yml \
+ -i '${INVENTORY_PATH}' \
+ --limit='${TARGET_FQDN}' \
+ -e ansible_ssh_user='${SSH_USER}'"
+
+ run_playbook "play1-ansible_managed" "${CMD}" || true
+)
+
+##############################################
+# PLAYBOOK 2 — users.yml
+##############################################
+(
+ cd "${ANSIBLE_DIR}"
+
+ ADMIN_USERS_JSON="$(build_admin_users_json)"
+
+ CMD="ANSIBLE_HOST_KEY_CHECKING=False ANSIBLE_STDOUT_CALLBACK=json ansible-playbook users.yml -v \
+ -i '${INVENTORY_PATH}' \
+ --limit='${TARGET_FQDN}' \
+ --tags='user,pubkeys' \
+ --extra-vars='${ADMIN_USERS_JSON}'"
+
+ run_playbook "play2-users" "${CMD}" || true
+)
+
+##############################################
+# PLAYBOOK 3 — jenkins-builder-disk.yml
+##############################################
+(
+ cd "${ANSIBLE_DIR}"
+
+ CMD="ANSIBLE_HOST_KEY_CHECKING=False ANSIBLE_STDOUT_CALLBACK=json ansible-playbook -v tools/jenkins-builder-disk.yml \
+ -i '${INVENTORY_PATH}' \
+ --limit='${TARGET_FQDN}'"
+
+ run_playbook "play3-tools-disk" "${CMD}" || true
+)
+
+##############################################
+# PLAYBOOK 4 — builder.yml (USES VAULT)
+##############################################
+EXTRA_VARS=""
+
+if [[ "${LIBVIRT}" == "true" ]]; then
+ echo "[ansible_runner] libvirt detected for ${TARGET_FQDN}"
+ EXTRA_VARS="-e libvirt=true"
+fi
+
+(
+ cd "${MAIN_DIR}/ansible"
+
+ CMD="ANSIBLE_HOST_KEY_CHECKING=False ANSIBLE_STDOUT_CALLBACK=json ansible-playbook -v \
+ -i '${INVENTORY_PATH}' \
+ ${VAULT_ARG} \
+ -M ./library/ \
+ examples/builder.yml \
+ -e '{\"token\":\"${BUILDER_TOKEN}\", \"jenkins_credentials_uuid\":\"jenkins-build\", \"api_uri\":\"https://jenkins.ceph.com\"}' \
+ -e permanent=true \
+ ${EXTRA_VARS} \
+ --limit='${TARGET_FQDN}'"
+
+ echo "[ansible_runner][DEBUG] EXTRA_VARS=${EXTRA_VARS}"
+ echo "[ansible_runner][DEBUG] FINAL CMD=${CMD}"
+
+ run_playbook "play4-main-builder" "${CMD}" || true
+)
+
+echo "========================================"
+echo "[ansible_runner] FINAL EXECUTION SUMMARY"
+echo "========================================"
+
+if [[ ${#FAILED_PLAYBOOKS[@]} -eq 0 ]]; then
+ echo "All playbooks completed successfully for ${TARGET_FQDN}"
+ exit 0
+fi
+
+echo "Failed playbooks detected:"
+
+for i in "${!FAILED_PLAYBOOKS[@]}"; do
+ echo "----------------------------------------"
+ echo "Playbook : ${FAILED_PLAYBOOKS[$i]}"
+ echo "Log file : ${FAILED_LOGS[$i]}"
+ echo "Extracting failure details..."
+
+ # Extract Ansible failures
+ grep -E "FAILED!|fatal:" "${FAILED_LOGS[$i]}" || echo "No detailed failure lines found"
+done
+
+echo "========================================"
+echo "Build FAILED due to above errors"
+echo "========================================"
+
+exit 1
--- /dev/null
+#!/usr/bin/env bash
+set -e
+
+# -----------------------------------------------------------------------------
+# Script Name: bootstrap_env.sh
+# Description:
+# Bootstraps the local Python environment for running the MAAS reimage script.
+# - Installs MAAS CLI (Linux/macOS)
+# - Creates Python virtual environment
+# - Installs dependencies
+# -----------------------------------------------------------------------------
+
+echo "Starting environment setup..."
+
+# -----------------------------------------------------------------------------
+# Step 1: Install MAAS CLI based on OS
+# -----------------------------------------------------------------------------
+if [[ "$OSTYPE" == "linux-gnu"* ]]; then
+ # Detect Linux distribution
+ if [ -f /etc/os-release ]; then
+ . /etc/os-release
+ DISTRO=$ID
+ else
+ DISTRO="unknown"
+ fi
+
+ echo "Detected Linux distribution: $DISTRO"
+
+ case "$DISTRO" in
+ ubuntu|debian)
+ echo "Installing MAAS CLI (Debian/Ubuntu)..."
+ sudo apt update -y
+ sudo apt install -y maas-cli
+ ;;
+ rhel|centos|fedora|rocky|almalinux)
+ echo "Installing MAAS CLI (RHEL/CentOS/Fedora)..."
+ # Ensure EPEL repo is enabled for dependencies
+ if command -v dnf >/dev/null 2>&1; then
+ sudo dnf install -y epel-release || true
+ sudo dnf install -y snapd
+ else
+ sudo yum install -y epel-release || true
+ sudo yum install -y snapd
+ fi
+
+ # Enable and use snap to install MAAS CLI
+ sudo systemctl enable --now snapd.socket
+ sudo ln -sf /var/lib/snapd/snap /snap
+ echo "Installing MAAS CLI via snap..."
+ sudo snap install maas --classic
+ ;;
+ *)
+ echo "Unsupported Linux distribution: $DISTRO"
+ echo "Please install MAAS CLI manually."
+ exit 1
+ ;;
+ esac
+
+elif [[ "$OSTYPE" == "darwin"* ]]; then
+ echo "Installing MAAS CLI (macOS)..."
+ brew install maas || true
+else
+ echo "Unsupported OS type: $OSTYPE"
+ exit 1
+fi
+
+# -----------------------------------------------------------------------------
+# Step 2: Create and activate Python virtual environment
+# -----------------------------------------------------------------------------
+echo "Creating Python virtual environment..."
+python3 -m venv .venv
+source .venv/bin/activate
+
+echo "Installing Python dependencies..."
+pip install --upgrade pip
+pip install -r requirements.txt
+
+# -----------------------------------------------------------------------------
+# Step 3: Check for encrypted secrets
+# -----------------------------------------------------------------------------
+if [[ ! -f "maas_api.key" || ! -f "maas_api_key.encrypted" ]]; then
+ echo "Missing 'maas_api.key' or 'maas_api_key.encrypted'. Ensure both files are present."
+ exit 1
+fi
+
+echo
+echo "Setup complete."
--- /dev/null
+#!/usr/bin/env python3
+
+"""
+get_node_os.py
+
+Description:
+------------
+This script reads the Jenkins builders inventory file and extracts:
+ - The target operating system from the "installed-os-*" label
+ - Whether the "libvirt" label is present
+
+It returns the result in the format:
+ <os_name>,<libvirt_flag>
+
+Where:
+ os_name -> e.g. jammy, centos9
+ libvirt_flag -> "true" or "false"
+
+This output is consumed by the Jenkins pipeline to:
+ - determine which OS to use for MaaS reimage
+ - optionally enable libvirt-specific configuration in Ansible
+
+Usage:
+------
+ get_node_os.py <inventory_file> <node_name>
+"""
+
+import yaml
+import sys
+
+inventory_file = sys.argv[1]
+node_input = sys.argv[2]
+
+node_name = node_input if "." in node_input else node_input + ".front.sepia.ceph.com"
+
+with open(inventory_file) as f:
+ data = yaml.safe_load(f)
+
+labels_map = data.get("jenkins_labels", {})
+labels_str = labels_map.get(node_name, "")
+
+labels = labels_str.split()
+
+os_name = ""
+libvirt = "false"
+
+for label in labels:
+ if label.startswith("installed-os-"):
+ os_name = label.replace("installed-os-", "")
+ if label == "libvirt":
+ libvirt = "true"
+
+# Print both values in structured way
+print(f"{os_name},{libvirt}")
--- /dev/null
+[maas]
+maas_url=http://soko02.front.sepia.ceph.com:5240/MAAS
--- /dev/null
+#!/usr/bin/env bash
+set -euo pipefail
+
+# prepare_env.sh
+# Usage:
+# prepare_env.sh <target_fqdn> <work_dir> <ssh_available> <ansible_repo> <main_repo> <secrets_repo>
+
+TARGET_FQDN="$1"
+WORK_DIR="$2"
+SSH_AVAILABLE="${3:-false}"
+MAIN_REPO="${4:-git@github.com:ceph/ceph-build.git}"
+ANSIBLE_REPO="${5:-git@github.com:ceph/ceph-cm-ansible.git}"
+SECRETS_REPO="${6:-git@github.com:ceph/ceph-sepia-secrets.git}"
+
+SHORTNAME="${TARGET_FQDN%%.*}"
+
+REPOS_DIR="${WORK_DIR}/repos"
+ANSIBLE_DIR="${REPOS_DIR}/ansible"
+MAIN_DIR="${REPOS_DIR}/main"
+SECRETS_DIR="${REPOS_DIR}/secret-repo"
+VENV_DIR="${WORK_DIR}/.venv"
+
+mkdir -p "${REPOS_DIR}"
+cd "${REPOS_DIR}"
+
+log() { echo "[prepare_env] $*"; }
+
+adjust_url() {
+ local url="$1"
+ if [ "${SSH_AVAILABLE}" = "true" ]; then
+ echo "${url}"
+ else
+ echo "${url}" | sed 's|git@github.com:|https://github.com/|'
+ fi
+}
+
+clone_repo() {
+ local url="$1"
+ local dir="$2"
+ if [ -d "${dir}/.git" ]; then
+ log "Updating existing repo ${dir}"
+ (cd "${dir}" && git fetch --all --prune)
+ else
+ log "Cloning ${url} -> ${dir}"
+ if [ "${SSH_AVAILABLE}" = "true" ] && [ -f /tmp/jenkins_git_key ]; then
+ GIT_SSH_COMMAND='ssh -i /tmp/jenkins_git_key -o StrictHostKeyChecking=no' git clone --depth 1 "${url}" "${dir}"
+ else
+ git clone --depth 1 "${url}" "${dir}"
+ fi
+ fi
+}
+
+ANSIBLE_URL=$(adjust_url "${ANSIBLE_REPO}")
+MAIN_URL=$(adjust_url "${MAIN_REPO}")
+SECRETS_URL=$(adjust_url "${SECRETS_REPO}")
+
+clone_repo "${ANSIBLE_URL}" "${ANSIBLE_DIR}"
+clone_repo "${MAIN_URL}" "${MAIN_DIR}"
+clone_repo "${SECRETS_URL}" "${SECRETS_DIR}"
+
+# Ensure venv exists
+if [ ! -d "${VENV_DIR}/bin" ]; then
+ log "Virtualenv not found at ${VENV_DIR}, creating..."
+ python3 -m venv "${VENV_DIR}"
+ source "${VENV_DIR}/bin/activate"
+ pip install --upgrade pip
+ pip install -r "${ANSIBLE_DIR}/requirements.txt" 2>/dev/null || true
+else
+ source "${VENV_DIR}/bin/activate"
+fi
+
+# Check ansible installation
+if command -v ansible-playbook >/dev/null 2>&1; then
+ log "Ansible already installed."
+else
+ log "Ansible not found, attempting to install into venv..."
+
+ install_ansible_system() {
+ if command -v apt-get >/dev/null 2>&1; then
+ log "Detected apt-based system. Installing ansible..."
+ sudo apt-get update && sudo apt-get install -y ansible || true
+
+ elif command -v dnf >/dev/null 2>&1; then
+ log "Detected dnf-based system. Installing ansible-core..."
+ sudo dnf install -y ansible-core || sudo dnf install -y ansible || true
+
+ elif command -v yum >/dev/null 2>&1; then
+ log "Detected yum-based system. Installing ansible..."
+ sudo yum install -y ansible || true
+
+ else
+ log "No supported package manager found. Cannot install ansible."
+ fi
+ }
+
+ if [ -n "${VENV_DIR}" ] && [ -f "${VENV_DIR}/bin/activate" ]; then
+ pip install ansible || {
+ log "Failed to install ansible in venv; trying system install (requires sudo)"
+ install_ansible_system
+ }
+ else
+ log "No venv available; attempting system install (requires sudo)"
+ install_ansible_system
+ fi
+fi
+
+# -------------------------------------------------------------------
+# NEW: Workspace-local ansible configuration (replaces /etc/ansible)
+# -------------------------------------------------------------------
+
+INVENTORY_PATH="${SECRETS_DIR}/ansible/inventory"
+SECRETS_PATH="${SECRETS_DIR}/ansible/secrets"
+ANSIBLE_CFG="${WORK_DIR}/ansible.cfg"
+
+# Validate inventory presence
+if [ ! -e "${INVENTORY_PATH}" ]; then
+ log "ERROR: Inventory not found at ${INVENTORY_PATH}"
+ exit 1
+fi
+
+# Create ansible.cfg in workspace
+cat > "${ANSIBLE_CFG}" <<EOF
+[defaults]
+inventory = ${INVENTORY_PATH}
+host_key_checking = False
+retry_files_enabled = False
+stdout_callback = yaml
+interpreter_python = auto
+roles_path = ${ANSIBLE_DIR}/roles
+EOF
+
+export ANSIBLE_CONFIG="${ANSIBLE_CFG}"
+export INVENTORY_PATH
+export SECRETS_PATH
+export ANSIBLE_SECRETS_PATH="${SECRETS_PATH}"
+
+log "Using ANSIBLE_CONFIG=${ANSIBLE_CONFIG}"
+log "Inventory path: ${INVENTORY_PATH}"
+log "Secrets path: ${SECRETS_PATH}"
+
+# Optional: Warn if target not present in inventory
+if ! grep -R -q -E "^${TARGET_FQDN}\\b" "${INVENTORY_PATH}" 2>/dev/null; then
+ log "Warning: ${TARGET_FQDN} not found in inventory. Ensure dynamic inventory or update inventory."
+fi
+
+log "prepare_env completed for ${TARGET_FQDN} (shortname=${SHORTNAME})"
--- /dev/null
+requests>=2.31.0
+aiohttp>=3.9.0
+asyncio>=3.4.3
+python-dotenv>=1.0.1
+tabulate>=0.9.0
+cryptography>=36.0.0
+maas>=1.3.2
+python-libmaas>=0.6.8
+setuptools>=65.0.0
--- /dev/null
+#!/usr/bin/env python3
+
+"""
+Sync Jenkins node labels with jenkins_builders.yml
+
+- Reads labels from inventory
+- Resolves actual Jenkins node name (handles IP+hostname format)
+- Fetches current labels from Jenkins
+- Updates only if mismatch
+"""
+
+import yaml
+import requests
+import argparse
+from urllib.parse import quote
+import xml.etree.ElementTree as ET
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--file", required=True)
+parser.add_argument("--jenkins_url", required=True)
+parser.add_argument("--user", required=True)
+parser.add_argument("--token", required=True)
+parser.add_argument("--node", required=True)
+
+args = parser.parse_args()
+
+# --- Normalize base URL ---
+base_url = args.jenkins_url.rstrip('/')
+
+# --- Read inventory ---
+with open(args.file) as f:
+ data = yaml.safe_load(f)
+
+labels_map = data.get("jenkins_labels", {})
+inventory_labels = labels_map.get(args.node, "").strip()
+
+if not inventory_labels:
+ print(f"[WARN] No labels found in inventory for {args.node}")
+ exit(0)
+
+# Normalize labels (sorted for consistent comparison)
+inventory_labels = " ".join(sorted(inventory_labels.split()))
+
+# --- Resolve Jenkins node name dynamically ---
+short_name = args.node.split('.')[0]
+
+api_url = f"{base_url}/computer/api/json"
+r = requests.get(api_url, auth=(args.user, args.token))
+r.raise_for_status()
+
+nodes = r.json().get("computer", [])
+
+node_name = None
+for n in nodes:
+ display_name = n.get("displayName", "")
+ if short_name in display_name:
+ node_name = display_name
+ break
+
+if not node_name:
+ print(f"[ERROR] Could not find Jenkins node for {short_name}")
+ exit(1)
+
+# Encode node name (important for '+' → '%2B')
+encoded_node = quote(node_name, safe='')
+
+# --- Build config.xml URL ---
+url = f"{base_url}/computer/{encoded_node}/config.xml"
+
+print(f"[DEBUG] Matched Jenkins node: {node_name}")
+print(f"[DEBUG] URL: {url}")
+
+# --- Fetch Jenkins config.xml ---
+r = requests.get(url, auth=(args.user, args.token))
+r.raise_for_status()
+
+xml_data = r.text
+
+# Strip XML declaration if needed (handles XML 1.1 safely)
+if xml_data.startswith("<?xml"):
+ xml_data = xml_data.split("?>", 1)[1]
+
+root = ET.fromstring(xml_data)
+
+label_node = root.find("label")
+
+if label_node is None:
+ print(f"[ERROR] <label> tag not found for {node_name}")
+ exit(1)
+
+current_labels = label_node.text or ""
+current_labels = " ".join(sorted(current_labels.split()))
+
+# --- Compare ---
+if current_labels == inventory_labels:
+ print(f"[SYNC] {args.node}: labels already in sync")
+ exit(0)
+
+# --- Update labels ---
+print(f"[SYNC] Updating {args.node}")
+print(f" OLD: {current_labels}")
+print(f" NEW: {inventory_labels}")
+
+label_node.text = inventory_labels
+
+updated_xml = ET.tostring(root, encoding="unicode")
+
+# --- Push update ---
+resp = requests.post(
+ url,
+ data=updated_xml,
+ headers={"Content-Type": "application/xml"},
+ auth=(args.user, args.token)
+)
+
+resp.raise_for_status()
+
+print(f"[SYNC] {args.node} labels updated successfully")
--- /dev/null
+- job:
+ name: builder-reimage
+ project-type: pipeline
+ description: "Pipeline to reimage Jenkins builders and run post-reimage ansible tasks (OS auto-detected from labels)"
+
+ concurrent: false
+
+ parameters:
+ - choice:
+ name: ACTION
+ choices:
+ - reimage
+ - reimage-all
+ - wait_online (debug)
+ default: reimage
+ description: "Reimage action"
+
+ - string:
+ name: MACHINE
+ default: ""
+ description: |
+ Target node(s)
+ Examples:
+ node1
+ node1,node2
+
+ - bool:
+ name: SKIP_REIMAGE
+ default: false
+ description: "Skip reimage and only run post-reimage tasks"
+
+ scm:
+ - git:
+ url: https://github.com/ceph/ceph-build.git
+ branches:
+ - main
+ clean: true
+
+ pipeline-scm:
+ scm:
+ - git:
+ url: https://github.com/ceph/ceph-build.git
+ branches:
+ - main
+ script-path: builder-reimage/build/Jenkinsfile
+ lightweight-checkout: true
projects / ceph-build.git / commitdiff