feat(dependencies): upgrade supported cluster components (#681)

* fix(calico): support split CRDs for current releases

- Download the v1_crd_projectcalico_org.yaml bundle before the operator
- Apply both files with server-side apply and force-conflicts per the
  upstream upgrade procedure
- Wait for the operator Deployment and for the managed CRDs to be
  Established after the operator starts
- Replace the create/rescue/replace flow with an idempotent apply that
  no longer conceals partial failures
- Verify TigeraStatus for calico and apiserver is Available, not just
  that Pods exist

* feat(dependencies): upgrade supported cluster components

- Bump K3s to v1.36.2+k3s1, Calico to v3.32.1, Cilium to v1.20.0,
  kube-vip to v1.2.2, kube-vip cloud provider to v0.0.12, and MetalLB to
  v0.16.0 across sample inventory, role defaults, and argument specs
- Pin the Cilium CLI with a new cilium_cli_tag (v0.19.7) instead of the
  floating stable.txt lookup
- Replace the CiliumBGPPeeringPolicy v2alpha1 BGP template with the
  v2 CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement,
  and CiliumLoadBalancerIPPool resource set
- Move Cilium load balancer Helm keys from bpf.loadBalancer to the valid
  top-level loadBalancer path
- Add preflight schema validation and remove the deprecated policy after
  the v2 objects are accepted
- Wait for cilium status after installation
- Pin kube-vip RBAC in a repository template instead of fetching a
  mutable URL, and include EndpointSlice permissions
- Fix the kube-vip bgppeers format to address:ASN comma-separated peers
- Fail clearly when the MetalLB speaker tag replacement does not apply
- Drop the obsolete MetalLB webhook service name version branch

* test(molecule): verify upgraded cluster components

- Assert every node reports the expected K3s kubelet version
- Verify the active CNI (Flannel / Calico / Cilium) is Ready and runs
  the expected image tag, including Calico TigeraStatus Available
- Verify the active load balancer (MetalLB / kube-vip) runs the expected
  image tags and that MetalLB is absent when kube-vip is active
- Assert no Flannel DaemonSet remains when Calico or Cilium is enabled
- Assert the example LoadBalancer address falls inside the configured
  pool range
- Add a manifest-only Cilium BGP regression test that renders the v2
  template with zero, one, and multiple neighbors and rejects any v2alpha1
  or CiliumBGPPeeringPolicy output

* fix(dependencies): correct dependency version pins

- Set the sample kube-vip image to v1.2.2 and repair the damaged comment
- Pin the kube-vip cloud provider default to v0.0.12 in the task URL
- Set the MetalLB controller argument-spec default to v0.16.0
- Restore the MetalLB available timeout default to 240s

* docs(dependencies): document current cluster versions

- Update kube-vip, kube-vip cloud provider, and MetalLB defaults
- Add cilium_tag and cilium_cli_tag rows
- Explain that MetalLB v0.16.0 is the application image target even though
  a newer chart-only tag (metallb-chart-0.16.1) exists
- Add an existing-cluster upgrade warning covering the K3s etcd 3.5.26
  bridge and one-minor-at-a-time rule, consecutive Cilium minor upgrades,
  Calico v3 resource UID handling, and MetalLB app vs chart tags

* fix(dependencies): address PR review findings

- Read the MetalLB speaker tag check from the managed host with slurp
  instead of a controller-side file lookup, and match the full image
  reference
- Restore the tigera-operator namespace on the Calico operator Deployment
  wait while keeping the managed CRD waits cluster-scoped
- Make Molecule verify inputs durable and scenario-specific via a
  per-scenario verify-vars.yml, driven by explicit verify_cni/verify_lb
  values instead of non-persisted converge facts
- Rename the kube-vip multi-peer BGP env var from bgppeers to bgp_peers
  and vip_cidr to vip_subnet so v1.2.2 actually reads them
- Map the legacy Cilium routed mode to tunnel and stop passing the alias
  directly to the chart
- Use return-code based failed_when on apply and preflight commands so
  non-error failures are no longer treated as success
- Clarify the sequential K3s upgrade path and backups in the README
- Add kube-vip and MetalLB regression tests and a Cilium mode mapping unit

* fix(dependencies): resolve re-review findings

- correct the Calico TigeraStatus resource kind\n- document tunnel as the supported Cilium routing mode\n- validate load balancer addresses across range and CIDR pools

* fix(molecule): verify embedded flannel instead of a flannel DaemonSet

- K3s 1.36 runs flannel embedded in the k3s agent rather than as a
  kube-flannel-ds DaemonSet, so the flannel verifier queried a workload
  that no longer exists and failed the verify step
- For the flannel scenarios, assert every node is Ready and that neither
  the Calico nor the Cilium namespace exists
- Drop the now-invalid kube-flannel-ds DaemonSet assertion

* fix(molecule): wait for the LoadBalancer address before asserting reachability

- The nginx LoadBalancer service had no ingress address when the
  reachability assertion ran, so status.loadBalancer.ingress[0].ip was
  undefined and the ipwrap filter failed during verify
- Poll the service until MetalLB or kube-vip assigns an external IP
- Record the assigned address once and reuse it for the reachability probe
  and the pool membership checks

* fix(ci): harden calico apiserver wait and extend molecule job timeout

- Bump calico system resources wait retries 30->60 and delay 7->10 so the
  slow-to-reconcile calico-apiserver deployment has enough time under nested-virt
- Raise the molecule step timeout-minutes from 90 to 150 to accommodate
  contended 5-node scenarios (cilium, kube-vip) that were hitting the 90-min cap

* fix(calico): treat optional API server as best-effort on converge

- The Calico API server (calico-apiserver) is an optional add-on for managing
  Calico policy through the projectcalico.org/v3 Kubernetes API; it is not
  required for Calico CNI data plane operation
- With Calico v3.32.1 on K3s 1.36 the tigera-operator never provisions the
  calico-apiserver namespace, causing the converge wait to fail deterministically
- Keep the strict wait for core Calico components (typha, kube-controllers,
  calico-node, csi-node-driver) and make the API server wait tolerate failure
- Restrict the TigeraStatus Available check to the calico status, matching the
  upstream v3.32.1 K3s quickstart which validates without the API server

.ansible-lint

@@ -3,6 +3,7 @@ profile: production
 exclude_paths:
   # default paths
   - .cache/
+  - .ansible/
   - .github/
   - test/fixtures/formatting-before/
   - test/fixtures/formatting-prettier/
@@ -14,8 +15,17 @@ exclude_paths:
   - molecule/**/prepare.yml
   - molecule/**/reset.yml
 
+  # Scenario verify inputs are plain variable files, not playbooks. They are
+  # loaded as vars, not executed, so ansible-lint must not treat them as plays.
+  - molecule/**/verify-vars.yml
+
   # The file was generated by galaxy ansible - don't mess with it.
   - galaxy.yml
 
 skip_list:
   - var-naming[no-role-prefix]
+
+# The Molecule Vagrant driver injects this module at runtime. The custom create
+# playbook is syntax-checked separately against the exact pinned plugin module.
+mock_modules:
+  - vagrant

.github/ISSUE_TEMPLATE.md

@@ -1,5 +1,5 @@
 
-<!-- It's a good idea to check this post first for general troubleshooting https://github.com/techno-tim/k3s-ansible/discussions/19   -->
+<!-- It's a good idea to check this post first for general troubleshooting https://github.com/timothystewart6/k3s-ansible/discussions/19   -->
 
 <!--- Provide a general summary of the issue in the Title above -->
 
@@ -82,4 +82,4 @@ node
 ## Possible Solution
 <!--- Not obligatory, but suggest a fix/reason for the bug, -->
 
-- [ ] I've checked the [General Troubleshooting Guide](https://github.com/techno-tim/k3s-ansible/discussions/20)
+- [ ] I've checked the [General Troubleshooting Guide](https://github.com/timothystewart6/k3s-ansible/discussions/20)

.github/actionlint.yaml

@@ -0,0 +1,6 @@
+---
+self-hosted-runner:
+  labels:
+    - k3s-ci
+    - virtualbox
+    - nested-virt

.github/download-boxes.sh

@@ -1,42 +1,91 @@
 #!/bin/bash
 
 # download-boxes.sh
-# Check all molecule.yml files for required Vagrant boxes and download the ones that are not
-# already present on the system.
+# Validate the pinned Vagrant box set and download exact versions that are not
+# already present in VAGRANT_HOME.
 
 set -euo pipefail
 
 GIT_ROOT=$(git rev-parse --show-toplevel)
 PROVIDER=virtualbox
+LOCK_FILE="${VAGRANT_BOX_LOCK_FILE:-${GIT_ROOT}/.github/vagrant-boxes.lock}"
 
-yq --version
+MOLECULE_YML_PATH=("${GIT_ROOT}"/molecule/*/molecule.yml)
 
-# Define the path to the molecule.yml files
-MOLECULE_YML_PATH="${GIT_ROOT}/molecule/*/molecule.yml"
-
-# Extract and sort unique boxes from all molecule.yml files
-all_boxes=$(for file in $MOLECULE_YML_PATH; do
-    yq eval '.platforms[].box' "$file"
+# Extract the unique boxes referenced by the scenarios.
+declared_boxes=$(for file in "${MOLECULE_YML_PATH[@]}"; do
+    yq -r '.platforms[].box' "$file"
 done | sort -u)
 
-echo all_boxes: "$all_boxes"
+if [[ ! -r "$LOCK_FILE" ]]; then
+    printf 'Vagrant box lock file is missing or unreadable: %s\n' "$LOCK_FILE" >&2
+    exit 1
+fi
 
-# Read the boxes that are currently present on the system (for the current provider)
+lock_entries=$(awk '
+    /^[[:space:]]*#/ || NF == 0 { next }
+    NF != 3 {
+        printf "Invalid lock entry on line %d: expected box, version, architecture\n", NR > "/dev/stderr"
+        invalid = 1
+        next
+    }
+    { print $1 " " $2 " " $3 }
+    END { exit invalid }
+' "$LOCK_FILE")
+
+duplicate_boxes=$(printf '%s\n' "$lock_entries" | awk '{ print $1 }' | sort | uniq -d)
+if [[ -n "$duplicate_boxes" ]]; then
+    printf 'Duplicate Vagrant box lock entries:\n%s\n' "$duplicate_boxes" >&2
+    exit 1
+fi
+
+locked_boxes=$(printf '%s\n' "$lock_entries" | sort)
+locked_names=$(printf '%s\n' "$locked_boxes" | awk '{ print $1 }')
+missing_locks=$(comm -23 <(printf '%s\n' "$declared_boxes") <(printf '%s\n' "$locked_names"))
+unused_locks=$(comm -13 <(printf '%s\n' "$declared_boxes") <(printf '%s\n' "$locked_names"))
+
+if [[ -n "$missing_locks" || -n "$unused_locks" ]]; then
+    if [[ -n "$missing_locks" ]]; then
+        printf 'Scenario boxes missing from the lock file:\n%s\n' "$missing_locks" >&2
+    fi
+    if [[ -n "$unused_locks" ]]; then
+        printf 'Lock entries not referenced by a scenario:\n%s\n' "$unused_locks" >&2
+    fi
+    exit 1
+fi
+
+printf 'Pinned Vagrant boxes:\n%s\n' "$locked_boxes"
+
+# Read exact box, provider, version, and architecture tuples already present.
 present_boxes=$(
-    (vagrant box list |
-        grep "${PROVIDER}" | # Filter by boxes available for the current provider
-        awk '{print $1;}' |  # The box name is the first word in each line
-        sort |
-        uniq) ||
-        echo "" # In case any of these commands fails, just use an empty list
+    vagrant box list --machine-readable |
+        awk -F, -v expected_provider="$PROVIDER" '
+            $3 == "box-name" { name = $4; next }
+            $3 == "box-provider" { provider = $4; next }
+            $3 == "box-version" { version = $4; next }
+            $3 == "box-architecture" {
+                architecture = $4
+                if (provider == expected_provider) {
+                    print name " " version " " architecture
+                }
+                name = provider = version = architecture = ""
+            }
+        ' |
+        sort -u
 )
 
-# The boxes that we need to download are the ones present in $all_boxes, but not $present_boxes.
-download_boxes=$(comm -2 -3 <(echo "${all_boxes}") <(echo "${present_boxes}"))
+download_boxes=$(comm -23 \
+    <(printf '%s\n' "$locked_boxes") \
+    <(printf '%s\n' "$present_boxes"))
 
-# Actually download the necessary boxes
-if [ -n "${download_boxes}" ]; then
-    echo "${download_boxes}" | while IFS= read -r box; do
-        vagrant box add --provider "${PROVIDER}" "${box}"
+if [[ -n "$download_boxes" ]]; then
+    printf '%s\n' "$download_boxes" | while read -r box version architecture; do
+        vagrant box add \
+            --provider "$PROVIDER" \
+            --box-version "$version" \
+            --architecture "$architecture" \
+            "$box"
     done
+else
+    printf 'All pinned Vagrant boxes are already present.\n'
 fi

.github/scripts/cleanup-runner-resources.sh

@@ -0,0 +1,249 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+usage() {
+  printf '%s\n' \
+    'Usage: cleanup-runner-resources.sh [--snapshot|--dry-run|--apply]' \
+    '' \
+    'Discover and, with --apply, remove only VirtualBox resources referenced by' \
+    'repository-owned Molecule Vagrant state. The default is --dry-run.'
+}
+
+mode="dry-run"
+case "${1:-}" in
+  "") ;;
+  --snapshot) mode="snapshot" ;;
+  --dry-run) mode="dry-run" ;;
+  --apply) mode="apply" ;;
+  --help|-h) usage; exit 0 ;;
+  *) usage >&2; exit 2 ;;
+esac
+
+home_dir="${HOME:?HOME must be set}"
+molecule_root="${K3S_CI_MOLECULE_ROOT:-${home_dir}/.cache/molecule}"
+repository_name="${K3S_CI_MOLECULE_PROJECT:-k3s-ansible}"
+virtualbox_root="${K3S_CI_VIRTUALBOX_ROOT:-${home_dir}/VirtualBox VMs}"
+hostonly_marker="${K3S_CI_HOSTONLY_MARKER:-${home_dir}/.cache/k3s-ci/hostonly-interfaces}"
+
+record_hostonly() {
+  local marker_dir="${hostonly_marker%/*}"
+  local marker_tmp="${hostonly_marker}.tmp"
+  local hostonly_inventory
+  if ! hostonly_inventory="$(VBoxManage list hostonlyifs)"; then
+    fail_closed 'unable to inventory VirtualBox host-only interfaces'
+  fi
+  mkdir -p -- "$marker_dir"
+  awk -F': ' '
+    /^Name:/ { name=$2 }
+    /^IPAddress:/ { print name "|" $2 }
+  ' <<< "$hostonly_inventory" > "$marker_tmp"
+  mv -- "$marker_tmp" "$hostonly_marker"
+  chmod 600 "$hostonly_marker"
+  printf 'Recorded host-only interface baseline: %s\n' "$hostonly_marker"
+}
+
+cleanup_hostonly() {
+  local hostonly_inventory
+  if [[ ! -f "$hostonly_marker" ]]; then
+    printf 'No host-only interface baseline found; leaving interfaces unchanged.\n'
+    return 0
+  fi
+
+  if ! hostonly_inventory="$(VBoxManage list hostonlyifs)"; then
+    fail_closed 'unable to inventory VirtualBox host-only interfaces'
+  fi
+
+  while IFS='|' read -r interface_name interface_ip; do
+    [[ "$interface_name" == vboxnet* ]] || continue
+    [[ "$interface_ip" == 192.168.30.* || "$interface_ip" == fdad:bad:ba55:* ]] || continue
+    if grep -Fqx "${interface_name}|${interface_ip}" "$hostonly_marker"; then
+      continue
+    fi
+    if [[ "$mode" == apply ]]; then
+      VBoxManage hostonlyif remove "$interface_name"
+      printf 'Removed host-only interface %s (%s)\n' "$interface_name" "$interface_ip"
+    else
+      printf 'Would remove host-only interface %s (%s)\n' "$interface_name" "$interface_ip"
+    fi
+  done < <(awk -F': ' '
+    /^Name:/ { name=$2 }
+    /^IPAddress:/ { print name "|" $2 }
+  ' <<< "$hostonly_inventory")
+}
+
+if [[ "$mode" == snapshot ]]; then
+  record_hostonly
+  exit 0
+fi
+
+resolve_existing_dir() {
+  local candidate="$1"
+  if [[ ! -d "$candidate" ]]; then
+    return 1
+  fi
+  readlink -f -- "$candidate"
+}
+
+root_contains() {
+  local root="$1"
+  local path="$2"
+  [[ "$path" == "$root"/* ]]
+}
+
+is_supported_scenario() {
+  case "$1" in
+    default|single_node|calico|cilium|kube-vip|ipv6) return 0 ;;
+    *) return 1 ;;
+  esac
+}
+
+is_unregistered_vm_error() {
+  grep -Eq 'Could not find a registered machine|VBOX_E_OBJECT_NOT_FOUND'
+}
+
+fail_closed() {
+  printf 'cleanup refused: %s\n' "$1" >&2
+  exit 3
+}
+
+if [[ ! "$repository_name" =~ ^[A-Za-z0-9._-]+$ ]]; then
+  fail_closed 'invalid Molecule repository name'
+fi
+
+print_inventory() {
+  local phase="$1"
+  printf '%s VirtualBox inventory:\n' "$phase"
+  VBoxManage list vms || true
+  VBoxManage list hdds || true
+  VBoxManage list hostonlyifs || true
+}
+
+molecule_root_real="$(resolve_existing_dir "$molecule_root" || true)"
+if [[ -z "$molecule_root_real" ]]; then
+  printf 'No Molecule root exists: %s\n' "$molecule_root"
+  cleanup_hostonly
+  exit 0
+fi
+
+print_inventory before
+
+repository_root_real="$(resolve_existing_dir "$molecule_root_real/$repository_name" || true)"
+if [[ -z "$repository_root_real" ]]; then
+  printf 'No repository Molecule state root exists: %s\n' "$molecule_root_real/$repository_name"
+  cleanup_hostonly
+  print_inventory after
+  exit 0
+fi
+if ! root_contains "$molecule_root_real" "$repository_root_real"; then
+  fail_closed "repository Molecule state root is outside Molecule root: $repository_root_real"
+fi
+
+declare -a state_files=()
+while IFS= read -r -d '' state_file; do
+  state_files+=("$state_file")
+done < <(find "$repository_root_real" -mindepth 6 -maxdepth 6 -type f \
+  -path '*/.vagrant/machines/*/virtualbox/id' -print0 2>/dev/null)
+
+if ((${#state_files[@]} == 0)); then
+  printf 'No repository-owned Molecule Vagrant state found under %s\n' "$repository_root_real"
+  cleanup_hostonly
+  print_inventory after
+  exit 0
+fi
+
+virtualbox_root_real="$(resolve_existing_dir "$virtualbox_root" || true)"
+
+declare -a vm_records=()
+for state_file in "${state_files[@]}"; do
+  if [[ ! -f "$state_file" ]]; then
+    printf 'Skipping Vagrant state removed with its stale scenario directory: %s\n' "$state_file"
+    continue
+  fi
+  state_file_real="$(readlink -f -- "$state_file")"
+  state_dir="${state_file_real%/.vagrant/machines/*/virtualbox/id}"
+  machine_dir="${state_file_real%/virtualbox/id}"
+  machine_name="${machine_dir##*/}"
+  scenario_name="${state_dir##*/}"
+
+  if ! root_contains "$repository_root_real" "$state_dir"; then
+    fail_closed "state path is outside the repository Molecule root: $state_file_real"
+  fi
+  if ! is_supported_scenario "$scenario_name"; then
+    fail_closed "unexpected Molecule scenario: $scenario_name"
+  fi
+  if [[ "$machine_name" != control* && "$machine_name" != node* ]]; then
+    fail_closed "unexpected Molecule machine name: $machine_name"
+  fi
+
+  vm_uuid="$(tr -d '[:space:]' < "$state_file_real")"
+  if [[ ! "$vm_uuid" =~ ^[0-9a-fA-F-]{36}$ ]]; then
+    fail_closed "invalid VirtualBox UUID in $state_file_real"
+  fi
+
+  if ! vm_info="$(VBoxManage showvminfo "$vm_uuid" --machinereadable 2>&1)"; then
+    if is_unregistered_vm_error <<< "$vm_info"; then
+      printf 'Stale Vagrant state without a registered VM: %s (%s)\n' "$machine_name" "$vm_uuid"
+      if [[ "$mode" == apply ]]; then
+        rm -rf -- "${state_dir}/.vagrant"
+        printf 'Removed stale Vagrant state: %s\n' "${state_dir}/.vagrant"
+      fi
+      continue
+    fi
+    fail_closed "unable to inspect VirtualBox VM $vm_uuid: $vm_info"
+  fi
+
+  if [[ -z "$virtualbox_root_real" ]]; then
+    fail_closed "VirtualBox VM root does not exist: $virtualbox_root"
+  fi
+
+  cfg_file="$(awk -F= '$1 == "CfgFile" {gsub(/"/, "", $2); print $2; exit}' <<< "$vm_info")"
+  if [[ -z "$cfg_file" ]]; then
+    fail_closed "VirtualBox configuration path missing for $vm_uuid"
+  fi
+  cfg_file_real="$(readlink -f -- "$cfg_file")"
+  if ! root_contains "$virtualbox_root_real" "$cfg_file_real"; then
+    fail_closed "VM configuration is outside VirtualBox root: $cfg_file_real"
+  fi
+
+  while IFS= read -r disk_path; do
+    [[ -z "$disk_path" ]] && continue
+    disk_path_real="$(readlink -f -- "$disk_path" 2>/dev/null || true)"
+    if [[ -z "$disk_path_real" ]] || ! root_contains "$virtualbox_root_real" "$disk_path_real"; then
+      fail_closed "attached disk is outside VirtualBox root: $disk_path"
+    fi
+  done < <(awk -F= '$1 ~ /^(SATA|IDE|SCSI|SAS|VirtioSCSI|NVMe)-[0-9]+-[0-9]+$/ {gsub(/"/, "", $2); print $2}' <<< "$vm_info")
+
+  vm_records+=("$vm_uuid|$machine_name|$cfg_file_real")
+done
+
+if ((${#vm_records[@]} == 0)); then
+  printf 'No live repository-owned VirtualBox resources found\n'
+  cleanup_hostonly
+  print_inventory after
+  exit 0
+fi
+
+for record in "${vm_records[@]}"; do
+  IFS='|' read -r vm_uuid machine_name cfg_file_real <<< "$record"
+  if [[ "$mode" == dry-run ]]; then
+    printf 'Would remove VM %s (%s) config=%s\n' "$machine_name" "$vm_uuid" "$cfg_file_real"
+    continue
+  fi
+
+  vm_state="$(VBoxManage showvminfo "$vm_uuid" --machinereadable | awk -F= '$1 == "VMState" {gsub(/"/, "", $2); print $2; exit}')"
+  if [[ "$vm_state" != poweroff && "$vm_state" != saved ]]; then
+    VBoxManage controlvm "$vm_uuid" poweroff
+  fi
+  VBoxManage unregistervm "$vm_uuid" --delete
+  printf 'Removed VM %s (%s)\n' "$machine_name" "$vm_uuid"
+done
+
+cleanup_hostonly
+print_inventory after
+
+if [[ "$mode" == apply ]]; then
+  printf 'Repository-owned VM and host-only interface cleanup complete.\n'
+else
+  printf 'Dry run complete. No resources were modified.\n'
+fi

.github/scripts/collect-runner-diagnostics.sh

@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+output_dir="${1:-${RUNNER_TEMP:-/tmp}/k3s-ci-diagnostics}"
+mkdir -p -- "$output_dir"
+umask 077
+
+run_capture() {
+  local output_file="$1"
+  shift
+  {
+    printf '$'
+    printf ' %q' "$@"
+    printf '\n'
+    "$@"
+  } > "$output_dir/$output_file" 2>&1 || true
+}
+
+run_capture system.txt uname -a
+run_capture runner-user.txt id
+run_capture memory.txt free -h
+run_capture disk.txt df -h
+run_capture virtualbox-version VBoxManage --version
+run_capture virtualbox-vms VBoxManage list vms
+run_capture virtualbox-running-vms VBoxManage list runningvms
+run_capture virtualbox-disks VBoxManage list hdds
+run_capture virtualbox-hostonlyifs VBoxManage list hostonlyifs
+run_capture virtualbox-groups VBoxManage list groups
+run_capture vagrant-status vagrant global-status
+run_capture molecule-state find "${HOME}/.cache/molecule" -maxdepth 6 -type f -path '*/.vagrant/machines/*/virtualbox/id' -print
+
+scenario_name="${K3S_CI_SCENARIO_NAME:-}"
+if [[ "$scenario_name" =~ ^[A-Za-z0-9_-]+$ ]]; then
+  molecule_state_dir="${HOME}/.cache/molecule/k3s-ansible/${scenario_name}"
+  for log_name in vagrant.out vagrant.err; do
+    if [[ -r "${molecule_state_dir}/${log_name}" ]]; then
+      cp -- "${molecule_state_dir}/${log_name}" "$output_dir/${scenario_name}-${log_name}"
+    fi
+  done
+fi
+
+if [[ -r /etc/vbox/networks.conf ]]; then
+  cp -- /etc/vbox/networks.conf "$output_dir/virtualbox-networks.conf"
+fi
+
+printf 'Diagnostics written to %s\n' "$output_dir"

.github/scripts/monitor-runner-resources.sh

@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+output_dir="${1:?output directory is required}"
+interval="${2:-10}"
+[[ "$interval" =~ ^[1-9][0-9]*$ ]] || {
+  printf 'monitor interval must be a positive integer\n' >&2
+  exit 2
+}
+
+mkdir -p -- "$output_dir"
+umask 077
+
+free -h > "$output_dir/memory-before.txt"
+df -h > "$output_dir/disk-before.txt"
+vmstat -w "$interval" > "$output_dir/vmstat.txt" &
+vmstat_pid=$!
+
+iostat_pid=""
+if command -v iostat >/dev/null 2>&1; then
+  iostat -dx "$interval" > "$output_dir/iostat.txt" &
+  iostat_pid=$!
+else
+  printf 'iostat is not installed on this runner\n' > "$output_dir/iostat-unavailable.txt"
+fi
+
+cleanup() {
+  local rc=$?
+  trap - EXIT INT TERM
+  kill "$vmstat_pid" 2>/dev/null || true
+  [[ -z "$iostat_pid" ]] || kill "$iostat_pid" 2>/dev/null || true
+  wait "$vmstat_pid" 2>/dev/null || true
+  [[ -z "$iostat_pid" ]] || wait "$iostat_pid" 2>/dev/null || true
+  free -h > "$output_dir/memory-after.txt"
+  df -h > "$output_dir/disk-after.txt"
+  exit "$rc"
+}
+trap cleanup EXIT INT TERM
+
+while :; do
+  sleep 3600 &
+  wait $!
+done

.github/scripts/prepare-vagrant-box-masters.sh

@@ -0,0 +1,211 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+fail() {
+  printf 'Vagrant box master preparation refused: %s\n' "$1" >&2
+  exit 3
+}
+
+root_contains() {
+  local root="$1"
+  local path="$2"
+  [[ "$path" == "$root"/* ]]
+}
+
+read_machine_value() {
+  local machine_info="$1"
+  local key="$2"
+  awk -F= -v key="$key" '$1 == key {gsub(/"/, "", $2); print $2; exit}' <<< "$machine_info"
+}
+
+read_extra_data() {
+  local uuid="$1"
+  local key="$2"
+  local value
+  value="$(VBoxManage getextradata "$uuid" "$key" 2>/dev/null || true)"
+  [[ "$value" == 'Value: '* ]] || return 1
+  printf '%s\n' "${value#Value: }"
+}
+
+validate_owned_master() {
+  local uuid="$1"
+  local box="$2"
+  local version="$3"
+  local architecture="$4"
+  local machine_info cfg_file cfg_file_real vm_state groups disk_path disk_path_real
+
+  [[ "$uuid" =~ ^[0-9a-fA-F-]{36}$ ]] || return 1
+  machine_info="$(VBoxManage showvminfo "$uuid" --machinereadable 2>/dev/null)" || return 1
+  vm_state="$(read_machine_value "$machine_info" VMState)"
+  groups="$(read_machine_value "$machine_info" groups)"
+  cfg_file="$(read_machine_value "$machine_info" CfgFile)"
+  [[ "$vm_state" == poweroff ]] || return 1
+  [[ ",$groups," == *,/k3s-ansible/box-masters,* ]] || return 1
+  [[ -n "$cfg_file" ]] || return 1
+  cfg_file_real="$(readlink -f -- "$cfg_file" 2>/dev/null || true)"
+  [[ -n "$cfg_file_real" ]] || return 1
+  root_contains "$virtualbox_root_real" "$cfg_file_real" || return 1
+
+  [[ "$(read_extra_data "$uuid" k3s-ansible/owner || true)" == box-master ]] || return 1
+  [[ "$(read_extra_data "$uuid" k3s-ansible/box || true)" == "$box" ]] || return 1
+  [[ "$(read_extra_data "$uuid" k3s-ansible/version || true)" == "$version" ]] || return 1
+  [[ "$(read_extra_data "$uuid" k3s-ansible/architecture || true)" == "$architecture" ]] || return 1
+
+  while IFS= read -r disk_path; do
+    [[ -z "$disk_path" || "$disk_path" == none ]] && continue
+    disk_path_real="$(readlink -f -- "$disk_path" 2>/dev/null || true)"
+    [[ -n "$disk_path_real" ]] || return 1
+    root_contains "$virtualbox_root_real" "$disk_path_real" || return 1
+  done < <(awk -F= '$1 ~ /^(SATA|IDE|SCSI|SAS|VirtioSCSI|NVMe)-[0-9]+-[0-9]+$/ {
+    gsub(/"/, "", $2); print $2
+  }' <<< "$machine_info")
+}
+
+write_prewarm_vagrantfile() {
+  local destination="$1"
+  local box="$2"
+  local version="$3"
+  {
+    printf '%s\n' "Vagrant.configure('2') do |config|"
+    printf '  config.vm.box = "%s"\n' "$box"
+    printf '  config.vm.box_version = "%s"\n' "$version"
+    printf '%s\n' \
+      '  config.vm.synced_folder ".", "/vagrant", disabled: true' \
+      '  config.vm.hostname = "k3s-ansible-box-prewarm"' \
+      '  config.vm.boot_timeout = 600' \
+      '  config.vm.provider "virtualbox" do |virtualbox|' \
+      '    virtualbox.linked_clone = true' \
+      '    virtualbox.memory = 1024' \
+      '    virtualbox.cpus = 2' \
+      '  end' \
+      'end'
+  } > "$destination"
+}
+
+cleanup_prewarm() {
+  local rc=$?
+  trap - EXIT
+  if [[ -n "${prewarm_dir:-}" && -d "$prewarm_dir" ]]; then
+    VAGRANT_CWD="$prewarm_dir" vagrant destroy --force >/dev/null 2>&1 || true
+    rm -rf -- "$prewarm_dir"
+  fi
+  exit "$rc"
+}
+
+create_owned_master() {
+  local box="$1"
+  local version="$2"
+  local architecture="$3"
+  local master_id_file="$4"
+  local mapping_file="$5"
+  local uuid machine_info cfg_file cfg_file_real vm_state mapping_tmp
+
+  # A master_id restored from an immutable cache is only a hint. Without the
+  # runner-local ownership record and matching VirtualBox metadata it is not
+  # trusted, adopted, modified, or deleted.
+  rm -f -- "$master_id_file"
+
+  prewarm_dir="$(mktemp -d "${master_root}/prewarm.XXXXXX")"
+  trap cleanup_prewarm EXIT
+  write_prewarm_vagrantfile "$prewarm_dir/Vagrantfile" "$box" "$version"
+
+  printf 'Creating runner-owned linked-clone master for %s %s %s\n' \
+    "$box" "$version" "$architecture"
+  VAGRANT_CWD="$prewarm_dir" vagrant up --provider virtualbox --no-provision
+
+  [[ -r "$master_id_file" ]] || fail "Vagrant did not record a master UUID for $box"
+  uuid="$(tr -d '[:space:]' < "$master_id_file")"
+  [[ "$uuid" =~ ^[0-9a-fA-F-]{36}$ ]] || fail "Vagrant recorded an invalid master UUID for $box"
+
+  machine_info="$(VBoxManage showvminfo "$uuid" --machinereadable 2>/dev/null)" || \
+    fail "Vagrant master $uuid for $box is not registered"
+  vm_state="$(read_machine_value "$machine_info" VMState)"
+  cfg_file="$(read_machine_value "$machine_info" CfgFile)"
+  cfg_file_real="$(readlink -f -- "$cfg_file" 2>/dev/null || true)"
+  [[ "$vm_state" == poweroff ]] || fail "new Vagrant master $uuid is not powered off"
+  if [[ -z "$cfg_file_real" ]] || ! root_contains "$virtualbox_root_real" "$cfg_file_real"; then
+    fail "new Vagrant master $uuid is outside the runner VirtualBox root"
+  fi
+
+  VAGRANT_CWD="$prewarm_dir" vagrant destroy --force
+  VBoxManage modifyvm "$uuid" --groups /k3s-ansible/box-masters
+  VBoxManage setextradata "$uuid" k3s-ansible/owner box-master
+  VBoxManage setextradata "$uuid" k3s-ansible/box "$box"
+  VBoxManage setextradata "$uuid" k3s-ansible/version "$version"
+  VBoxManage setextradata "$uuid" k3s-ansible/architecture "$architecture"
+  validate_owned_master "$uuid" "$box" "$version" "$architecture" || \
+    fail "new Vagrant master $uuid failed ownership validation"
+
+  rm -rf -- "$prewarm_dir"
+  prewarm_dir=""
+  trap - EXIT
+
+  mapping_tmp="${mapping_file}.tmp"
+  printf '%s\n' "$uuid" > "$mapping_tmp"
+  chmod 600 "$mapping_tmp"
+  mv -- "$mapping_tmp" "$mapping_file"
+  printf '%s\n' "$uuid" > "$master_id_file"
+  chmod 600 "$master_id_file"
+  printf 'Created and recorded owned master %s for %s\n' "$uuid" "$box"
+}
+
+repository_root="${K3S_CI_REPOSITORY_ROOT:-$(git rev-parse --show-toplevel)}"
+lock_file="${VAGRANT_BOX_LOCK_FILE:-${repository_root}/.github/vagrant-boxes.lock}"
+vagrant_home="${VAGRANT_HOME:?VAGRANT_HOME must be set}"
+master_root="${K3S_CI_VAGRANT_MASTER_ROOT:-${HOME:?HOME must be set}/.cache/k3s-ci/vagrant-masters}"
+virtualbox_root="${K3S_CI_VIRTUALBOX_ROOT:-${HOME}/VirtualBox VMs}"
+
+[[ -r "$lock_file" ]] || fail "box lock file is missing or unreadable: $lock_file"
+[[ -d "$vagrant_home/boxes" ]] || fail "Vagrant box directory is missing: $vagrant_home/boxes"
+[[ -d "$virtualbox_root" ]] || fail "VirtualBox root is missing: $virtualbox_root"
+
+box_root_real="$(readlink -f -- "$vagrant_home/boxes")"
+virtualbox_root_real="$(readlink -f -- "$virtualbox_root")"
+mkdir -p -- "$master_root"
+chmod 700 "$master_root"
+
+exec 9> "${master_root}/prepare.lock"
+flock 9
+
+lock_entries="$(awk '
+  /^[[:space:]]*#/ || NF == 0 { next }
+  NF != 3 { invalid = 1; next }
+  { print $1 " " $2 " " $3 }
+  END { exit invalid }
+' "$lock_file")" || fail 'invalid Vagrant box lock entry'
+[[ -n "$lock_entries" ]] || fail 'Vagrant box lock is empty'
+
+while read -r box version architecture; do
+  [[ "$box" =~ ^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$ ]] || fail "invalid box name: $box"
+  [[ "$version" =~ ^[A-Za-z0-9._-]+$ ]] || fail "invalid box version: $version"
+  [[ "$architecture" =~ ^[A-Za-z0-9._-]+$ ]] || fail "invalid box architecture: $architecture"
+
+  box_slug="${box//\//-VAGRANTSLASH-}"
+  record_slug="${box//\//_}-${version}-${architecture}"
+  box_dir="${vagrant_home}/boxes/${box_slug}/${version}/${architecture}/virtualbox"
+  [[ -d "$box_dir" ]] || fail "pinned box is not installed: $box $version $architecture"
+  box_dir_real="$(readlink -f -- "$box_dir")"
+  root_contains "$box_root_real" "$box_dir_real" || fail "box directory is outside VAGRANT_HOME: $box_dir_real"
+
+  master_id_file="${box_dir_real}/master_id"
+  mapping_file="${master_root}/${record_slug}.uuid"
+  uuid=""
+  if [[ -r "$mapping_file" ]]; then
+    uuid="$(tr -d '[:space:]' < "$mapping_file")"
+  fi
+
+  if [[ -n "$uuid" ]] && validate_owned_master "$uuid" "$box" "$version" "$architecture"; then
+    printf '%s\n' "$uuid" > "$master_id_file"
+    chmod 600 "$master_id_file"
+    printf 'Reusing owned master %s for %s %s %s\n' "$uuid" "$box" "$version" "$architecture"
+    continue
+  fi
+
+  if [[ -e "$mapping_file" ]]; then
+    printf 'Owned master record is stale for %s; rebuilding without deleting any VM or disk.\n' "$box"
+  fi
+  create_owned_master "$box" "$version" "$architecture" "$master_id_file" "$mapping_file"
+done <<< "$lock_entries"
+
+printf 'All pinned Vagrant box masters are ready.\n'

.github/scripts/test-cilium-bgp-manifest.py

@@ -0,0 +1,134 @@
+#!/usr/bin/env python3
+"""Render the Cilium BGP CRD template and assert it uses the v2 API.
+
+This is a manifest-only regression test used where no real BGP peer is
+available. It renders roles/k3s_server_post/templates/cilium.crs.j2 with
+zero, one, and multiple neighbors, then checks that the output:
+  - never contains CiliumBGPPeeringPolicy or cilium.io/v2alpha1
+  - emits the Cilium v2 BGP resources
+  - emits deterministic DNS-safe peer and instance names
+  - advertises Pod CIDRs only when cilium_exportPodCIDR is true
+"""
+
+from __future__ import print_function
+
+import os
+import subprocess
+
+from jinja2 import Environment, FileSystemLoader, StrictUndefined
+
+
+def repo_root():
+    return subprocess.check_output(
+        ["git", "rev-parse", "--show-toplevel"], text=True
+    ).strip()
+
+
+def fail(message):
+    raise SystemExit("Cilium BGP manifest test failed: " + message)
+
+
+def render(env, extra_vars):
+    base_vars = {
+        "cilium_bgp_my_asn": "64513",
+        "cilium_bgp_peer_asn": "64512",
+        "cilium_bgp_peer_address": "192.168.30.1",
+        "cilium_exportPodCIDR": True,
+        "cilium_bgp_lb_cidr": "192.168.31.0/24",
+    }
+    base_vars.update(extra_vars)
+    template = env.get_template("cilium.crs.j2")
+    return template.render(**base_vars)
+
+
+def check_common(output):
+    if "cilium.io/v2alpha1" in output:
+        fail("rendered output still contains cilium.io/v2alpha1")
+    if "kind: CiliumBGPPeeringPolicy" in output:
+        fail("rendered output still contains CiliumBGPPeeringPolicy")
+    for kind in (
+        "CiliumBGPPeerConfig",
+        "CiliumBGPClusterConfig",
+        "CiliumBGPAdvertisement",
+        "CiliumLoadBalancerIPPool",
+    ):
+        if ("kind: " + kind) not in output:
+            fail("rendered output is missing kind: " + kind)
+
+
+def main():
+    root = repo_root()
+    template_dir = os.path.join(
+        root, "roles", "k3s_server_post", "templates"
+    )
+    env = Environment(
+        loader=FileSystemLoader(template_dir), undefined=StrictUndefined
+    )
+
+    # Zero neighbors -> fall back to the single default peer.
+    output = render(env, {"_cilium_bgp_neighbors": []})
+    check_common(output)
+    if "peer-64512-1" not in output:
+        fail("default single peer name was not rendered")
+    if "peerAddress: 192.168.30.1" not in output:
+        fail("default peer address was not rendered")
+    if 'advertisementType: "PodCIDR"' not in output:
+        fail("PodCIDR advertisement missing when exportPodCIDR is true")
+
+    # One neighbor via the merged list.
+    output = render(
+        env,
+        {"_cilium_bgp_neighbors": [{"peer_address": "10.0.0.1", "peer_asn": "65001"}]},
+    )
+    check_common(output)
+    if "peer-65001-1" not in output:
+        fail("single merged peer name was not rendered")
+    if "peerAddress: 10.0.0.1" not in output:
+        fail("single merged peer address was not rendered")
+
+    # Multiple neighbors.
+    output = render(
+        env,
+        {
+            "_cilium_bgp_neighbors": [
+                {"peer_address": "10.0.0.1", "peer_asn": "65001"},
+                {"peer_address": "10.0.0.2", "peer_asn": "65002"},
+            ]
+        },
+    )
+    check_common(output)
+    if "peer-65001-1" not in output or "peer-65002-2" not in output:
+        fail("multiple merged peer names were not rendered")
+    if "peerAddress: 10.0.0.2" not in output:
+        fail("second merged peer address was not rendered")
+
+    # exportPodCIDR false -> no PodCIDR advertisement, service remains.
+    output = render(
+        env, {"_cilium_bgp_neighbors": [], "cilium_exportPodCIDR": False}
+    )
+    check_common(output)
+    if 'advertisementType: "PodCIDR"' in output:
+        fail("PodCIDR advertisement present when exportPodCIDR is false")
+    if 'advertisementType: "Service"' not in output:
+        fail("Service advertisement missing when exportPodCIDR is false")
+
+    # Load balancer pools: CIDR and start/stop forms.
+    output = render(env, {"_cilium_bgp_neighbors": []})
+    if "cidr: 192.168.31.0/24" not in output:
+        fail("CIDR load balancer pool was not rendered")
+    output = render(
+        env,
+        {
+            "_cilium_bgp_neighbors": [],
+            "cilium_bgp_lb_cidr": "192.168.31.80-192.168.31.90",
+        },
+    )
+    check_common(output)
+    if "start: 192.168.31.80" not in output or "stop: 192.168.31.90" not in output:
+        fail("start/stop load balancer pool was not rendered")
+
+    print("Cilium BGP manifest regression test passed")
+
+
+if __name__ == "__main__":
+    main()

.github/scripts/test-cleanup-runner-resources.sh

@@ -0,0 +1,129 @@
+#!/usr/bin/env bash
+
+# shellcheck disable=SC2016
+
+set -Eeuo pipefail
+
+repo_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)"
+test_root="$(mktemp -d "${TMPDIR:-/tmp}/k3s-ci-cleanup-test.XXXXXX")"
+trap 'rm -rf -- "$test_root"' EXIT
+
+molecule_root="$test_root/molecule"
+virtualbox_root="$test_root/VirtualBox VMs"
+fake_bin="$test_root/bin"
+mkdir -p -- "$molecule_root/k3s-ansible/single_node/.vagrant/machines/control1/virtualbox" \
+  "$molecule_root/k3s-ansible/single_node/.vagrant/machines/control2/virtualbox" \
+  "$virtualbox_root/control1" "$virtualbox_root/unmarked" "$fake_bin"
+printf '%s\n' '11111111-1111-1111-1111-111111111111' \
+  > "$molecule_root/k3s-ansible/single_node/.vagrant/machines/control1/virtualbox/id"
+printf '%s\n' '22222222-2222-2222-2222-222222222222' \
+  > "$molecule_root/k3s-ansible/single_node/.vagrant/machines/control2/virtualbox/id"
+touch "$virtualbox_root/control1/control1.vbox" "$virtualbox_root/control1/disk.vdi" \
+  "$virtualbox_root/unmarked/unmarked.vbox"
+
+printf '%s\n' \
+  '#!/usr/bin/env bash' \
+  'set -Eeuo pipefail' \
+  'case "${1:-}" in' \
+  '  list)' \
+  '    if [[ "${2:-}" == hostonlyifs && "${FAKE_HOSTONLY_FAIL:-false}" == true ]]; then exit 1; fi' \
+  '    exit 0' \
+  '    ;;' \
+  '  showvminfo)' \
+  '    if [[ "${FAKE_VM_MODE:-normal}" == missing ]]; then' \
+  '      printf '\''VBoxManage: error: Could not find a registered machine named "missing"\n'\'' >&2' \
+  '      exit 1' \
+  '    fi' \
+  '    if [[ "${FAKE_VM_MODE:-normal}" == fault ]]; then' \
+  '      printf '\''VBoxManage: error: VirtualBox service is unavailable\n'\'' >&2' \
+  '      exit 1' \
+  '    fi' \
+  '    printf '\''CfgFile="%s"\n'\'' "${FAKE_VBOX_ROOT}/control1/control1.vbox"' \
+  '    printf '\''SATA-0-0="%s"\n'\'' "${FAKE_VBOX_ROOT}/control1/disk.vdi"' \
+  '    printf '\''VMState="running"\n'\''' \
+  '    ;;' \
+  '  controlvm) printf '\''controlvm %s\n'\'' "$*" >> "${FAKE_LOG}" ;;' \
+  '  unregistervm)' \
+  '    printf '\''unregistervm %s\n'\'' "$*" >> "${FAKE_LOG}"' \
+  '    rm -f -- "${FAKE_VBOX_ROOT}/control1/control1.vbox" "${FAKE_VBOX_ROOT}/control1/disk.vdi"' \
+  '    ;;' \
+  '  *) : ;;' \
+  'esac' > "$fake_bin/VBoxManage"
+chmod 700 "$fake_bin/VBoxManage"
+
+output="$test_root/output.txt"
+if PATH="$fake_bin:$PATH" \
+  HOME="$test_root/home" \
+  K3S_CI_MOLECULE_ROOT="$molecule_root" \
+  K3S_CI_MOLECULE_PROJECT=k3s-ansible \
+  K3S_CI_VIRTUALBOX_ROOT="$virtualbox_root" \
+  K3S_CI_HOSTONLY_MARKER="$test_root/hostonly-baseline" \
+  FAKE_VM_MODE=fault \
+  FAKE_VBOX_ROOT="$virtualbox_root" \
+  FAKE_LOG="$test_root/vbox.log" \
+  bash "$repo_root/.github/scripts/cleanup-runner-resources.sh" --apply > "$output" 2>&1; then
+  printf '%s\n' 'cleanup unexpectedly accepted a VirtualBox inspection failure' >&2
+  exit 1
+fi
+grep -Fq 'cleanup refused: unable to inspect VirtualBox VM' "$output"
+[[ -f "$molecule_root/k3s-ansible/single_node/.vagrant/machines/control1/virtualbox/id" ]]
+
+printf '%s\n' 'vboxnet0|192.168.30.1' > "$test_root/hostonly-baseline"
+if PATH="$fake_bin:$PATH" \
+  HOME="$test_root/home" \
+  K3S_CI_MOLECULE_ROOT="$molecule_root" \
+  K3S_CI_MOLECULE_PROJECT=k3s-ansible \
+  K3S_CI_VIRTUALBOX_ROOT="$virtualbox_root" \
+  K3S_CI_HOSTONLY_MARKER="$test_root/hostonly-baseline" \
+  FAKE_HOSTONLY_FAIL=true \
+  FAKE_VBOX_ROOT="$virtualbox_root" \
+  FAKE_LOG="$test_root/vbox.log" \
+  bash "$repo_root/.github/scripts/cleanup-runner-resources.sh" --dry-run > "$output" 2>&1; then
+  printf '%s\n' 'cleanup unexpectedly accepted a host-only inventory failure' >&2
+  exit 1
+fi
+grep -Fq 'cleanup refused: unable to inventory VirtualBox host-only interfaces' "$output"
+
+PATH="$fake_bin:$PATH" \
+  HOME="$test_root/home" \
+  K3S_CI_MOLECULE_ROOT="$molecule_root" \
+  K3S_CI_MOLECULE_PROJECT=k3s-ansible \
+  K3S_CI_VIRTUALBOX_ROOT="$virtualbox_root" \
+  K3S_CI_HOSTONLY_MARKER="$test_root/hostonly-baseline" \
+  FAKE_VBOX_ROOT="$virtualbox_root" \
+  FAKE_LOG="$test_root/vbox.log" \
+  bash "$repo_root/.github/scripts/cleanup-runner-resources.sh" --dry-run > "$output"
+
+grep -Fq 'Would remove VM control1 (11111111-1111-1111-1111-111111111111)' "$output"
+[[ ! -e "$test_root/vbox.log" ]]
+
+PATH="$fake_bin:$PATH" \
+  HOME="$test_root/home" \
+  K3S_CI_MOLECULE_ROOT="$molecule_root" \
+  K3S_CI_MOLECULE_PROJECT=k3s-ansible \
+  K3S_CI_VIRTUALBOX_ROOT="$virtualbox_root" \
+  K3S_CI_HOSTONLY_MARKER="$test_root/hostonly-baseline" \
+  FAKE_VBOX_ROOT="$virtualbox_root" \
+  FAKE_LOG="$test_root/vbox.log" \
+  bash "$repo_root/.github/scripts/cleanup-runner-resources.sh" --apply > "$output"
+
+grep -Fq 'controlvm 11111111-1111-1111-1111-111111111111 poweroff' "$test_root/vbox.log"
+grep -Fq 'unregistervm unregistervm 11111111-1111-1111-1111-111111111111 --delete' "$test_root/vbox.log"
+[[ ! -e "$virtualbox_root/control1/control1.vbox" ]]
+[[ -e "$virtualbox_root/unmarked/unmarked.vbox" ]]
+
+PATH="$fake_bin:$PATH" \
+  HOME="$test_root/home" \
+  K3S_CI_MOLECULE_ROOT="$molecule_root" \
+  K3S_CI_MOLECULE_PROJECT=k3s-ansible \
+  K3S_CI_VIRTUALBOX_ROOT="$virtualbox_root" \
+  K3S_CI_HOSTONLY_MARKER="$test_root/hostonly-baseline" \
+  FAKE_VM_MODE=missing \
+  FAKE_VBOX_ROOT="$virtualbox_root" \
+  FAKE_LOG="$test_root/vbox.log" \
+  bash "$repo_root/.github/scripts/cleanup-runner-resources.sh" --apply > "$output"
+
+grep -Fq 'Stale Vagrant state without a registered VM' "$output"
+[[ ! -d "$molecule_root/k3s-ansible/single_node/.vagrant" ]]
+
+printf 'cleanup-runner-resources fixture test passed\n'

.github/scripts/test-fixtures/mock-flock

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+[[ "${1:-}" =~ ^[0-9]+$ ]]

.github/scripts/test-fixtures/mock-vagrant

@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+printf '%s\n' "$*" >> "$MOCK_VAGRANT_LOG"
+case "${1:-}" in
+  up)
+    if [[ -n "${MOCK_VAGRANTFILE_CAPTURE:-}" ]]; then
+      cp -- "$VAGRANT_CWD/Vagrantfile" "$MOCK_VAGRANTFILE_CAPTURE"
+    fi
+    counter_file="$MOCK_VBOX_STATE/counter"
+    counter=0
+    [[ ! -r "$counter_file" ]] || counter="$(cat "$counter_file")"
+    counter=$((counter + 1))
+    printf '%s\n' "$counter" > "$counter_file"
+    uuid="00000000-0000-4000-8000-$(printf '%012d' "$counter")"
+    vm_dir="$MOCK_VBOX_ROOT/master-$counter"
+    mkdir -p -- "$vm_dir"
+    : > "$vm_dir/master.vbox"
+    : > "$vm_dir/master.vdi"
+    printf '%s\n' \
+      'VMState="poweroff"' \
+      'groups="/"' \
+      "CfgFile=\"$vm_dir/master.vbox\"" \
+      "SATA-0-0=\"$vm_dir/master.vdi\"" > "$MOCK_VBOX_STATE/vm-$uuid"
+    printf '%s\n' "$uuid" > "$MOCK_BOX_DIR/master_id"
+    ;;
+  destroy) ;;
+  *) exit 2 ;;
+esac

.github/scripts/test-fixtures/mock-vboxmanage

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+printf '%s\n' "$*" >> "$MOCK_VBOX_LOG"
+command_name="${1:-}"
+uuid="${2:-}"
+
+case "$command_name" in
+  showvminfo)
+    [[ -r "$MOCK_VBOX_STATE/vm-$uuid" ]] || exit 1
+    cat "$MOCK_VBOX_STATE/vm-$uuid"
+    ;;
+  getextradata)
+    key_slug="${3//\//_}"
+    if [[ ! -r "$MOCK_VBOX_STATE/extra-$uuid-$key_slug" ]]; then
+      printf '%s\n' 'No value set!'
+      exit 0
+    fi
+    printf 'Value: '
+    cat "$MOCK_VBOX_STATE/extra-$uuid-$key_slug"
+    ;;
+  modifyvm)
+    [[ "${3:-}" == --groups ]]
+    awk -v groups="${4:-}" '
+      $1 !~ /^groups=/ { print }
+      END { printf "groups=\"%s\"\n", groups }
+    ' "$MOCK_VBOX_STATE/vm-$uuid" > "$MOCK_VBOX_STATE/vm-$uuid.tmp"
+    mv "$MOCK_VBOX_STATE/vm-$uuid.tmp" "$MOCK_VBOX_STATE/vm-$uuid"
+    ;;
+  setextradata)
+    key_slug="${3//\//_}"
+    printf '%s\n' "${4:-}" > "$MOCK_VBOX_STATE/extra-$uuid-$key_slug"
+    ;;
+  unregistervm|closemedium)
+    printf '%s\n' 'destructive VirtualBox command invoked' >&2
+    exit 99
+    ;;
+  *) exit 2 ;;
+esac

.github/scripts/test-k3s-server-bootstrap.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+repo_root="$(git rev-parse --show-toplevel)"
+main_tasks="$repo_root/roles/k3s_server/tasks/main.yml"
+join_tasks="$repo_root/roles/k3s_server/tasks/join_master.yml"
+
+for task_file in "$main_tasks" "$join_tasks"; do
+  for property in \
+    'Delegate=yes' \
+    'TasksMax=infinity' \
+    'KillMode=process' \
+    'LimitNOFILE=1048576' \
+    'LimitNPROC=infinity' \
+    'LimitCORE=infinity'; do
+    grep -Fq -- "$property" "$task_file" || {
+      printf '%s is missing transient K3s property %s\n' "$task_file" "$property" >&2
+      exit 1
+    }
+  done
+done
+
+if grep -Fq -- "node-role.kubernetes.io/master=true' -o=jsonpath" "$main_tasks"; then
+  printf 'control-plane registration still depends on the optional legacy master role key\n' >&2
+  exit 1
+fi
+
+grep -Fq -- "map('extract', hostvars, 'ansible_hostname')" "$main_tasks"
+grep -Fq -- 'difference(nodes.stdout.split())' "$main_tasks"
+grep -Fq -- 'crd/addons.k3s.cattle.io' "$main_tasks"
+grep -Fq -- 'crd/helmcharts.helm.cattle.io' "$main_tasks"
+grep -Fq -- 'crd/helmchartconfigs.helm.cattle.io' "$main_tasks"
+
+printf 'K3s transient bootstrap regression test passed\n'

.github/scripts/test-kube-vip-manifest.py

@@ -0,0 +1,130 @@
+#!/usr/bin/env python3
+"""Render the kube-vip DaemonSet template and assert env key correctness.
+
+kube-vip v1.2.2 reads `bgp_peers` and `vip_subnet`; it ignores the older
+`bgppeers` and `vip_cidr` names. This test proves the rendered manifest uses
+the keys the target image actually parses.
+"""
+
+from __future__ import print_function
+
+import os
+import subprocess
+
+from jinja2 import Environment, FileSystemLoader, StrictUndefined
+
+
+def repo_root():
+    return subprocess.check_output(
+        ["git", "rev-parse", "--show-toplevel"], text=True
+    ).strip()
+
+
+def fail(message):
+    raise SystemExit("kube-vip manifest test failed: " + message)
+
+
+def fake_ipsubnet(value):
+    # ansible.utils.ipsubnet -> network of the address as x.y.z.0/24
+    parts = value.split(".")
+    return ".".join(parts[:3]) + ".0/24"
+
+
+def fake_ipaddr(_value, expr=None):
+    # ansible.utils.ipaddr('prefix') -> prefix length
+    return "24"
+
+
+def fake_bool(value):
+    # Minimal stand-in for Ansible's truthiness filter used by the template.
+    if isinstance(value, bool):
+        return value
+    return str(value).lower() in ("1", "true", "yes", "on")
+
+
+def fake_map(seq, *args, **kwargs):
+    # Minimal stand-in for Ansible's map() filter in the two forms used by the
+    # template: map(attribute='x') on a list of dicts, and map('join', sep) on
+    # a list of sequences.
+    if "attribute" in kwargs:
+        return [item[kwargs["attribute"]] for item in seq]
+    if kwargs:
+        # e.g. map(default='x') not used here; ignore unknown kwargs.
+        return list(seq)
+    if args:
+        filter_name = args[0]
+        sep = args[1] if len(args) > 1 else ""
+        if filter_name == "join":
+            return [sep.join(str(x) for x in item) for item in seq]
+    return list(seq)
+
+
+def fake_zip(*seqs):
+    return list(zip(*seqs))
+
+
+def render(env, extra_vars):
+    base_vars = {
+        "apiserver_endpoint": "192.168.30.222",
+        "kube_vip_iface": "",
+        "kube_vip_arp": True,
+        "kube_vip_bgp": True,
+        "kube_vip_bgp_routerid": "127.0.0.1",
+        "_kube_vip_bgp_peers": [
+            {"peer_address": "192.168.30.1", "peer_asn": "64512"},
+            {"peer_address": "192.168.30.2", "peer_asn": "64513"},
+        ],
+        "kube_vip_tag_version": "v1.2.2",
+    }
+    base_vars.update(extra_vars)
+    template = env.get_template("vip.yaml.j2")
+    return template.render(**base_vars)
+
+
+def main():
+    root = repo_root()
+    template_dir = os.path.join(root, "roles", "k3s_server", "templates")
+    env = Environment(
+        loader=FileSystemLoader(template_dir), undefined=StrictUndefined
+    )
+    env.filters["ansible.utils.ipsubnet"] = fake_ipsubnet
+    env.filters["ansible.utils.ipaddr"] = fake_ipaddr
+    env.filters["bool"] = fake_bool
+    env.filters["map"] = fake_map
+    env.filters["zip"] = fake_zip
+
+    # Multi-peer BGP armed: must emit bgp_peers, never bgppeers.
+    output = render(env, {})
+    if "name: bgp_peers" not in output:
+        fail("rendered manifest is missing bgp_peers")
+    if "name: bgppeers" in output:
+        fail("rendered manifest still uses the ignored bgppeers key")
+    if "name: vip_subnet" not in output:
+        fail("rendered manifest is missing vip_subnet")
+    if "name: vip_cidr" in output:
+        fail("rendered manifest still uses the ignored vip_cidr key")
+    if "192.168.30.1:64512,192.168.30.2:64513" not in output:
+        fail("bgp_peers value is not comma-separated address:ASN entries")
+    if "ghcr.io/kube-vip/kube-vip:v1.2.2" not in output:
+        fail("kube-vip image tag is not v1.2.2")
+
+    # BGP enabled with no merged peers: single-peer fallback vars, no bgp_peers.
+    output = render(
+        env,
+        {
+            "_kube_vip_bgp_peers": [],
+            "kube_vip_bgp_as": "64513",
+            "kube_vip_bgp_peeraddress": "192.168.30.1",
+            "kube_vip_bgp_peeras": "64512",
+        },
+    )
+    if "name: bgp_as" not in output:
+        fail("single-peer bgp_as was not rendered")
+    if "name: bgp_peers" in output:
+        fail("bgp_peers present even though the peer list is empty")
+
+    print("kube-vip manifest regression test passed")
+
+
+if __name__ == "__main__":
+    main()

.github/scripts/test-metallb-remote-read.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+repo_root="$(git rev-parse --show-toplevel)"
+metallb_task="$repo_root/roles/k3s_server/tasks/metallb.yml"
+
+# The speaker tag verification must read the rendered manifest on the managed
+# host with slurp. A controller-side lookup('ansible.builtin.file', ...) would
+# read from the Ansible control node, which does not have the file, and would
+# fail on every MetalLB scenario.
+grep -Fq -- 'ansible.builtin.slurp' "$metallb_task" || {
+  printf 'MetalLB speaker tag check does not use slurp on the managed host\n' >&2
+  exit 1
+}
+
+grep -Eq -- 'lookup\(.?ansible\.builtin\.file' "$metallb_task" && {
+  printf 'MetalLB speaker tag check uses a controller-side file lookup\n' >&2
+  exit 1
+}
+
+# The check must reference the full image reference, not just a bare version
+# string that could appear anywhere in the manifest.
+grep -Fq -- 'quay.io/metallb/speaker:' "$metallb_task" || {
+  printf 'MetalLB speaker tag check does not match the full image reference\n' >&2
+  exit 1
+}
+
+printf 'MetalLB remote manifest read regression test passed\n'

.github/scripts/test-prepare-vagrant-box-masters.sh

@@ -0,0 +1,89 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+repo_root="$(git rev-parse --show-toplevel)"
+fixture="$(mktemp -d)"
+trap 'rm -rf -- "$fixture"' EXIT
+
+mock_bin="$fixture/bin"
+mock_state="$fixture/state"
+mock_home="$fixture/home"
+mock_vagrant_home="$fixture/vagrant-home"
+mock_box_dir="$mock_vagrant_home/boxes/bento-VAGRANTSLASH-ubuntu-26.04/202606.01.0/amd64/virtualbox"
+mock_vbox_root="$mock_home/VirtualBox VMs"
+mock_master_root="$mock_home/.cache/k3s-ci/vagrant-masters"
+lock_file="$fixture/vagrant-boxes.lock"
+mkdir -p -- "$mock_bin" "$mock_state" "$mock_box_dir" "$mock_vbox_root"
+printf '%s\n' 'bento/ubuntu-26.04 202606.01.0 amd64' > "$lock_file"
+ln -s "$repo_root/.github/scripts/test-fixtures/mock-vboxmanage" "$mock_bin/VBoxManage"
+ln -s "$repo_root/.github/scripts/test-fixtures/mock-vagrant" "$mock_bin/vagrant"
+ln -s "$repo_root/.github/scripts/test-fixtures/mock-flock" "$mock_bin/flock"
+
+export PATH="$mock_bin:$PATH"
+export HOME="$mock_home"
+export VAGRANT_HOME="$mock_vagrant_home"
+export VAGRANT_BOX_LOCK_FILE="$lock_file"
+export K3S_CI_REPOSITORY_ROOT="$repo_root"
+export K3S_CI_VAGRANT_MASTER_ROOT="$mock_master_root"
+export K3S_CI_VIRTUALBOX_ROOT="$mock_vbox_root"
+export MOCK_VBOX_STATE="$mock_state"
+export MOCK_VBOX_ROOT="$mock_vbox_root"
+export MOCK_BOX_DIR="$mock_box_dir"
+export MOCK_VBOX_LOG="$fixture/vbox.log"
+export MOCK_VAGRANT_LOG="$fixture/vagrant.log"
+export MOCK_VAGRANTFILE_CAPTURE="$fixture/prewarm-Vagrantfile"
+: > "$MOCK_VBOX_LOG"
+: > "$MOCK_VAGRANT_LOG"
+
+unowned_uuid='99999999-9999-4999-8999-999999999999'
+printf '%s\n' "$unowned_uuid" > "$mock_box_dir/master_id"
+
+script="$repo_root/.github/scripts/prepare-vagrant-box-masters.sh"
+first_output="$fixture/first-output"
+second_output="$fixture/second-output"
+third_output="$fixture/third-output"
+
+"$script" > "$first_output"
+if grep -Fq 'config.ssh.insert_key' "$MOCK_VAGRANTFILE_CAPTURE"; then
+  printf 'prewarm Vagrantfile unexpectedly overrides Vagrant SSH key insertion\n' >&2
+  exit 1
+fi
+grep -Fq 'virtualbox.memory = 1024' "$MOCK_VAGRANTFILE_CAPTURE"
+grep -Fq 'virtualbox.cpus = 2' "$MOCK_VAGRANTFILE_CAPTURE"
+grep -Fq 'config.vm.boot_timeout = 600' "$MOCK_VAGRANTFILE_CAPTURE"
+mapping_file="$mock_master_root/bento_ubuntu-26.04-202606.01.0-amd64.uuid"
+test -s "$mapping_file"
+cmp -s "$mapping_file" "$mock_box_dir/master_id"
+grep -Fq 'Created and recorded owned master' "$first_output"
+grep -Fq 'modifyvm' "$MOCK_VBOX_LOG"
+grep -Fq 'setextradata' "$MOCK_VBOX_LOG"
+if grep -Fq "$unowned_uuid" "$MOCK_VBOX_LOG"; then
+  printf 'unowned cached master UUID was unexpectedly inspected or modified\n' >&2
+  exit 1
+fi
+if grep -Eq 'unregistervm|closemedium' "$MOCK_VBOX_LOG"; then
+  printf 'master preparation invoked a destructive VirtualBox command\n' >&2
+  exit 1
+fi
+
+: > "$MOCK_VAGRANT_LOG"
+"$script" > "$second_output"
+grep -Fq 'Reusing owned master' "$second_output"
+if grep -Fq 'up ' "$MOCK_VAGRANT_LOG"; then
+  printf 'valid owned master was unexpectedly rebuilt\n' >&2
+  exit 1
+fi
+
+stale_uuid="$(tr -d '[:space:]' < "$mapping_file")"
+rm -f -- "$mock_state/vm-$stale_uuid"
+: > "$MOCK_VAGRANT_LOG"
+"$script" > "$third_output"
+grep -Fq 'rebuilding without deleting any VM or disk' "$third_output"
+grep -Fq 'up ' "$MOCK_VAGRANT_LOG"
+if grep -Eq 'unregistervm|closemedium' "$MOCK_VBOX_LOG"; then
+  printf 'stale master recovery invoked a destructive VirtualBox command\n' >&2
+  exit 1
+fi
+
+printf 'Vagrant box master preparation fixture test passed\n'

.github/scripts/vagrant-up-timed.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+if (($# < 2)); then
+  printf 'Usage: vagrant-up-timed.sh WORKDIR MACHINE [MACHINE ...]\n' >&2
+  exit 2
+fi
+
+workdir="$1"
+shift
+timing_log="${K3S_CI_CREATE_TIMING_LOG:-${RUNNER_TEMP:-/tmp}/k3s-ci-create-timing.log}"
+mkdir -p -- "${timing_log%/*}"
+
+printf '%s batch-start machines=%s\n' "$(date --iso-8601=ns)" "$*" | tee -a "$timing_log"
+set +e
+VAGRANT_CWD="$workdir" vagrant up "$@" --provider virtualbox --no-provision 2>&1 |
+  while IFS= read -r line; do
+    printf '%s %s\n' "$(date --iso-8601=ns)" "$line"
+  done | tee -a "$timing_log"
+rc=${PIPESTATUS[0]}
+set -e
+printf '%s batch-end rc=%d machines=%s\n' "$(date --iso-8601=ns)" "$rc" "$*" | tee -a "$timing_log"
+exit "$rc"

.github/test-download-boxes.sh

@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+
+# The single-quoted expressions below are written into fake executables and
+# intentionally expand only when those executables run.
+# shellcheck disable=SC2016
+
+set -Eeuo pipefail
+
+repo_root=$(git rev-parse --show-toplevel)
+test_root=$(mktemp -d)
+fake_bin="$test_root/bin"
+fake_log="$test_root/vagrant.log"
+output="$test_root/output.txt"
+mkdir -p "$fake_bin"
+trap 'rm -rf "$test_root"' EXIT
+
+printf '%s\n' \
+    '#!/usr/bin/env bash' \
+    'set -Eeuo pipefail' \
+    'printf "%s\n" bento/debian-13 bento/rockylinux-10.1 bento/ubuntu-26.04' \
+    >"$fake_bin/yq"
+
+printf '%s\n' \
+    '#!/usr/bin/env bash' \
+    'set -Eeuo pipefail' \
+    'emit_box() {' \
+    '    case "$1" in' \
+    '        bento/debian-13) version=202510.26.0 ;;' \
+    '        bento/rockylinux-10.1) version=202512.01.0 ;;' \
+    '        bento/ubuntu-26.04) version=202606.01.0 ;;' \
+    '        *) printf "Unexpected box: %s\n" "$1" >&2; exit 1 ;;' \
+    '    esac' \
+    '    printf "0,,box-name,%s\n" "$1"' \
+    '    printf "0,,box-provider,virtualbox\n"' \
+    '    printf "0,,box-version,%s\n" "$version"' \
+    '    printf "0,,box-architecture,amd64\n"' \
+    '}' \
+    'if [[ "${1:-}" == box && "${2:-}" == list ]]; then' \
+    '    emit_box bento/debian-13' \
+    '    emit_box bento/rockylinux-10.1' \
+    '    if [[ "${FAKE_PRESENT_MODE:-all}" == all ]]; then' \
+    '        emit_box bento/ubuntu-26.04' \
+    '    fi' \
+    'elif [[ "${1:-}" == box && "${2:-}" == add ]]; then' \
+    '    printf "%s\n" "$*" >>"${FAKE_VAGRANT_LOG:?}"' \
+    'else' \
+    '    printf "Unexpected vagrant arguments: %s\n" "$*" >&2' \
+    '    exit 1' \
+    'fi' \
+    >"$fake_bin/vagrant"
+chmod +x "$fake_bin/yq" "$fake_bin/vagrant"
+
+PATH="$fake_bin:$PATH" \
+    FAKE_VAGRANT_LOG="$fake_log" \
+    "$repo_root/.github/download-boxes.sh" >"$output"
+grep -Fq 'All pinned Vagrant boxes are already present.' "$output"
+[[ ! -e "$fake_log" ]]
+
+PATH="$fake_bin:$PATH" \
+    FAKE_PRESENT_MODE=partial \
+    FAKE_VAGRANT_LOG="$fake_log" \
+    "$repo_root/.github/download-boxes.sh" >"$output"
+grep -Fxq \
+    'box add --provider virtualbox --box-version 202606.01.0 --architecture amd64 bento/ubuntu-26.04' \
+    "$fake_log"
+
+incomplete_lock="$test_root/incomplete.lock"
+printf '%s\n' \
+    'bento/debian-13 202510.26.0 amd64' \
+    'bento/rockylinux-10.1 202512.01.0 amd64' \
+    >"$incomplete_lock"
+if PATH="$fake_bin:$PATH" \
+    VAGRANT_BOX_LOCK_FILE="$incomplete_lock" \
+    FAKE_VAGRANT_LOG="$fake_log" \
+    "$repo_root/.github/download-boxes.sh" >"$output" 2>&1; then
+    printf 'Download script accepted a lock missing a scenario box.\n' >&2
+    exit 1
+fi
+grep -Fq 'Scenario boxes missing from the lock file:' "$output"
+grep -Fq 'bento/ubuntu-26.04' "$output"
+
+duplicate_lock="$test_root/duplicate.lock"
+printf '%s\n' \
+    'bento/debian-13 202510.26.0 amd64' \
+    'bento/debian-13 202508.10.0 amd64' \
+    'bento/rockylinux-10.1 202512.01.0 amd64' \
+    'bento/ubuntu-26.04 202606.01.0 amd64' \
+    >"$duplicate_lock"
+if PATH="$fake_bin:$PATH" \
+    VAGRANT_BOX_LOCK_FILE="$duplicate_lock" \
+    FAKE_VAGRANT_LOG="$fake_log" \
+    "$repo_root/.github/download-boxes.sh" >"$output" 2>&1; then
+    printf 'Download script accepted duplicate box lock entries.\n' >&2
+    exit 1
+fi
+grep -Fq 'Duplicate Vagrant box lock entries:' "$output"
+
+printf 'Vagrant box download tests passed.\n'

.github/vagrant-boxes.lock

@@ -0,0 +1,4 @@
+# box version architecture
+bento/debian-13 202510.26.0 amd64
+bento/rockylinux-10.1 202512.01.0 amd64
+bento/ubuntu-26.04 202606.01.0 amd64

.github/workflows/cache.yml

@@ -5,38 +5,53 @@ on:
 jobs:
   molecule:
     name: cache
-    runs-on: self-hosted
+    runs-on: [self-hosted, linux, x64, k3s-ci, virtualbox, nested-virt]
     env:
       PYTHON_VERSION: "3.11"
+      VAGRANT_DEFAULT_PROVIDER: virtualbox
+      VAGRANT_HOME: ${{ github.workspace }}/.vagrant-home
 
     steps:
       - name: Check out the codebase
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # 4.2.2
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # 7.0.1
         with:
-          ref: ${{ github.event.pull_request.head.sha }}
+          ref: ${{ github.event.pull_request.head.sha || github.sha }}
+
+      - name: Check nested VirtualBox platform
+        run: |
+          set -Eeuo pipefail
+          grep -Eq 'vmx|svm' /proc/cpuinfo
+          test -c /dev/kvm
+          test -c /dev/vboxdrv
+          VBoxManage --version
+          vagrant --version
+          test -r /etc/vbox/networks.conf
+          test "$(stat -c '%u' /etc/vbox/networks.conf)" -eq 0
+          free -h
+          df -Pk "${RUNNER_TEMP}"
 
       - name: Set up Python ${{ env.PYTHON_VERSION }}
-        uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # 5.3.0
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # 7.0.0
         with:
           python-version: ${{ env.PYTHON_VERSION }}
           cache: 'pip' # caching pip dependencies
 
       - name: Cache Vagrant boxes
         id: cache-vagrant
-        uses: actions/cache@6849a6489940f00c2f30c0fb92c6274307ccb58a # 4.1.2
+        uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # 6.1.0
         with:
-          lookup-only: true #if it exists, we don't need to restore and can skip the next step
+          # This producer only needs to know whether the immutable cache exists.
+          # Molecule jobs restore it after this job completes.
+          lookup-only: true
           path: |
-            ~/.vagrant.d/boxes
-          key: vagrant-boxes-${{ hashFiles('**/molecule.yml') }}
-          restore-keys: |
-            vagrant-boxes
+            .vagrant-home/boxes
+          key: vagrant-boxes-${{ runner.name }}-${{ runner.os }}-${{ runner.arch }}-virtualbox-7.2-vagrant-2.4-${{ hashFiles('.github/vagrant-boxes.lock') }} # yamllint disable-line rule:line-length
 
       - name: Download Vagrant boxes for all scenarios
-        # To save some cache space, all scenarios share the same cache key.
-        # On the other hand, this means that the cache contents should be
-        # the same across all scenarios. This step ensures that.
-        if: steps.cache-vagrant.outputs.cache-hit != 'true' # only run if false since this is just a cache step
+        # An exact hit skips both cache restoration and upstream downloads.
+        # A lock change builds and saves one clean, version-pinned cache.
+        if: steps.cache-vagrant.outputs.cache-hit != 'true'
         run: |
           ./.github/download-boxes.sh
+          ./.github/scripts/prepare-vagrant-box-masters.sh
           vagrant box list

.github/workflows/ci.yml

@@ -2,9 +2,13 @@
 name: "CI"
 on:
   pull_request:
+    branches:
+      - master
     types:
       - opened
       - synchronize
+      - reopened
+      - ready_for_review
     paths-ignore:
       - '**/.gitignore'
       - '**/FUNDING.yml'
@@ -16,12 +20,21 @@ on:
       - '**/LICENSE'
       - '**/reboot.sh'
       - '**/reset.sh'
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ci-${{ github.event.pull_request.number || github.run_id }}
+  cancel-in-progress: true
+
 jobs:
   pre:
     uses: ./.github/workflows/cache.yml
+    needs: [lint]
   lint:
     uses: ./.github/workflows/lint.yml
-    needs: [pre]
   test:
     uses: ./.github/workflows/test.yml
     needs: [pre, lint]

.github/workflows/lint.yml

@@ -5,24 +5,25 @@ on:
 jobs:
   pre-commit-ci:
     name: Pre-Commit
-    runs-on: self-hosted
+    runs-on: ubuntu-latest
     env:
-      PYTHON_VERSION: "3.11"
+      PYTHON_VERSION: "3.12"
 
     steps:
       - name: Check out the codebase
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # 4.2.2
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # 7.0.1
         with:
-          ref: ${{ github.event.pull_request.head.sha }}
+          ref: ${{ github.event.pull_request.head.sha || github.sha }}
 
       - name: Set up Python ${{ env.PYTHON_VERSION }}
-        uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # 5.3.0
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # 7.0.0
         with:
           python-version: ${{ env.PYTHON_VERSION }}
           cache: 'pip' # caching pip dependencies
 
       - name: Restore Ansible cache
-        uses: actions/cache/restore@6849a6489940f00c2f30c0fb92c6274307ccb58a # 4.1.2
+        id: cache-ansible
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # 6.1.0
         with:
           path: ~/.ansible/collections
           key: ansible-${{ hashFiles('collections/requirements.yml') }}
@@ -37,17 +38,37 @@ jobs:
           python3 -m pip install -r requirements.txt
           echo "::endgroup::"
 
+      - name: Install Ansible collections with retries
+        if: steps.cache-ansible.outputs.cache-hit != 'true'
+        run: |
+          set -Eeuo pipefail
+          for attempt in 1 2 3 4 5; do
+            if ansible-galaxy collection install -r collections/requirements.yml; then
+              exit 0
+            fi
+            echo "Ansible Galaxy attempt ${attempt} failed; retrying."
+            sleep $((attempt * 10))
+          done
+          exit 1
+
+      - name: Save Ansible collection cache
+        if: steps.cache-ansible.outputs.cache-hit != 'true'
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # 6.1.0
+        with:
+          path: ~/.ansible/collections
+          key: ansible-${{ hashFiles('collections/requirements.yml') }}
+
       - name: Run pre-commit
         uses: pre-commit/action@2c7b3805fd2a0fd8c1884dcaebf91fc102a13ecd # 3.0.1
 
   ensure-pinned-actions:
     name: Ensure SHA Pinned Actions
-    runs-on: self-hosted
+    runs-on: ubuntu-latest
     steps:
       - name: Checkout code
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # 4.2.2
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # 7.0.1
       - name: Ensure SHA pinned actions
-        uses: zgosalvez/github-actions-ensure-sha-pinned-actions@38608ef4fb69adae7f1eac6eeb88e67b7d083bfd # 3.0.16
+        uses: zgosalvez/github-actions-ensure-sha-pinned-actions@3db98c0363e2fa5df3e1c4c471777a7c10b24cc9 # 5.0.5
         with:
           allowlist: |
             aws-actions/

.github/workflows/test.yml

@@ -5,72 +5,64 @@ on:
 jobs:
   molecule:
     name: Molecule
-    runs-on: self-hosted
+    runs-on: [self-hosted, linux, x64, k3s-ci, virtualbox, nested-virt]
     strategy:
       matrix:
         scenario:
           - default
-          # - ipv6
           - single_node
           - calico
           - cilium
           - kube-vip
-      fail-fast: false
+          # - ipv6
+      fail-fast: true
+      max-parallel: 1
     env:
       PYTHON_VERSION: "3.11"
+      VAGRANT_DEFAULT_PROVIDER: virtualbox
+      VAGRANT_HOME: ${{ github.workspace }}/.vagrant-home
 
     steps:
       - name: Check out the codebase
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # 4.2.2
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # 7.0.1
         with:
-          ref: ${{ github.event.pull_request.head.sha }}
+          ref: ${{ github.event.pull_request.head.sha || github.sha }}
 
-      # these steps are necessary if not using ephemeral nodes
-      - name: Delete old Vagrant box versions
-        if: always() # do this even if a step before has failed
-        run: vagrant box prune --force
+      - name: Clean repository-owned resources before testing
+        run: ./.github/scripts/cleanup-runner-resources.sh --apply
 
-      - name: Remove all local Vagrant boxes
-        if: always() # do this even if a step before has failed
-        run: if vagrant box list 2>/dev/null; then vagrant box list | cut -f 1 -d ' ' | xargs -L 1 vagrant box remove -f 2>/dev/null && echo "All Vagrant boxes removed." || echo "No Vagrant boxes found."; else echo "No Vagrant boxes found."; fi
+      - name: Record host-only network baseline
+        run: ./.github/scripts/cleanup-runner-resources.sh --snapshot
 
-      - name: Remove all Virtualbox VMs
-        if: always() # do this even if a step before has failed
-        run: VBoxManage list vms | awk -F'"' '{print $2}' | xargs -I {} VBoxManage unregistervm --delete "{}"
-
-      - name: Remove all Virtualbox HDs
-        if: always() # do this even if a step before has failed
-        run: VBoxManage list hdds | awk -F':' '/^UUID:/ {print $2}' | xargs -I {} VBoxManage closemedium disk "{}" --delete
-
-      - name: Remove all Virtualbox Networks
-        if: always() # do this even if a step before has failed
-        run: VBoxManage list hostonlyifs | grep '^Name:' | awk '{print $2}' | grep '^vboxnet' | xargs -I {} VBoxManage hostonlyif remove {}
-
-      - name: Remove Virtualbox network config
-        if: always() # do this even if a step before has failed
-        run: sudo rm /etc/vbox/networks.conf || true
-
-      - name: Configure VirtualBox
-        run: |-
-          sudo mkdir -p /etc/vbox
-          cat <<EOF | sudo tee -a /etc/vbox/networks.conf > /dev/null
-          * 192.168.30.0/24
-          * fdad:bad:ba55::/64
-          EOF
+      - name: Check nested VirtualBox platform
+        run: |
+          set -Eeuo pipefail
+          grep -Eq 'vmx|svm' /proc/cpuinfo
+          test -c /dev/kvm
+          test -c /dev/vboxdrv
+          VBoxManage --version
+          vagrant --version
+          test -r /etc/vbox/networks.conf
+          test "$(stat -c '%u' /etc/vbox/networks.conf)" -eq 0
+          free -h
+          df -Pk "${RUNNER_TEMP}"
 
       - name: Set up Python ${{ env.PYTHON_VERSION }}
-        uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # 5.3.0
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # 7.0.0
         with:
           python-version: ${{ env.PYTHON_VERSION }}
           cache: 'pip' # caching pip dependencies
 
       - name: Restore vagrant Boxes cache
-        uses: actions/cache/restore@6849a6489940f00c2f30c0fb92c6274307ccb58a # 4.1.2
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # 6.1.0
         with:
-          path: ~/.vagrant.d/boxes
-          key: vagrant-boxes-${{ hashFiles('**/molecule.yml') }}
+          path: .vagrant-home/boxes
+          key: vagrant-boxes-${{ runner.name }}-${{ runner.os }}-${{ runner.arch }}-virtualbox-7.2-vagrant-2.4-${{ hashFiles('.github/vagrant-boxes.lock') }} # yamllint disable-line rule:line-length
           fail-on-cache-miss: true
 
+      - name: Prepare runner-owned Vagrant box masters
+        run: ./.github/scripts/prepare-vagrant-box-masters.sh
+
       - name: Install dependencies
         run: |
           echo "::group::Upgrade pip"
@@ -82,45 +74,45 @@ jobs:
           echo "::endgroup::"
 
       - name: Test with molecule
-        run: molecule test --scenario-name ${{ matrix.scenario }}
-        timeout-minutes: 90
+        run: |
+          set -Eeuo pipefail
+          resource_dir="${RUNNER_TEMP}/logs/resources/${{ matrix.scenario }}"
+          timing_file="${RUNNER_TEMP}/logs/timing/${{ matrix.scenario }}.txt"
+          mkdir -p -- "${timing_file%/*}"
+          ./.github/scripts/monitor-runner-resources.sh "$resource_dir" 10 &
+          monitor_pid=$!
+          stop_monitor() {
+            kill -TERM "$monitor_pid" 2>/dev/null || true
+            wait "$monitor_pid" 2>/dev/null || true
+          }
+          trap stop_monitor EXIT
+          /usr/bin/time -v -o "$timing_file" \
+            molecule test --scenario-name ${{ matrix.scenario }}
+        timeout-minutes: 150
         env:
           ANSIBLE_K3S_LOG_DIR: ${{ runner.temp }}/logs/k3s-ansible/${{ matrix.scenario }}
           ANSIBLE_SSH_RETRIES: 4
           ANSIBLE_TIMEOUT: 120
           PY_COLORS: 1
           ANSIBLE_FORCE_COLOR: 1
+          K3S_CI_CREATE_TIMING_LOG: ${{ runner.temp }}/logs/timing/${{ matrix.scenario }}-create.log
 
-      # these steps are necessary if not using ephemeral nodes
-      - name: Delete old Vagrant box versions
-        if: always() # do this even if a step before has failed
-        run: vagrant box prune --force
+      - name: Collect runner diagnostics
+        if: always()
+        run: ./.github/scripts/collect-runner-diagnostics.sh "${RUNNER_TEMP}/logs/runner"
+        env:
+          K3S_CI_SCENARIO_NAME: ${{ matrix.scenario }}
 
-      - name: Remove all local Vagrant boxes
-        if: always() # do this even if a step before has failed
-        run: if vagrant box list 2>/dev/null; then vagrant box list | cut -f 1 -d ' ' | xargs -L 1 vagrant box remove -f 2>/dev/null && echo "All Vagrant boxes removed." || echo "No Vagrant boxes found."; else echo "No Vagrant boxes found."; fi
-
-      - name: Remove all Virtualbox VMs
-        if: always() # do this even if a step before has failed
-        run: VBoxManage list vms | awk -F'"' '{print $2}' | xargs -I {} VBoxManage unregistervm --delete "{}"
-
-      - name: Remove all Virtualbox HDs
-        if: always() # do this even if a step before has failed
-        run: VBoxManage list hdds | awk -F':' '/^UUID:/ {print $2}' | xargs -I {} VBoxManage closemedium disk "{}" --delete
-
-      - name: Remove all Virtualbox Networks
-        if: always() # do this even if a step before has failed
-        run: VBoxManage list hostonlyifs | grep '^Name:' | awk '{print $2}' | grep '^vboxnet' | xargs -I {} VBoxManage hostonlyif remove {}
-
-      - name: Remove Virtualbox network config
-        if: always() # do this even if a step before has failed
-        run: sudo rm /etc/vbox/networks.conf || true
+      - name: Clean repository-owned resources after testing
+        if: always()
+        run: ./.github/scripts/cleanup-runner-resources.sh --apply
 
       - name: Upload log files
         if: always() # do this even if a step before has failed
-        uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # 4.4.3
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # 7.0.1
         with:
-          name: logs
+          name: logs-${{ matrix.scenario }}-${{ github.run_id }}-${{ github.run_attempt }}
           path: |
             ${{ runner.temp }}/logs
-          overwrite: true
+          if-no-files-found: warn
+          retention-days: 14

.gitignore

@@ -1,4 +1,6 @@
 .env/
 *.log
 ansible.cfg
+.ansible/
 kubeconfig
+zIgnore/

.pre-commit-config.yaml

@@ -20,6 +20,9 @@ repos:
     rev: v6.22.2
     hooks:
       - id: ansible-lint
+        additional_dependencies: [ansible-core==2.18.0]
+        language_version: python3.12
+        args: [--offline]
   - repo: https://github.com/shellcheck-py/shellcheck-py
     rev: v0.9.0.6
     hooks:
@@ -33,3 +36,51 @@ repos:
     rev: 0.6.4
     hooks:
       - id: fix-smartquotes
+  - repo: local
+    hooks:
+      - id: cleanup-runner-resources-test
+        name: cleanup runner resources test
+        entry: .github/scripts/test-cleanup-runner-resources.sh
+        language: system
+        pass_filenames: false
+        files: ^\.github/scripts/(cleanup-runner-resources|test-cleanup-runner-resources)\.sh$
+      - id: download-vagrant-boxes-test
+        name: Vagrant box download test
+        entry: .github/test-download-boxes.sh
+        language: system
+        pass_filenames: false
+        files: ^\.github/(download-boxes|test-download-boxes)\.sh$|^\.github/vagrant-boxes\.lock$
+      - id: prepare-vagrant-box-masters-test
+        name: Vagrant box master preparation test
+        entry: .github/scripts/test-prepare-vagrant-box-masters.sh
+        language: system
+        pass_filenames: false
+        files: ^\.github/scripts/(prepare-vagrant-box-masters|test-prepare-vagrant-box-masters)\.sh$
+      - id: k3s-server-bootstrap-test
+        name: K3s transient bootstrap test
+        entry: .github/scripts/test-k3s-server-bootstrap.sh
+        language: system
+        pass_filenames: false
+        files: ^roles/k3s_server/tasks/(main|join_master)\.yml$|^\.github/scripts/test-k3s-server-bootstrap\.sh$
+      - id: cilium-bgp-manifest-test
+        name: Cilium BGP manifest test
+        entry: python3 .github/scripts/test-cilium-bgp-manifest.py
+        language: python
+        additional_dependencies:
+          - Jinja2>=3.1
+        pass_filenames: false
+        files: ^roles/k3s_server_post/templates/cilium\.crs\.j2$|^\.github/scripts/test-cilium-bgp-manifest\.py$
+      - id: kube-vip-manifest-test
+        name: kube-vip manifest test
+        entry: python3 .github/scripts/test-kube-vip-manifest.py
+        language: python
+        additional_dependencies:
+          - Jinja2>=3.1
+        pass_filenames: false
+        files: ^roles/k3s_server/templates/vip\.yaml\.j2$|^\.github/scripts/test-kube-vip-manifest\.py$
+      - id: metallb-remote-read-test
+        name: MetalLB remote read test
+        entry: .github/scripts/test-metallb-remote-read.sh
+        language: system
+        pass_filenames: false
+        files: ^roles/k3s_server/tasks/metallb\.yml$|^\.github/scripts/test-metallb-remote-read\.sh$

README.md

@@ -8,7 +8,7 @@ This is based on the work from [this fork](https://github.com/212850a/k3s-ansibl
 
 If you want more context on how this works, see:
 
-📄 [Documentation](https://technotim.live/posts/k3s-etcd-ansible/) (including example commands)
+📄 [Documentation](https://technotim.com/posts/k3s-etcd-ansible/) (including example commands)
 
 📺 [Watch the Video](https://www.youtube.com/watch?v=CbkEWcUZ7zM)
 
@@ -16,9 +16,9 @@ If you want more context on how this works, see:
 
 Build a Kubernetes cluster using Ansible with k3s. The goal is easily install a HA Kubernetes cluster on machines running:
 
-- [x] Debian (tested on version 11)
-- [x] Ubuntu (tested on version 22.04)
-- [x] Rocky (tested on version 9)
+- [x] Debian (tested on version 13)
+- [x] Ubuntu (tested on version 26.04 LTS)
+- [x] Rocky (tested on version 10)
 
 on processor architecture:
 
@@ -28,7 +28,7 @@ on processor architecture:
 
 ## ✅ System requirements
 
-- Control Node (the machine you are running `ansible` commands) must have Ansible 2.11+ If you need a quick primer on Ansible [you can check out my docs and setting up Ansible](https://technotim.live/posts/ansible-automation/).
+- Control Node (the machine you are running `ansible` commands) must have Ansible 2.11+ If you need a quick primer on Ansible [you can check out my docs and setting up Ansible](https://technotim.com/posts/ansible-automation/).
 
 - You will also need to install collections that this playbook uses by running `ansible-galaxy collection install -r ./collections/requirements.yml` (important❗)
 
@@ -91,6 +91,32 @@ ansible-playbook reset.yml -i inventory/my-cluster/hosts.ini
 
 >You should also reboot these nodes due to the VIP not being destroyed
 
+## 🔁 Upgrading an existing cluster
+
+These version variables select the components used for a **fresh** installation.
+They are not a supported direct in-place upgrade path for an existing cluster.
+K3s, Calico, and Cilium each require staged upgrades for long-lived clusters.
+
+- **K3s**: do not jump an embedded-etcd cluster straight to Kubernetes 1.36.
+  Upgrade one Kubernetes minor version at a time. From the sample default
+  (`v1.30.2+k3s2`) the sequence is: the latest supported 1.30 patch, then 1.31,
+  1.32, a 1.33 patch that contains etcd 3.5.26 (for example `v1.33.7+k3s3`),
+  then 1.34, 1.35, and finally 1.36. Upgrade servers one at a time before
+  agents. Take backups and confirm cluster health at each step; this playbook
+  does not automate the upgrade, so those remain manual operational steps. See
+  [K3s manual upgrades](https://docs.k3s.io/upgrades/manual) and the
+  [v1.34 release notes](https://docs.k3s.io/release-notes/v1.34.X).
+- **Cilium**: upstream supports only consecutive minor upgrades. Update to the
+  latest patch of the current minor, then upgrade 1.17, 1.18, 1.19, and 1.20 in
+  order, reading each version's upgrade notes and running preflight checks.
+  Do not attempt a direct upgrade from an old Cilium to 1.20.
+- **Calico**: starting with 3.28 the v3 resource UID behavior changed. If you
+  have operators with OwnerReferences pointing to `projectcalico.org/v3`
+  resources, remove and recreate those references around an in-place upgrade.
+- **MetalLB**: this project installs application tag `v0.16.0`. A newer
+  chart-only tag such as `metallb-chart-0.16.1` is not an application or image
+  release and must not be used as the controller or speaker image tag.
+
 ## ⚙️ Kube Config
 
 To copy your `kube config` locally so that you can access your **Kubernetes** cluster run:
@@ -115,7 +141,7 @@ Then change `server: https://127.0.0.1:6443` to match your master IP: `server: h
 
 ### 🔨 Testing your cluster
 
-See the commands [here](https://technotim.live/posts/k3s-etcd-ansible/#testing-your-cluster).
+See the commands [here](https://technotim.com/posts/k3s-etcd-ansible/#testing-your-cluster).
 
 ### Variables
 
@@ -146,11 +172,11 @@ See the commands [here](https://technotim.live/posts/k3s-etcd-ansible/#testing-y
 | `k3s_server` | `kube_vip_bgp_peers` | list | `[]` | Not required | List of BGP peer ASN & address pairs |
 | `k3s_server` | `kube_vip_bgp_peers_groups` | list | `['k3s_master']` | Not required | Inventory group in which to search for additional `kube_vip_bgp_peers` parameters to merge. |
 | `k3s_server` | `kube_vip_iface` | string | `~` | Not required | Explicitly define an interface that ALL control nodes should use to propagate the VIP, define it here. Otherwise, kube-vip will determine the right interface automatically at runtime. |
-| `k3s_server` | `kube_vip_tag_version` | string | `v0.7.2` | Not required | Image tag for kube-vip |
-| `k3s_server` | `kube_vip_cloud_provider_tag_version` | string | `main` | Not required | Tag for kube-vip-cloud-provider manifest when enable |
+| `k3s_server` | `kube_vip_tag_version` | string | `v1.2.2` | Not required | Image tag for kube-vip |
+| `k3s_server` | `kube_vip_cloud_provider_tag_version` | string | `v0.0.12` | Not required | Tag for kube-vip-cloud-provider manifest when enable |
 | `k3s_server`, `k3_server_post` | `kube_vip_lb_ip_range` | string | `~` | Not required | IP range for kube-vip load balancer |
-| `k3s_server`, `k3s_server_post` | `metal_lb_controller_tag_version` | string | `v0.14.3` | Not required | Image tag for MetalLB |
-| `k3s_server` | `metal_lb_speaker_tag_version` | string | `v0.14.3` | Not required | Image tag for MetalLB |
+| `k3s_server`, `k3s_server_post` | `metal_lb_controller_tag_version` | string | `v0.16.0` | Not required | Image tag for MetalLB |
+| `k3s_server` | `metal_lb_speaker_tag_version` | string | `v0.16.0` | Not required | Image tag for MetalLB |
 | `k3s_server` | `metal_lb_type` | string | `native` | Not required | Use FRR mode or native. Valid values are `frr` and `native` |
 | `k3s_server` | `retry_count` | int | `20` | Not required | Amount of retries when verifying that nodes joined |
 | `k3s_server` | `server_init_args` | string | ❌ | Not required | Arguments for server nodes |
@@ -162,7 +188,7 @@ See the commands [here](https://technotim.live/posts/k3s-etcd-ansible/#testing-y
 | `k3s_server_post` | `calico_natOutgoing` | string | `Enabled` | Not required | IP pool NAT outgoing |
 | `k3s_server_post` | `calico_nodeSelector` | string | `all()` | Not required | IP pool node selector |
 | `k3s_server_post` | `calico_iface` | string | `~` | Not required | The network interface used for when Calico is enabled |
-| `k3s_server_post` | `calico_tag` | string | `v3.27.2` | Not required | Calico version tag |
+| `k3s_server_post` | `calico_tag` | string | `v3.32.1` | Not required | Calico version tag |
 | `k3s_server_post` | `cilium_bgp_my_asn` | int | `64513` | Not required | Local ASN for BGP peer |
 | `k3s_server_post` | `cilium_bgp_peer_asn` | int | `64512` | Not required | BGP peer ASN |
 | `k3s_server_post` | `cilium_bgp_peer_address` | string | `~` | Not required | BGP peer address |
@@ -171,14 +197,15 @@ See the commands [here](https://technotim.live/posts/k3s-etcd-ansible/#testing-y
 | `k3s_server_post` | `cilium_bgp_lb_cidr` | string | `192.168.31.0/24` | Not required | BGP load balancer IP range |
 | `k3s_server_post` | `cilium_exportPodCIDR` | bool | `true` | Not required | Export pod CIDR |
 | `k3s_server_post` | `cilium_hubble` | bool | `true` | Not required | Enable Cilium Hubble |
-| `k3s_server_post` | `cilium_hubble` | bool | `true` | Not required | Enable Cilium Hubble |
-| `k3s_server_post` | `cilium_mode` | string | `native` | Not required | Inner-node communication mode (choices are `native` and `routed`) |
+| `k3s_server_post` | `cilium_mode` | string | `native` | Not required | Inner-node communication mode (choices are `native` and `tunnel`; `routed` is a deprecated alias for `tunnel`) |
+| `k3s_server_post` | `cilium_tag` | string | `v1.20.0` | Not required | Cilium version tag |
+| `k3s_server_post` | `cilium_cli_tag` | string | `v0.19.7` | Not required | Cilium CLI version tag |
 | `k3s_server_post` | `cluster_cidr` | string | `10.52.0.0/16` | Not required | Inner-cluster IP range |
 | `k3s_server_post` | `enable_bpf_masquerade` | bool | `true` | Not required | Use IP masquerading |
 | `k3s_server_post` | `kube_proxy_replacement` | bool | `true` | Not required | Replace the native kube-proxy with Cilium |
 | `k3s_server_post` | `metal_lb_available_timeout` | string | `240s` | Not required | Wait for MetalLB resources |
 | `k3s_server_post` | `metal_lb_ip_range` | string | `192.168.30.80-192.168.30.90` | Not required | MetalLB ip range for load balancer |
-| `k3s_server_post` | `metal_lb_controller_tag_version` | string | `v0.14.3` | Not required | Image tag for MetalLB |
+| `k3s_server_post` | `metal_lb_controller_tag_version` | string | `v0.16.0` | Not required | Image tag for MetalLB |
 | `k3s_server_post` | `metal_lb_mode` | string | `layer2` | Not required | Metallb mode (choices are `bgp` and `layer2`) |
 | `k3s_server_post` | `metal_lb_bgp_my_asn` | string | `~` | Not required | BGP ASN configurations |
 | `k3s_server_post` | `metal_lb_bgp_peer_asn` | string | `~` | Not required | BGP peer ASN configurations |
@@ -191,7 +218,7 @@ See the commands [here](https://technotim.live/posts/k3s-etcd-ansible/#testing-y
 
 ### Troubleshooting
 
-Be sure to see [this post](https://github.com/techno-tim/k3s-ansible/discussions/20) on how to troubleshoot common problems
+Be sure to see [this post](https://github.com/timothystewart6/k3s-ansible/discussions/20) on how to troubleshoot common problems
 
 ### Testing the playbook using molecule
 
@@ -218,7 +245,7 @@ collections:
   - name: community.general
   - name: ansible.posix
   - name: kubernetes.core
-  - name: https://github.com/techno-tim/k3s-ansible.git
+  - name: https://github.com/timothystewart6/k3s-ansible.git
     type: git
     version: master
 ```

galaxy.yml

@@ -56,16 +56,16 @@ dependencies:
   kubernetes.core: '*'
 
 # The URL of the originating SCM repository
-repository: https://github.com/techno-tim/k3s-ansible
+repository: https://github.com/timothystewart6/k3s-ansible
 
 # The URL to any online docs
-documentation: https://github.com/techno-tim/k3s-ansible
+documentation: https://github.com/timothystewart6/k3s-ansible
 
 # The URL to the homepage of the collection/project
 homepage: https://www.youtube.com/watch?v=CbkEWcUZ7zM
 
 # The URL to the collection issue tracker
-issues: https://github.com/techno-tim/k3s-ansible/issues
+issues: https://github.com/timothystewart6/k3s-ansible/issues
 
 # A list of file glob-like patterns used to filter any files or directories that should not be included in the build
 # artifact. A pattern is matched from the relative path of the file or directory of the collection directory. This

inventory/sample/group_vars/all.yml

@@ -1,5 +1,5 @@
 ---
-k3s_version: v1.30.2+k3s2
+k3s_version: v1.36.2+k3s1
 # this is the user that has ssh access to these machines
 ansible_user: ansibleuser
 systemd_dir: /etc/systemd/system
@@ -13,13 +13,14 @@ flannel_iface: eth0
 # uncomment calico_iface to use tigera operator/calico cni instead of flannel https://docs.tigera.io/calico/latest/about
 # calico_iface: "eth0"
 calico_ebpf: false # use eBPF dataplane instead of iptables
-calico_tag: v3.28.0 # calico version tag
+calico_tag: v3.32.1 # calico version tag
 
 # uncomment cilium_iface to use cilium cni instead of flannel or calico
 # ensure v4.19.57, v5.1.16, v5.2.0 or more recent kernel
 # cilium_iface: "eth0"
-cilium_mode: native # native when nodes on same subnet or using bgp, else set routed
-cilium_tag: v1.16.0 # cilium version tag
+cilium_mode: native # native when nodes are on the same subnet or use BGP, otherwise set tunnel
+cilium_tag: v1.20.0 # cilium version tag
+cilium_cli_tag: v0.19.7 # cilium cli version tag
 cilium_hubble: true # enable hubble observability relay and ui
 
 # if using calico or cilium, you may specify the cluster pod cidr pool
@@ -84,10 +85,10 @@ extra_agent_args: >-
   {{ extra_args }}
 
 # image tag for kube-vip
-kube_vip_tag_version: v0.8.2
+kube_vip_tag_version: v1.2.2
 
 # tag for kube-vip-cloud-provider manifest
-# kube_vip_cloud_provider_tag_version: "main"
+# kube_vip_cloud_provider_tag_version: "v0.0.12"
 
 # kube-vip ip range for load balancer
 # (uncomment to use kube-vip for services instead of MetalLB)
@@ -105,8 +106,8 @@ metal_lb_mode: layer2
 # metal_lb_bgp_peer_address: "192.168.30.1"
 
 # image tag for metal lb
-metal_lb_speaker_tag_version: v0.14.8
-metal_lb_controller_tag_version: v0.14.8
+metal_lb_speaker_tag_version: v0.16.0
+metal_lb_controller_tag_version: v0.16.0
 
 # metallb ip range for load balancer
 metal_lb_ip_range: 192.168.30.80-192.168.30.90

molecule/calico/molecule.yml

@@ -5,14 +5,9 @@ driver:
   name: vagrant
 platforms:
   - name: control1
-    box: generic/ubuntu2204
+    box: bento/ubuntu-26.04
     memory: 4096
     cpus: 4
-    config_options:
-      # We currently can not use public-key based authentication on Ubuntu 22.04,
-      # see: https://github.com/chef/bento/issues/1405
-      ssh.username: vagrant
-      ssh.password: vagrant
     groups:
       - k3s_cluster
       - master

molecule/calico/overrides.yml

@@ -1,12 +1,14 @@
 ---
 - name: Apply overrides
   hosts: all
+  serial: 1
   tasks:
     - name: Override host variables
       ansible.builtin.set_fact:
         # See:
         # https://github.com/flannel-io/flannel/blob/67d603aaf45ef80f5dd39f43714fc5e6f8a637eb/Documentation/troubleshooting.md#Vagrant
-        calico_iface: eth1
+        calico_iface: "{{ 'eth1' if 'eth1' in ansible_facts.interfaces else 'enp0s8' }}"
+        kube_vip_iface: "{{ 'eth1' if 'eth1' in ansible_facts.interfaces else 'enp0s8' }}"
 
         # The test VMs might be a bit slow, so we give them more time to join the cluster:
         retry_count: 45

molecule/calico/verify-vars.yml

@@ -0,0 +1,6 @@
+---
+# Durable verify inputs for the calico (Calico CNI + MetalLB) scenario.
+verify_cni: calico
+verify_lb: metallb
+verify_lb_ip_range:
+  - 192.168.30.100-192.168.30.109

molecule/cilium/molecule.yml

@@ -5,14 +5,9 @@ driver:
   name: vagrant
 platforms:
   - name: control1
-    box: generic/ubuntu2204
+    box: bento/ubuntu-26.04
     memory: 4096
     cpus: 4
-    config_options:
-      # We currently can not use public-key based authentication on Ubuntu 22.04,
-      # see: https://github.com/chef/bento/issues/1405
-      ssh.username: vagrant
-      ssh.password: vagrant
     groups:
       - k3s_cluster
       - master

molecule/cilium/overrides.yml

@@ -1,12 +1,14 @@
 ---
 - name: Apply overrides
   hosts: all
+  serial: 1
   tasks:
     - name: Override host variables
       ansible.builtin.set_fact:
         # See:
         # https://github.com/flannel-io/flannel/blob/67d603aaf45ef80f5dd39f43714fc5e6f8a637eb/Documentation/troubleshooting.md#Vagrant
-        cilium_iface: eth1
+        cilium_iface: "{{ 'eth1' if 'eth1' in ansible_facts.interfaces else 'enp0s8' }}"
+        kube_vip_iface: "{{ 'eth1' if 'eth1' in ansible_facts.interfaces else 'enp0s8' }}"
 
         # The test VMs might be a bit slow, so we give them more time to join the cluster:
         retry_count: 45

molecule/cilium/verify-vars.yml

@@ -0,0 +1,6 @@
+---
+# Durable verify inputs for the cilium (Cilium CNI + MetalLB) scenario.
+verify_cni: cilium
+verify_lb: metallb
+verify_lb_ip_range:
+  - 192.168.30.110-192.168.30.119

molecule/default/create.yml

@@ -0,0 +1,80 @@
+---
+- name: Create
+  hosts: localhost
+  connection: local
+  gather_facts: false
+  no_log: "{{ molecule_no_log }}"
+  vars:
+    create_batches:
+      - [control1, control2]
+      - [control3, node1]
+      - [node2]
+  tasks:
+    - name: Verify that bounded batches cover the configured platforms exactly once
+      ansible.builtin.assert:
+        that:
+          - create_batches | flatten | sort == molecule_yml.platforms | map(attribute='name') | sort
+          - create_batches | flatten | length == create_batches | flatten | unique | length
+          - create_batches | map('length') | max <= 2
+        fail_msg: Bounded create batches do not match the configured default platforms.
+
+    - name: Generate the complete Vagrant configuration # noqa fqcn[action]
+      vagrant:
+        instances: "{{ molecule_yml.platforms }}"
+        default_box: "{{ molecule_yml.driver.default_box | default('generic/alpine316') }}"
+        provider_name: "{{ molecule_yml.driver.provider.name | default(omit, true) }}"
+        provision: "{{ molecule_yml.driver.provision | default(omit) }}"
+        cachier: "{{ molecule_yml.driver.cachier | default(omit) }}"
+        parallel: false
+        state: halt
+      changed_when: false
+
+    - name: Start clean Vagrant guests in bounded batches
+      ansible.builtin.command:
+        argv: >-
+          {{
+            [playbook_dir + '/../../.github/scripts/vagrant-up-timed.sh',
+             molecule_ephemeral_directory] + item
+          }}
+      loop: "{{ create_batches }}"
+      loop_control:
+        label: "{{ item | join(', ') }}"
+      changed_when: true
+
+    - name: Reconcile all instances and collect their connection configuration # noqa fqcn[action]
+      vagrant:
+        instances: "{{ molecule_yml.platforms }}"
+        default_box: "{{ molecule_yml.driver.default_box | default('generic/alpine316') }}"
+        provider_name: "{{ molecule_yml.driver.provider.name | default(omit, true) }}"
+        provision: "{{ molecule_yml.driver.provision | default(omit) }}"
+        cachier: "{{ molecule_yml.driver.cachier | default(omit) }}"
+        parallel: false
+        state: up
+      register: server
+      no_log: false
+
+    - name: Populate instance configuration dictionaries
+      ansible.builtin.set_fact:
+        instance_conf_dict:
+          instance: "{{ item.Host }}"
+          address: "{{ item.HostName }}"
+          user: "{{ item.User }}"
+          port: "{{ item.Port }}"
+          identity_file: "{{ item.IdentityFile }}"
+      loop: "{{ server.results }}"
+      register: instance_config_dict
+
+    - name: Convert instance configuration dictionaries to a list
+      ansible.builtin.set_fact:
+        instance_conf: >-
+          {{
+            instance_config_dict.results
+            | map(attribute='ansible_facts.instance_conf_dict')
+            | list
+          }}
+
+    - name: Write Molecule instance configuration
+      ansible.builtin.copy:
+        content: "{{ instance_conf | to_json | from_json | to_yaml }}"
+        dest: "{{ molecule_instance_config }}"
+        mode: "0600"

molecule/default/molecule.yml

@@ -3,68 +3,73 @@ dependency:
   name: galaxy
 driver:
   name: vagrant
+  # The Vagrant driver warns that parallel VirtualBox creation can cause
+  # platform issues. Keep this five-node, mixed-distribution scenario serial.
+  parallel: false
 platforms:
   - name: control1
-    box: generic/ubuntu2204
+    box: bento/ubuntu-26.04
     memory: 1024
     cpus: 2
     groups:
       - k3s_cluster
       - master
+    # Keep adapter 2 stable across linked-clone rebuilds so stale host-only
+    # neighbor state still identifies the current scenario guest.
+    provider_raw_config_args:
+      - "customize ['modifyvm', :id, '--mac-address2', '080027A13038']"
     interfaces:
       - network_name: private_network
         ip: 192.168.30.38
-    config_options:
-      # We currently can not use public-key based authentication on Ubuntu 22.04,
-      # see: https://github.com/chef/bento/issues/1405
-      ssh.username: vagrant
-      ssh.password: vagrant
 
   - name: control2
-    box: generic/debian12
+    box: bento/debian-13
     memory: 1024
     cpus: 2
     groups:
       - k3s_cluster
       - master
+    provider_raw_config_args:
+      - "customize ['modifyvm', :id, '--mac-address2', '080027A13039']"
     interfaces:
       - network_name: private_network
         ip: 192.168.30.39
 
   - name: control3
-    box: generic/rocky9
+    box: bento/rockylinux-10.1
     memory: 1024
     cpus: 2
     groups:
       - k3s_cluster
       - master
+    provider_raw_config_args:
+      - "customize ['modifyvm', :id, '--mac-address2', '080027A13040']"
     interfaces:
       - network_name: private_network
         ip: 192.168.30.40
 
   - name: node1
-    box: generic/ubuntu2204
+    box: bento/ubuntu-26.04
     memory: 1024
     cpus: 2
     groups:
       - k3s_cluster
       - node
+    provider_raw_config_args:
+      - "customize ['modifyvm', :id, '--mac-address2', '080027A13041']"
     interfaces:
       - network_name: private_network
         ip: 192.168.30.41
-    config_options:
-      # We currently can not use public-key based authentication on Ubuntu 22.04,
-      # see: https://github.com/chef/bento/issues/1405
-      ssh.username: vagrant
-      ssh.password: vagrant
 
   - name: node2
-    box: generic/rocky9
+    box: bento/rockylinux-10.1
     memory: 1024
     cpus: 2
     groups:
       - k3s_cluster
       - node
+    provider_raw_config_args:
+      - "customize ['modifyvm', :id, '--mac-address2', '080027A13042']"
     interfaces:
       - network_name: private_network
         ip: 192.168.30.42
@@ -74,6 +79,7 @@ provisioner:
   env:
     ANSIBLE_VERBOSITY: 1
   playbooks:
+    create: create.yml
     converge: ../resources/converge.yml
     side_effect: ../resources/reset.yml
     verify: ../resources/verify.yml

molecule/default/overrides.yml

@@ -1,12 +1,16 @@
 ---
 - name: Apply overrides
   hosts: all
+  serial: 1
   tasks:
     - name: Override host variables
       ansible.builtin.set_fact:
         # See:
         # https://github.com/flannel-io/flannel/blob/67d603aaf45ef80f5dd39f43714fc5e6f8a637eb/Documentation/troubleshooting.md#Vagrant
-        flannel_iface: eth1
+        flannel_iface: "{{ 'eth1' if 'eth1' in ansible_facts.interfaces else 'enp0s8' }}"
+        # kube-vip cannot infer the cluster interface in these multi-NIC
+        # Vagrant guests because the default route is on eth0.
+        kube_vip_iface: "{{ 'eth1' if 'eth1' in ansible_facts.interfaces else 'enp0s8' }}"
 
         # The test VMs might be a bit slow, so we give them more time to join the cluster:
         retry_count: 45

molecule/default/prepare.yml

@@ -5,18 +5,137 @@
 
 - name: Network setup
   hosts: all
+  vars:
+    primary_master: "{{ groups[group_name_master | default('master')][0] }}"
+    primary_cluster_ip: >-
+      {{ hostvars[primary_master].k3s_node_ip | split(',') | first }}
+    cluster_interface: >-
+      {{ cilium_iface | default(calico_iface | default(flannel_iface)) }}
+    primary_cluster_interface: >-
+      {{ hostvars[primary_master].cilium_iface
+      | default(hostvars[primary_master].calico_iface
+      | default(hostvars[primary_master].flannel_iface)) }}
+    primary_cluster_mac: >-
+      {{ hostvars[primary_master].ansible_facts[primary_cluster_interface].macaddress }}
   tasks:
-    - name: Disable firewalld
-      when: ansible_distribution == "Rocky"
-      # Rocky Linux comes with firewalld enabled. It blocks some of the network
-      # connections needed for our k3s cluster. For our test setup, we just disable
-      # it since the VM host's firewall is still active for connections to and from
-      # the Internet.
+    - name: Gather service facts
+      ansible.builtin.service_facts:
+
+    - name: Disable guest firewall services
+      # The disposable test guests use an isolated VirtualBox network. A distro
+      # firewall can allow ICMP while silently blocking the inter-node Kubernetes
+      # API connection, so disable the known guest firewalls consistently.
       # When building your own cluster, please DO NOT blindly copy this. Instead,
       # please create a custom firewall configuration that fits your network design
       # and security needs.
       ansible.builtin.systemd:
-        name: firewalld
+        name: "{{ item }}"
         enabled: false
         state: stopped
       become: true
+      loop:
+        - firewalld.service
+        - nftables.service
+        - ufw.service
+      when: item in ansible_facts.services
+
+    - name: Verify the private cluster interface
+      ansible.builtin.assert:
+        that:
+          - flannel_iface in ansible_facts
+          - ansible_facts[flannel_iface].ipv4 is defined
+          - ansible_facts[flannel_iface].ipv4.address is defined
+        fail_msg: >-
+          The Vagrant private interface {{ flannel_iface }} does not have an
+          IPv4 address on {{ inventory_hostname }}.
+
+    - name: Pin disposable cluster peer neighbor entries
+      ansible.builtin.command:
+        argv:
+          - ip
+          - neigh
+          - replace
+          - "{{ peer_cluster_ip }}"
+          - lladdr
+          - "{{ peer_cluster_mac }}"
+          - nud
+          - permanent
+          - dev
+          - "{{ cluster_interface }}"
+      become: true
+      changed_when: false
+      loop: "{{ groups['k3s_cluster'] }}"
+      loop_control:
+        label: "{{ inventory_hostname }} -> {{ item }}"
+      vars:
+        peer_cluster_interface: >-
+          {{ hostvars[item].cilium_iface
+          | default(hostvars[item].calico_iface
+          | default(hostvars[item].flannel_iface)) }}
+        peer_cluster_ip: >-
+          {{ hostvars[item].k3s_node_ip | split(',') | first }}
+        peer_cluster_mac: >-
+          {{ hostvars[item].ansible_facts[peer_cluster_interface].macaddress }}
+      when: item != inventory_hostname
+
+    - name: Verify guest-to-guest cluster network reachability
+      ansible.builtin.command:
+        argv:
+          - ping
+          - -c
+          - "1"
+          - -W
+          - "1"
+          - "{{ primary_cluster_ip }}"
+      register: primary_cluster_ping
+      until: primary_cluster_ping.rc == 0
+      retries: 6
+      delay: 2
+      changed_when: false
+
+    - name: Read the primary neighbor entry
+      ansible.builtin.command:
+        argv:
+          - ip
+          - neigh
+          - show
+          - to
+          - "{{ primary_cluster_ip }}"
+          - dev
+          - "{{ cluster_interface }}"
+      register: primary_cluster_neighbor
+      changed_when: false
+      when: inventory_hostname != primary_master
+
+    - name: Verify the primary neighbor identity
+      ansible.builtin.assert:
+        that:
+          - (primary_cluster_mac | lower) in (primary_cluster_neighbor.stdout | lower)
+        fail_msg: >-
+          {{ inventory_hostname }} resolved primary {{ primary_cluster_ip }} to
+          an unexpected MAC on {{ cluster_interface }}. Expected
+          {{ primary_cluster_mac }}, got: {{ primary_cluster_neighbor.stdout }}
+      when: inventory_hostname != primary_master
+
+    - name: Verify GitHub release host DNS
+      ansible.builtin.getent:
+        database: hosts
+        key: github.com
+      register: github_dns
+      retries: 6
+      delay: 5
+      until: github_dns is succeeded
+
+    - name: Verify k3s checksum URL is reachable
+      ansible.builtin.uri:
+        url: >-
+          https://github.com/k3s-io/k3s/releases/download/{{ k3s_version
+          }}/sha256sum-amd64.txt
+        method: HEAD
+        follow_redirects: safe
+        status_code: [200, 302]
+        timeout: 15
+      register: k3s_checksum_request
+      retries: 3
+      delay: 5
+      until: k3s_checksum_request.status in [200, 302]

molecule/default/verify-vars.yml

@@ -0,0 +1,9 @@
+---
+# Durable verify inputs for the default (flannel + MetalLB) scenario.
+# These are plain inventory vars linked into the shared Molecule inventory so
+# the verify play can see them even though the converge play's set_fact values
+# are not persisted between the two Ansible processes.
+verify_cni: flannel
+verify_lb: metallb
+verify_lb_ip_range:
+  - 192.168.30.80-192.168.30.90

molecule/ipv6/molecule.yml

@@ -5,7 +5,7 @@ driver:
   name: vagrant
 platforms:
   - name: control1
-    box: generic/ubuntu2204
+    box: bento/ubuntu-26.04
     memory: 1024
     cpus: 2
     groups:
@@ -14,14 +14,9 @@ platforms:
     interfaces:
       - network_name: private_network
         ip: fdad:bad:ba55::de:11
-    config_options:
-      # We currently can not use public-key based authentication on Ubuntu 22.04,
-      # see: https://github.com/chef/bento/issues/1405
-      ssh.username: vagrant
-      ssh.password: vagrant
 
   - name: control2
-    box: generic/ubuntu2204
+    box: bento/ubuntu-26.04
     memory: 1024
     cpus: 2
     groups:
@@ -30,14 +25,9 @@ platforms:
     interfaces:
       - network_name: private_network
         ip: fdad:bad:ba55::de:12
-    config_options:
-      # We currently can not use public-key based authentication on Ubuntu 22.04,
-      # see: https://github.com/chef/bento/issues/1405
-      ssh.username: vagrant
-      ssh.password: vagrant
 
   - name: node1
-    box: generic/ubuntu2204
+    box: bento/ubuntu-26.04
     memory: 1024
     cpus: 2
     groups:
@@ -46,11 +36,6 @@ platforms:
     interfaces:
       - network_name: private_network
         ip: fdad:bad:ba55::de:21
-    config_options:
-      # We currently can not use public-key based authentication on Ubuntu 22.04,
-      # see: https://github.com/chef/bento/issues/1405
-      ssh.username: vagrant
-      ssh.password: vagrant
 provisioner:
   name: ansible
   env:

molecule/ipv6/overrides.yml

@@ -1,17 +1,18 @@
 ---
 - name: Apply overrides
   hosts: all
+  serial: 1
   tasks:
     - name: Override host variables (1/2)
       ansible.builtin.set_fact:
         # See:
         # https://github.com/flannel-io/flannel/blob/67d603aaf45ef80f5dd39f43714fc5e6f8a637eb/Documentation/troubleshooting.md#Vagrant
-        flannel_iface: eth1
+        flannel_iface: "{{ 'eth1' if 'eth1' in ansible_facts.interfaces else 'enp0s8' }}"
 
         # In this scenario, we have multiple interfaces that the VIP could be
         # broadcasted on. Since we have assigned a dedicated private network
         # here, let's make sure that it is used.
-        kube_vip_iface: eth1
+        kube_vip_iface: "{{ 'eth1' if 'eth1' in ansible_facts.interfaces else 'enp0s8' }}"
 
         # The test VMs might be a bit slow, so we give them more time to join the cluster:
         retry_count: 45

molecule/ipv6/verify-vars.yml

@@ -0,0 +1,7 @@
+---
+# Durable verify inputs for the ipv6 (flannel CNI + MetalLB) scenario.
+verify_cni: flannel
+verify_lb: metallb
+verify_lb_ip_range:
+  - fdad:bad:ba55::1b:0/112
+  - 192.168.123.80-192.168.123.90

molecule/kube-vip/molecule.yml

@@ -5,14 +5,9 @@ driver:
   name: vagrant
 platforms:
   - name: control1
-    box: generic/ubuntu2204
+    box: bento/ubuntu-26.04
     memory: 4096
     cpus: 4
-    config_options:
-      # We currently can not use public-key based authentication on Ubuntu 22.04,
-      # see: https://github.com/chef/bento/issues/1405
-      ssh.username: vagrant
-      ssh.password: vagrant
     groups:
       - k3s_cluster
       - master

molecule/kube-vip/overrides.yml

@@ -1,12 +1,14 @@
 ---
 - name: Apply overrides
   hosts: all
+  serial: 1
   tasks:
     - name: Override host variables
       ansible.builtin.set_fact:
         # See:
         # https://github.com/flannel-io/flannel/blob/67d603aaf45ef80f5dd39f43714fc5e6f8a637eb/Documentation/troubleshooting.md#Vagrant
-        flannel_iface: eth1
+        flannel_iface: "{{ 'eth1' if 'eth1' in ansible_facts.interfaces else 'enp0s8' }}"
+        kube_vip_iface: "{{ 'eth1' if 'eth1' in ansible_facts.interfaces else 'enp0s8' }}"
 
         # The test VMs might be a bit slow, so we give them more time to join the cluster:
         retry_count: 45

molecule/kube-vip/verify-vars.yml

@@ -0,0 +1,9 @@
+---
+# Durable verify inputs for the kube-vip (flannel CNI + kube-vip LB) scenario.
+verify_cni: flannel
+verify_lb: kube-vip
+# The kube-vip cloud provider tag is not defined in the linked sample group
+# vars (its sample entry is commented out), so it is supplied here.
+verify_kube_vip_cloud_provider_tag: v0.0.12
+verify_lb_ip_range:
+  - 192.168.30.110-192.168.30.119

molecule/resources/verify.yml

@@ -1,5 +1,8 @@
 ---
 - name: Verify
   hosts: all
+  vars_files:
+    - >-
+      {{ lookup("ansible.builtin.env", "MOLECULE_SCENARIO_DIRECTORY") }}/verify-vars.yml
   roles:
     - verify_from_outside

molecule/resources/verify_from_outside/tasks/main.yml

@@ -7,6 +7,8 @@
       ansible.builtin.import_tasks: kubecfg-fetch.yml
     - name: "TEST CASE: Get nodes"
       ansible.builtin.include_tasks: test/get-nodes.yml
+    - name: "TEST CASE: Verify components"
+      ansible.builtin.include_tasks: test/verify-components.yml
     - name: "TEST CASE: Deploy example"
       ansible.builtin.include_tasks: test/deploy-example.yml
   always:

molecule/resources/verify_from_outside/tasks/test/deploy-example.yml

@@ -32,17 +32,86 @@
         metallb_port: spec.ports[0].port
       register: nginx_services
 
+    - name: Wait for the load balancer address to be assigned
+      ansible.builtin.set_fact:
+        nginx_lb_ip: >-
+          {{
+            nginx_services.resources[0].status.loadBalancer.ingress[0].ip
+            if (nginx_services.resources | length > 0) and
+               (nginx_services.resources[0].status.loadBalancer.ingress is defined) and
+               (nginx_services.resources[0].status.loadBalancer.ingress | length > 0)
+            else ''
+          }}
+
+    - name: Retry until the load balancer service has an external IP
+      block:
+        - name: Refresh nginx service until it has an assigned address
+          kubernetes.core.k8s_info:
+            kind: service
+            name: nginx
+            namespace: "{{ testing_namespace }}"
+            kubeconfig: "{{ kubecfg_path }}"
+          register: nginx_lb_wait
+          until: >-
+            (nginx_lb_wait.resources | length > 0) and
+            (nginx_lb_wait.resources[0].status.loadBalancer.ingress is defined) and
+            (nginx_lb_wait.resources[0].status.loadBalancer.ingress | length > 0)
+          retries: 30
+          delay: 5
+
+        - name: Record the assigned load balancer address
+          ansible.builtin.set_fact:
+            nginx_lb_ip: >-
+              {{ nginx_lb_wait.resources[0].status.loadBalancer.ingress[0].ip }}
+
     - name: Assert that the nginx welcome page is available
       ansible.builtin.uri:
-        url: http://{{ ip | ansible.utils.ipwrap }}:{{ port_ }}/
+        url: http://{{ nginx_lb_ip | ansible.utils.ipwrap }}:{{ port_ }}/
         return_content: true
       register: result
       failed_when: "'Welcome to nginx!' not in result.content"
       vars:
-        ip: >-
-          {{ nginx_services.resources[0].status.loadBalancer.ingress[0].ip }}
         port_: >-
           {{ nginx_services.resources[0].spec.ports[0].port }}
+
+    - name: Initialize load balancer address range check
+      ansible.builtin.set_fact:
+        lb_addr_in_range: false
+        lb_ip_value: "{{ nginx_lb_ip }}"
+
+    - name: Check load balancer address against start-end pools
+      ansible.builtin.set_fact:
+        lb_addr_in_range: true
+      loop: "{{ verify_lb_ip_range }}"
+      loop_control:
+        label: "{{ item }}"
+      when:
+        - "'-' in item"
+        - "'/' not in item"
+        - >-
+          (lb_ip_value | ansible.utils.ipaddr('int') | int) >=
+          (item.split('-')[0] | ansible.utils.ipaddr('int') | int)
+        - >-
+          (lb_ip_value | ansible.utils.ipaddr('int') | int) <=
+          (item.split('-')[1] | ansible.utils.ipaddr('int') | int)
+
+    - name: Check load balancer address against CIDR pools
+      ansible.builtin.set_fact:
+        lb_addr_in_range: true
+      loop: "{{ verify_lb_ip_range }}"
+      loop_control:
+        label: "{{ item }}"
+      when:
+        - "'/' in item"
+        - (lb_ip_value | ansible.utils.ipaddr(item)) is string
+
+    - name: Assert that the load balancer address is within a configured pool
+      ansible.builtin.assert:
+        that: lb_addr_in_range
+        success_msg: "LoadBalancer address {{ lb_ip_value }} is in a configured range"
+        fail_msg: >-
+          LoadBalancer address {{ lb_ip_value }} is not in a configured
+          range {{ verify_lb_ip_range }}
   # Deactivated linter rules:
   #   - jinja[invalid]: As of version 6.6.0, ansible-lint complains that the input to ipwrap
   #                     would be undefined. This will not be the case during playbook execution.

molecule/resources/verify_from_outside/tasks/test/verify-components.yml

@@ -0,0 +1,313 @@
+---
+# Scenario-aware verification of cluster components and their live image tags.
+# Scenario identity (verify_cni / verify_lb) and expected address range come
+# from each scenario's verify-vars.yml, which is plain inventory data available
+# to the verify play. Converge-time set_fact values are not persisted between
+# the two Ansible processes, so they are never used here.
+- name: Verify cluster components report expected versions
+  block:
+    - name: Get all nodes with their kubelet versions
+      kubernetes.core.k8s_info:
+        kind: node
+        kubeconfig: "{{ kubecfg_path }}"
+      register: verify_nodes
+
+    - name: Assert each node reports the expected Kubernetes version
+      ansible.builtin.assert:
+        that: item.status.nodeInfo.kubeletVersion == k3s_version
+        success_msg: "{{ item.metadata.name }} reports {{ k3s_version }}"
+        fail_msg: >-
+          {{ item.metadata.name }} reports
+          {{ item.status.nodeInfo.kubeletVersion }},
+          expected {{ k3s_version }}
+      loop: "{{ verify_nodes.resources }}"
+      loop_control:
+        label: "{{ item.metadata.name }}"
+
+    - name: Verify Flannel is the active CNI
+      when: verify_cni == 'flannel'
+      block:
+        - name: Assert every node reports Ready
+          ansible.builtin.assert:
+            that: item.status.conditions
+              | selectattr('type', 'equalto', 'Ready')
+              | map(attribute='status') | first | default('') == 'True'
+            success_msg: "{{ item.metadata.name }} is Ready"
+            fail_msg: "{{ item.metadata.name }} is not Ready"
+          loop: "{{ verify_nodes.resources }}"
+          loop_control:
+            label: "{{ item.metadata.name }} ready"
+
+        - name: Get any Calico namespaces with Flannel enabled
+          kubernetes.core.k8s_info:
+            kind: Namespace
+            name: calico-system
+            kubeconfig: "{{ kubecfg_path }}"
+          register: flannel_calico_absent
+
+        - name: Assert there is no Calico system namespace
+          ansible.builtin.assert:
+            that: flannel_calico_absent.resources | length == 0
+            success_msg: "No Calico present with Flannel"
+            fail_msg: "A Calico namespace exists alongside Flannel"
+
+        - name: Get the Cilium namespace with Flannel enabled
+          kubernetes.core.k8s_info:
+            kind: Namespace
+            name: cilium
+            kubeconfig: "{{ kubecfg_path }}"
+          register: flannel_cilium
+
+        - name: Assert the Cilium namespace is absent
+          ansible.builtin.assert:
+            that: flannel_cilium.resources | length == 0
+            success_msg: "No Cilium present with Flannel"
+            fail_msg: "A Cilium namespace exists alongside Flannel"
+
+    - name: Verify Calico is the active CNI
+      when: verify_cni == 'calico'
+      block:
+        - name: Get the Calico node DaemonSet image
+          kubernetes.core.k8s_info:
+            kind: DaemonSet
+            name: calico-node
+            namespace: calico-system
+            kubeconfig: "{{ kubecfg_path }}"
+          register: calico_node_ds
+
+        - name: Assert the Calico node image uses the expected tag
+          ansible.builtin.assert:
+            that:
+              - calico_node_ds.resources | length == 1
+              - calico_node_image | regex_search(':' ~ calico_tag)
+            success_msg: "Calico node image uses tag {{ calico_tag }}"
+            fail_msg: >-
+              Calico node image {{ calico_node_image }},
+              expected {{ calico_tag }}
+          vars:
+            calico_node_image: "{{ calico_node_ds.resources[0].spec.template.spec.containers[0].image }}"
+
+        - name: Get Calico TigeraStatus for calico and apiserver
+          kubernetes.core.k8s_info:
+            api_version: operator.tigera.io/v1
+            kind: TigeraStatus
+            name: "{{ item }}"
+            kubeconfig: "{{ kubecfg_path }}"
+          register: calico_tigerastatus
+          loop:
+            - calico
+            - apiserver
+          loop_control:
+            label: "Tigerastatus/{{ item }}"
+
+        - name: Assert Calico TigeraStatus reports Available
+          ansible.builtin.assert:
+            that: >-
+              item.resources | length == 1 and
+              (item.resources[0].status.conditions
+               | selectattr('type', 'equalto', 'Available')
+               | map(attribute='status') | first | default('')) == 'True'
+            success_msg: "Tigerastatus {{ item.resources[0].metadata.name }} is Available"
+            fail_msg: "Tigerastatus is not Available"
+          loop: "{{ calico_tigerastatus.results }}"
+          loop_control:
+            label: "Tigerastatus Available"
+
+        - name: Get any Flannel DaemonSets with Calico enabled
+          kubernetes.core.k8s_info:
+            kind: DaemonSet
+            namespace: kube-flannel
+            kubeconfig: "{{ kubecfg_path }}"
+          register: no_flannel_ds
+
+        - name: Assert there are no Flannel DaemonSets
+          ansible.builtin.assert:
+            that: no_flannel_ds.resources | length == 0
+            success_msg: "No Flannel DaemonSet present with Calico"
+            fail_msg: "A Flannel DaemonSet exists alongside Calico"
+
+    - name: Verify Cilium is the active CNI
+      when: verify_cni == 'cilium'
+      block:
+        - name: Get the Cilium agent and operator images
+          kubernetes.core.k8s_info:
+            kind: "{{ item.kind }}"
+            name: "{{ item.name }}"
+            namespace: kube-system
+            kubeconfig: "{{ kubecfg_path }}"
+          register: cilium_info
+          loop:
+            - { kind: DaemonSet, name: cilium }
+            - { kind: Deployment, name: cilium-operator }
+          loop_control:
+            label: "{{ item.kind }}/{{ item.name }}"
+
+        - name: Assert Cilium agent and operator use the expected image tag
+          ansible.builtin.assert:
+            that:
+              - cilium_agent_image | regex_search(':' ~ cilium_tag)
+              - cilium_operator_image | regex_search(':' ~ cilium_tag)
+            success_msg: "Cilium agent and operator use {{ cilium_tag }}"
+            fail_msg: >-
+              Cilium agent {{ cilium_agent_image }},
+              operator {{ cilium_operator_image }},
+              expected {{ cilium_tag }}
+          vars:
+            cilium_agent_image: >-
+              {{ (cilium_info.results
+                  | selectattr('resources', 'defined')
+                  | map(attribute='resources')
+                  | list
+                  | map(attribute='0')
+                  | selectattr('kind', 'equalto', 'DaemonSet')
+                  | list)[0].spec.template.spec.containers[0].image }}
+            cilium_operator_image: >-
+              {{ (cilium_info.results
+                  | selectattr('resources', 'defined')
+                  | map(attribute='resources')
+                  | list
+                  | map(attribute='0')
+                  | selectattr('kind', 'equalto', 'Deployment')
+                  | list)[0].spec.template.spec.containers[0].image }}
+
+        - name: Get Hubble relay and UI deployments when enabled
+          kubernetes.core.k8s_info:
+            kind: Deployment
+            name: "{{ item }}"
+            namespace: kube-system
+            kubeconfig: "{{ kubecfg_path }}"
+          register: hubble_info
+          loop:
+            - hubble-relay
+            - hubble-ui
+          loop_control:
+            label: "Deployment/{{ item }}"
+          when: cilium_hubble | bool
+
+        - name: Assert Hubble components are Ready when enabled
+          ansible.builtin.assert:
+            that:
+              - item.resources | length == 1
+              - item.resources[0].status.readyReplicas | default(0) >= 1
+            success_msg: "Hubble deployment {{ item.resources[0].metadata.name }} is Ready"
+            fail_msg: "Hubble deployment is not Ready"
+          loop: "{{ hubble_info.results }}"
+          loop_control:
+            label: "Hubble deployment"
+          when: cilium_hubble | bool
+
+        - name: Get any Flannel DaemonSets with Cilium enabled
+          kubernetes.core.k8s_info:
+            kind: DaemonSet
+            namespace: kube-flannel
+            kubeconfig: "{{ kubecfg_path }}"
+          register: no_flannel_ds_cilium
+
+        - name: Assert there are no Flannel DaemonSets
+          ansible.builtin.assert:
+            that: no_flannel_ds_cilium.resources | length == 0
+            success_msg: "No Flannel DaemonSet present with Cilium"
+            fail_msg: "A Flannel DaemonSet exists alongside Cilium"
+
+    - name: Verify MetalLB is the active load balancer
+      when: verify_lb == 'metallb'
+      block:
+        - name: Get the MetalLB controller and speaker images
+          kubernetes.core.k8s_info:
+            kind: "{{ item.kind }}"
+            name: "{{ item.name }}"
+            namespace: metallb-system
+            kubeconfig: "{{ kubecfg_path }}"
+          register: metallb_info
+          loop:
+            - { kind: Deployment, name: controller }
+            - { kind: DaemonSet, name: speaker }
+          loop_control:
+            label: "{{ item.kind }}/{{ item.name }}"
+
+        - name: Assert MetalLB controller and speaker use the expected image tags
+          ansible.builtin.assert:
+            that:
+              - controller_image | regex_search(metal_lb_controller_tag_version)
+              - speaker_image | regex_search(metal_lb_speaker_tag_version)
+            success_msg: >-
+              MetalLB controller {{ metal_lb_controller_tag_version }},
+              speaker {{ metal_lb_speaker_tag_version }}
+            fail_msg: >-
+              MetalLB controller {{ controller_image }},
+              speaker {{ speaker_image }}
+          vars:
+            controller_image: >-
+              {{ (metallb_info.results
+                  | selectattr('resources', 'defined')
+                  | map(attribute='resources')
+                  | list
+                  | map(attribute='0')
+                  | selectattr('kind', 'equalto', 'Deployment')
+                  | list)[0].spec.template.spec.containers[0].image }}
+            speaker_image: >-
+              {{ (metallb_info.results
+                  | selectattr('resources', 'defined')
+                  | map(attribute='resources')
+                  | list
+                  | map(attribute='0')
+                  | selectattr('kind', 'equalto', 'DaemonSet')
+                  | list)[0].spec.template.spec.containers[0].image }}
+
+    - name: Verify kube-vip is the active load balancer
+      when: verify_lb == 'kube-vip'
+      block:
+        - name: Get the kube-vip and cloud provider images
+          kubernetes.core.k8s_info:
+            kind: "{{ item.kind }}"
+            name: "{{ item.name }}"
+            namespace: kube-system
+            kubeconfig: "{{ kubecfg_path }}"
+          register: kubevip_info
+          loop:
+            - { kind: DaemonSet, name: kube-vip-ds }
+            - { kind: Deployment, name: kube-vip-cloud-provider }
+          loop_control:
+            label: "{{ item.kind }}/{{ item.name }}"
+
+        - name: Assert the kube-vip and cloud provider image tags
+          ansible.builtin.assert:
+            that:
+              - kubevip_image | regex_search(':' ~ kube_vip_tag_version)
+              - cloud_provider_image | regex_search(verify_kube_vip_cloud_provider_tag)
+            success_msg: >-
+              kube-vip {{ kube_vip_tag_version }},
+              cloud provider {{ verify_kube_vip_cloud_provider_tag }}
+            fail_msg: >-
+              kube-vip {{ kubevip_image }},
+              cloud provider {{ cloud_provider_image }}
+          vars:
+            kubevip_image: >-
+              {{ (kubevip_info.results
+                  | selectattr('resources', 'defined')
+                  | map(attribute='resources')
+                  | list
+                  | map(attribute='0')
+                  | selectattr('kind', 'equalto', 'DaemonSet')
+                  | list)[0].spec.template.spec.containers[0].image }}
+            cloud_provider_image: >-
+              {{ (kubevip_info.results
+                  | selectattr('resources', 'defined')
+                  | map(attribute='resources')
+                  | list
+                  | map(attribute='0')
+                  | selectattr('kind', 'equalto', 'Deployment')
+                  | list)[0].spec.template.spec.containers[0].image }}
+
+        - name: Get the MetalLB namespace with kube-vip enabled
+          kubernetes.core.k8s_info:
+            kind: Namespace
+            name: metallb-system
+            kubeconfig: "{{ kubecfg_path }}"
+          register: metallb_absent
+
+        - name: Assert the MetalLB namespace does not exist
+          ansible.builtin.assert:
+            that: metallb_absent.resources | length == 0
+            success_msg: "MetalLB is not installed with kube-vip"
+            fail_msg: "MetalLB namespace exists alongside kube-vip"

molecule/single_node/molecule.yml

@@ -5,14 +5,9 @@ driver:
   name: vagrant
 platforms:
   - name: control1
-    box: generic/ubuntu2204
+    box: bento/ubuntu-26.04
     memory: 4096
     cpus: 4
-    config_options:
-      # We currently can not use public-key based authentication on Ubuntu 22.04,
-      # see: https://github.com/chef/bento/issues/1405
-      ssh.username: vagrant
-      ssh.password: vagrant
     groups:
       - k3s_cluster
       - master

molecule/single_node/overrides.yml

@@ -1,12 +1,14 @@
 ---
 - name: Apply overrides
   hosts: all
+  serial: 1
   tasks:
     - name: Override host variables
       ansible.builtin.set_fact:
         # See:
         # https://github.com/flannel-io/flannel/blob/67d603aaf45ef80f5dd39f43714fc5e6f8a637eb/Documentation/troubleshooting.md#Vagrant
-        flannel_iface: eth1
+        flannel_iface: "{{ 'eth1' if 'eth1' in ansible_facts.interfaces else 'enp0s8' }}"
+        kube_vip_iface: "{{ 'eth1' if 'eth1' in ansible_facts.interfaces else 'enp0s8' }}"
 
         # The test VMs might be a bit slow, so we give them more time to join the cluster:
         retry_count: 45

molecule/single_node/verify-vars.yml

@@ -0,0 +1,6 @@
+---
+# Durable verify inputs for the single_node (flannel + MetalLB) scenario.
+verify_cni: flannel
+verify_lb: metallb
+verify_lb_ip_range:
+  - 192.168.30.91-192.168.30.99

roles/download/tasks/main.yml

@@ -7,6 +7,10 @@
     owner: root
     group: root
     mode: "0755"
+  register: k3s_download_x64
+  retries: 5
+  delay: 10
+  until: k3s_download_x64 is succeeded
   when: ansible_facts.architecture == "x86_64"
 
 - name: Download k3s binary arm64
@@ -17,6 +21,10 @@
     owner: root
     group: root
     mode: "0755"
+  register: k3s_download_arm64
+  retries: 5
+  delay: 10
+  until: k3s_download_arm64 is succeeded
   when:
     - ( ansible_facts.architecture is search("arm") and ansible_facts.userspace_bits == "64" )
       or ansible_facts.architecture is search("aarch64")
@@ -29,6 +37,10 @@
     owner: root
     group: root
     mode: "0755"
+  register: k3s_download_armhf
+  retries: 5
+  delay: 10
+  until: k3s_download_armhf is succeeded
   when:
     - ansible_facts.architecture is search("arm")
     - ansible_facts.userspace_bits == "32"

roles/k3s_server/defaults/main.yml

@@ -7,8 +7,8 @@ group_name_master: master
 
 kube_vip_arp: true
 kube_vip_iface:
-kube_vip_cloud_provider_tag_version: main
-kube_vip_tag_version: v0.7.2
+kube_vip_cloud_provider_tag_version: v0.0.12
+kube_vip_tag_version: v1.2.2
 
 kube_vip_bgp: false
 kube_vip_bgp_routerid: 127.0.0.1
@@ -19,8 +19,8 @@ kube_vip_bgp_peeras: "64512"
 kube_vip_bgp_peers: []
 kube_vip_bgp_peers_groups: ['k3s_master']
 
-metal_lb_controller_tag_version: v0.14.3
-metal_lb_speaker_tag_version: v0.14.3
+metal_lb_controller_tag_version: v0.16.0
+metal_lb_speaker_tag_version: v0.16.0
 metal_lb_type: native
 
 retry_count: 20

roles/k3s_server/meta/main.yml

@@ -80,11 +80,11 @@ argument_specs:
 
       kube_vip_tag_version:
         description: Image tag for kube-vip
-        default: v0.7.2
+        default: v1.2.2
 
       kube_vip_cloud_provider_tag_version:
         description: Tag for kube-vip-cloud-provider manifest when enabled
-        default: main
+        default: v0.0.12
 
       kube_vip_lb_ip_range:
         description: IP range for kube-vip load balancer
@@ -92,11 +92,11 @@ argument_specs:
 
       metal_lb_controller_tag_version:
         description: Image tag for MetalLB
-        default: v0.14.3
+        default: v0.16.0
 
       metal_lb_speaker_tag_version:
         description: Image tag for MetalLB
-        default: v0.14.3
+        default: v0.16.0
 
       metal_lb_type:
         choices:

roles/k3s_server/tasks/fetch_k3s_init_logs.yml

@@ -4,10 +4,110 @@
 
 - name: Fetch k3s-init.service logs
   ansible.builtin.command:
-    cmd: journalctl --all --unit=k3s-init.service
+    cmd: >-
+      timeout --signal=TERM --kill-after=5s 30s journalctl --no-pager
+      --unit=k3s-init.service --since=-30min --lines=5000
   changed_when: false
+  failed_when: false
+  no_log: true
   register: k3s_init_log
 
+- name: Fetch k3s-init.service status
+  ansible.builtin.command:
+    cmd: >-
+      timeout --signal=TERM --kill-after=5s 15s systemctl status
+      k3s-init.service --no-pager --full
+  changed_when: false
+  failed_when: false
+  no_log: true
+  register: k3s_init_status
+
+- name: Fetch IP address state
+  ansible.builtin.command:
+    cmd: ip -br address
+  changed_when: false
+  failed_when: false
+  no_log: true
+  register: k3s_ip_address_state
+
+- name: Fetch IP link state
+  ansible.builtin.command:
+    cmd: ip -br link
+  changed_when: false
+  failed_when: false
+  no_log: true
+  register: k3s_ip_link_state
+
+- name: Fetch IP route state
+  ansible.builtin.command:
+    cmd: ip route show
+  changed_when: false
+  failed_when: false
+  no_log: true
+  register: k3s_ip_route_state
+
+- name: Fetch IP neighbor state
+  ansible.builtin.command:
+    cmd: ip neigh show
+  changed_when: false
+  failed_when: false
+  no_log: true
+  register: k3s_ip_neighbor_state
+
+- name: Fetch IP rule state
+  ansible.builtin.command:
+    cmd: ip rule show
+  changed_when: false
+  failed_when: false
+  no_log: true
+  register: k3s_ip_rule_state
+
+- name: Fetch listening TCP sockets
+  ansible.builtin.command:
+    cmd: ss -ltn
+  changed_when: false
+  failed_when: false
+  no_log: true
+  register: k3s_tcp_listener_state
+
+- name: Fetch guest firewall service state
+  ansible.builtin.service_facts:
+  no_log: true
+  register: k3s_service_facts
+
+- name: Fetch input firewall rules
+  ansible.builtin.command:
+    cmd: iptables -S INPUT
+  changed_when: false
+  failed_when: false
+  no_log: true
+  register: k3s_input_firewall_rules
+
+- name: Ping the primary Kubernetes API address from {{ ansible_hostname }}
+  ansible.builtin.command:
+    argv:
+      - ping
+      - -c
+      - "1"
+      - -W
+      - "1"
+      - "{{ hostvars[groups[group_name_master | default('master')][0]].k3s_node_ip | split(',') | first }}"
+  changed_when: false
+  failed_when: false
+  no_log: true
+  register: k3s_primary_api_ping
+
+- name: Probe primary Kubernetes API from {{ ansible_hostname }}
+  ansible.builtin.wait_for:
+    host: "{{ hostvars[groups[group_name_master | default('master')][0]].k3s_node_ip | split(',') | first }}"
+    port: 6443
+    connect_timeout: 1
+    timeout: 2
+  changed_when: false
+  failed_when: false
+  no_log: true
+  register: k3s_primary_api_probe
+
 - name: Create {{ log_destination }}
   delegate_to: localhost
   run_once: true
@@ -25,4 +125,56 @@
     dest: "{{ log_destination }}/k3s-init@{{ ansible_hostname }}.log"
     mode: "0644"
   vars:
-    content: "{{ k3s_init_log.stdout }}"
+    content: |
+      === k3s-init.service status ===
+      {{ k3s_init_status.stdout | regex_replace('--token(?:=| +)[^ ]+', '--token ***') }}
+
+      === k3s-init.service journal ===
+      {{ k3s_init_log.stdout | regex_replace('--token(?:=| +)[^ ]+', '--token ***') }}
+
+- name: Store network diagnostics to {{ log_destination }}
+  delegate_to: localhost
+  become: false
+  ansible.builtin.template:
+    src: content.j2
+    dest: "{{ log_destination }}/network@{{ ansible_hostname }}.log"
+    mode: "0644"
+  vars:
+    content: |
+      === ip -br address ===
+      {{ k3s_ip_address_state.stdout }}
+
+      === ip -br link ===
+      {{ k3s_ip_link_state.stdout }}
+
+      === ip route show ===
+      {{ k3s_ip_route_state.stdout }}
+
+      === ip neigh show ===
+      {{ k3s_ip_neighbor_state.stdout }}
+
+      === ip rule show ===
+      {{ k3s_ip_rule_state.stdout }}
+
+      === ss -ltn ===
+      {{ k3s_tcp_listener_state.stdout }}
+
+      === guest firewall services ===
+      firewalld={{ k3s_service_facts.ansible_facts.services.get('firewalld.service', {}).get('state', 'not-found') }}
+      nftables={{ k3s_service_facts.ansible_facts.services.get('nftables.service', {}).get('state', 'not-found') }}
+      ufw={{ k3s_service_facts.ansible_facts.services.get('ufw.service', {}).get('state', 'not-found') }}
+
+      === iptables -S INPUT ===
+      rc={{ k3s_input_firewall_rules.rc }}
+      {{ k3s_input_firewall_rules.stdout }}
+      {{ k3s_input_firewall_rules.stderr }}
+
+      === primary API ping ===
+      rc={{ k3s_primary_api_ping.rc }}
+      {{ k3s_primary_api_ping.stdout }}
+      {{ k3s_primary_api_ping.stderr }}
+
+      === primary API probe ===
+      failed={{ k3s_primary_api_probe.failed | default(false) }}
+      elapsed={{ k3s_primary_api_probe.elapsed | default('unknown') }}
+      msg={{ k3s_primary_api_probe.msg | default('connected') }}

roles/k3s_server/tasks/join_master.yml

@@ -0,0 +1,34 @@
+---
+- name: Verify primary Kubernetes API reachability from {{ joining_master }}
+  ansible.builtin.wait_for:
+    host: >-
+      {{ hostvars[groups[group_name_master | default('master')][0]].k3s_node_ip
+      | split(',') | first }}
+    port: 6443
+    connect_timeout: 2
+    timeout: 30
+  delegate_to: "{{ joining_master }}"
+
+- name: Join transient k3s-init service for {{ joining_master }}
+  ansible.builtin.command:
+    cmd: >-
+      systemd-run -p RestartSec=2 -p Restart=on-failure
+      -p Delegate=yes -p TasksMax=infinity -p KillMode=process
+      -p LimitNOFILE=1048576 -p LimitNPROC=infinity -p LimitCORE=infinity
+      --unit=k3s-init
+      k3s server {{ hostvars[joining_master].k3s_server_init_args }}
+    creates: "{{ systemd_dir }}/k3s-init.service"
+  delegate_to: "{{ joining_master }}"
+  no_log: true
+
+- name: Wait for primary registration of {{ joining_master }}
+  ansible.builtin.command:
+    cmd: >-
+      {{ k3s_kubectl_binary | default('k3s kubectl') }} get node
+      {{ hostvars[joining_master].ansible_hostname }}
+  delegate_to: "{{ groups[group_name_master | default('master')][0] }}"
+  register: joined_master
+  until: joined_master.rc == 0
+  retries: "{{ retry_count | default(20) }}"
+  delay: 2
+  changed_when: false

roles/k3s_server/tasks/kube-vip.yml

@@ -10,7 +10,7 @@
 
 - name: Download vip cloud provider manifest to first master
   ansible.builtin.get_url:
-    url: https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/{{ kube_vip_cloud_provider_tag_version | default('main') }}/manifest/kube-vip-cloud-controller.yaml # noqa yaml[line-length]
+    url: https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/{{ kube_vip_cloud_provider_tag_version | default('v0.0.12') }}/manifest/kube-vip-cloud-controller.yaml # noqa yaml[line-length]
     dest: /var/lib/rancher/k3s/server/manifests/kube-vip-cloud-controller.yaml
     owner: root
     group: root

roles/k3s_server/tasks/main.yml

@@ -34,22 +34,87 @@
   tags: kubevip
   when: kube_vip_lb_ip_range is defined
 
-- name: Init cluster inside the transient k3s-init service
-  ansible.builtin.command:
-    cmd: systemd-run -p RestartSec=2 -p Restart=on-failure --unit=k3s-init k3s server {{ server_init_args }}
-    creates: "{{ systemd_dir }}/k3s-init.service"
-
-- name: Verification
+- name: Initialize and verify the K3s control plane
+  any_errors_fatal: true
   when: not ansible_check_mode
   block:
-    - name: Verify that all nodes actually joined (check k3s-init.service if this fails)
-      ansible.builtin.command:
-        cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} get nodes -l 'node-role.kubernetes.io/master=true' -o=jsonpath='{.items[*].metadata.name}'" # yamllint disable-line rule:line-length
-      register: nodes
-      until: nodes.rc == 0 and (nodes.stdout.split() | length) == (groups[group_name_master | default('master')] | length) # yamllint disable-line rule:line-length
-      retries: "{{ retry_count | default(20) }}"
-      delay: 10
-      changed_when: false
+    - name: Materialize per-host server initialization arguments
+      ansible.builtin.set_fact:
+        k3s_server_init_args: "{{ server_init_args }}"
+      no_log: true
+      when: groups[group_name_master | default('master')] | length > 1
+
+    - name: Orchestrate control-plane initialization from the first master
+      when: inventory_hostname == groups[group_name_master | default('master')][0]
+      block:
+        - name: Init the first master inside the transient k3s-init service
+          ansible.builtin.command:
+            cmd: >-
+              systemd-run -p RestartSec=2 -p Restart=on-failure
+              -p Delegate=yes -p TasksMax=infinity -p KillMode=process
+              -p LimitNOFILE=1048576 -p LimitNPROC=infinity -p LimitCORE=infinity
+              --unit=k3s-init k3s server {{ server_init_args }}
+            creates: "{{ systemd_dir }}/k3s-init.service"
+          no_log: true
+
+        - name: Wait for the first master Kubernetes API
+          ansible.builtin.command:
+            cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} get --raw=/readyz"
+          register: first_master_api
+          until: first_master_api.rc == 0
+          retries: "{{ retry_count | default(20) }}"
+          delay: 2
+          changed_when: false
+
+        - name: Verify the first master API listener on its node address
+          ansible.builtin.wait_for:
+            host: >-
+              {{ hostvars[groups[group_name_master | default('master')][0]].k3s_node_ip
+              | split(',') | first }}
+            port: 6443
+            connect_timeout: 2
+            timeout: "{{ retry_count | default(20) * 2 }}"
+
+        - name: Join additional masters one at a time
+          ansible.builtin.include_tasks: join_master.yml
+          loop: "{{ groups[group_name_master | default('master')][1:] }}"
+          loop_control:
+            loop_var: joining_master
+          when: groups[group_name_master | default('master')] | length > 1
+
+        - name: Verify that all expected control-plane nodes registered
+          ansible.builtin.command:
+            cmd: >-
+              {{ k3s_kubectl_binary | default('k3s kubectl') }} get nodes
+              -o=jsonpath='{.items[*].metadata.name}'
+          register: nodes
+          until:
+            - nodes.rc == 0
+            - expected_control_plane_names | difference(nodes.stdout.split()) | length == 0
+          retries: "{{ retry_count | default(20) }}"
+          delay: 2
+          changed_when: false
+          vars:
+            expected_control_plane_names: >-
+              {{
+                groups[group_name_master | default('master')]
+                | map('extract', hostvars, 'ansible_hostname')
+                | list
+              }}
+
+        - name: Wait for K3s bootstrap CRDs to become established
+          ansible.builtin.command:
+            cmd: >-
+              {{ k3s_kubectl_binary | default('k3s kubectl') }} wait
+              --for=condition=Established --timeout=5s
+              crd/addons.k3s.cattle.io
+              crd/helmcharts.helm.cattle.io
+              crd/helmchartconfigs.helm.cattle.io
+          register: bootstrap_crds
+          until: bootstrap_crds.rc == 0
+          retries: "{{ retry_count | default(20) }}"
+          delay: 2
+          changed_when: false
   always:
     - name: Save logs of k3s-init.service
       ansible.builtin.include_tasks: fetch_k3s_init_logs.yml

roles/k3s_server/tasks/metallb.yml

@@ -28,3 +28,29 @@
   loop_control:
     label: "{{ item.change }} => {{ item.to }}"
   when: ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname']
+
+- name: Read back MetalLB manifest from first master
+  ansible.builtin.slurp:
+    src: /var/lib/rancher/k3s/server/manifests/metallb-crds.yaml
+  register: metallb_manifest
+  when: ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname']
+
+- name: Check the MetalLB speaker image reference is present
+  ansible.builtin.set_fact:
+    metallb_manifest_has_speaker_tag: >-
+      {{
+        ('quay.io/metallb/speaker:' ~ metal_lb_speaker_tag_version) in
+        (metallb_manifest.content | default('') | b64decode)
+      }}
+  when: ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname']
+
+- name: Fail if MetalLB speaker tag was not applied to the manifest
+  ansible.builtin.fail:
+    msg: >-
+      MetalLB speaker image reference
+      quay.io/metallb/speaker:{{ metal_lb_speaker_tag_version }}
+      was not found in the downloaded manifest. The upstream image
+      reference may have changed.
+  when:
+    - ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname']
+    - not metallb_manifest_has_speaker_tag

roles/k3s_server/tasks/vip.yml

@@ -12,9 +12,9 @@
     mode: "0644"
   when: ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname']
 
-- name: Download vip rbac manifest to first master
-  ansible.builtin.get_url:
-    url: https://kube-vip.io/manifests/rbac.yaml
+- name: Copy kube-vip RBAC manifest to first master
+  ansible.builtin.template:
+    src: kube-vip-rbac.yaml.j2
     dest: /var/lib/rancher/k3s/server/manifests/vip-rbac.yaml
     owner: root
     group: root

roles/k3s_server/templates/kube-vip-rbac.yaml.j2

@@ -0,0 +1,48 @@
+# Repository-owned kube-vip RBAC derived from the official v1.2.2
+# `kube-vip manifest rbac` output (run with `--inCluster`).
+# Source: https://github.com/kube-vip/kube-vip/blob/v1.2.2/pkg/kubevip/config_generator.go
+# Permissions are kept exactly as broad as the upstream manifest.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: kube-vip
+  namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  annotations:
+    rbac.authorization.kubernetes.io/autoupdate: "true"
+  name: system:kube-vip-role
+rules:
+  - apiGroups: [""]
+    resources: ["services/status"]
+    verbs: ["update"]
+  - apiGroups: [""]
+    resources: ["services", "endpoints"]
+    verbs: ["list", "get", "watch", "update"]
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["list", "get", "watch", "update", "patch"]
+  - apiGroups: ["coordination.k8s.io"]
+    resources: ["leases"]
+    verbs: ["list", "get", "watch", "update", "create"]
+  - apiGroups: ["discovery.k8s.io"]
+    resources: ["endpointslices"]
+    verbs: ["list", "get", "watch", "update"]
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["list"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: system:kube-vip-binding
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: system:kube-vip-role
+subjects:
+  - kind: ServiceAccount
+    name: kube-vip
+    namespace: kube-system

roles/k3s_server/templates/vip.yaml.j2

@@ -36,7 +36,7 @@ spec:
         - name: vip_interface
           value: {{ kube_vip_iface }}
 {% endif %}
-        - name: vip_cidr
+        - name: vip_subnet
           value: "{{ apiserver_endpoint | ansible.utils.ipsubnet | ansible.utils.ipaddr('prefix') }}"
         - name: cp_enable
           value: "true"
@@ -62,8 +62,8 @@ spec:
           value: "{{ kube_vip_bgp_routerid }}"
 {% endif %}
 {% if _kube_vip_bgp_peers | length > 0 %}
-        - name: bgppeers
-          value: "{{ _kube_vip_bgp_peers | map(attribute='peer_address') | zip(_kube_vip_bgp_peers| map(attribute='peer_asn')) | map('join', ',') | join(':') }}"  # yamllint disable-line rule:line-length
+        - name: bgp_peers
+          value: "{{ _kube_vip_bgp_peers | map(attribute='peer_address') | zip(_kube_vip_bgp_peers | map(attribute='peer_asn')) | map('join', ':') | join(',') }}"  # yamllint disable-line rule:line-length
 {% else %}
 {% if kube_vip_bgp_as is defined %}
         - name: bgp_as

roles/k3s_server_post/defaults/main.yml

@@ -9,7 +9,7 @@ calico_ebpf: false
 calico_encapsulation: VXLANCrossSubnet
 calico_natOutgoing: Enabled  # noqa var-naming
 calico_nodeSelector: all()  # noqa var-naming
-calico_tag: v3.27.2
+calico_tag: v3.32.1
 
 cilium_bgp: false
 cilium_exportPodCIDR: true  # noqa var-naming
@@ -20,6 +20,8 @@ cilium_bgp_neighbors_groups: ['k3s_all']
 cilium_bgp_lb_cidr: 192.168.31.0/24
 cilium_hubble: true
 cilium_mode: native
+cilium_tag: v1.20.0
+cilium_cli_tag: v0.19.7
 
 cluster_cidr: 10.52.0.0/16
 enable_bpf_masquerade: true
@@ -28,5 +30,5 @@ group_name_master: master
 
 metal_lb_mode: layer2
 metal_lb_available_timeout: 240s
-metal_lb_controller_tag_version: v0.14.3
+metal_lb_controller_tag_version: v0.16.0
 metal_lb_ip_range: 192.168.30.80-192.168.30.90

roles/k3s_server_post/meta/main.yml

@@ -43,7 +43,7 @@ argument_specs:
 
       calico_tag:
         description: Calico version tag
-        default: v3.27.2
+        default: v3.32.1
 
       cilium_bgp:
         description:
@@ -93,12 +93,24 @@ argument_specs:
         default: ~
 
       cilium_mode:
-        description: Inner-node communication mode
+        description:
+          - Inner-node communication mode.
+          - Cilium accepts `native` or `tunnel`. `routed` is accepted as a
+            legacy alias and is mapped to `tunnel` at install time.
         default: native
         choices:
           - native
+          - tunnel
           - routed
 
+      cilium_tag:
+        description: Cilium version tag
+        default: v1.20.0
+
+      cilium_cli_tag:
+        description: Cilium CLI version tag
+        default: v0.19.7
+
       cluster_cidr:
         description: Inner-cluster IP range
         default: 10.52.0.0/16
@@ -131,7 +143,7 @@ argument_specs:
 
       metal_lb_controller_tag_version:
         description: Image tag for MetalLB
-        default: v0.14.3
+        default: v0.16.0
 
       metal_lb_mode:
         description: Metallb mode

roles/k3s_server_post/tasks/calico.yml

@@ -11,6 +11,14 @@
         group: root
         mode: "0755"
 
+    - name: "Download to first master: Calico CRD bundle for {{ calico_tag }}"
+      ansible.builtin.get_url:
+        url: https://raw.githubusercontent.com/projectcalico/calico/{{ calico_tag }}/manifests/v1_crd_projectcalico_org.yaml # noqa yaml[line-length]
+        dest: /tmp/k3s/v1_crd_projectcalico_org.yaml
+        owner: root
+        group: root
+        mode: "0755"
+
     - name: "Download to first master: manifest for Tigera Operator and Calico CRDs"
       ansible.builtin.get_url:
         url: https://raw.githubusercontent.com/projectcalico/calico/{{ calico_tag }}/manifests/tigera-operator.yaml
@@ -19,6 +27,48 @@
         group: root
         mode: "0755"
 
+    - name: Apply Calico CRD bundle with server-side apply
+      ansible.builtin.command: >-
+        {{ k3s_kubectl_binary | default('k3s kubectl') }}
+        apply --server-side --force-conflicts
+        -f /tmp/k3s/v1_crd_projectcalico_org.yaml
+      register: apply_crds
+      changed_when: >-
+        'created' in apply_crds.stdout or
+        'configured' in apply_crds.stdout or
+        'applied' in apply_crds.stdout
+      failed_when: apply_crds.rc != 0
+
+    - name: Apply Tigera Operator manifest idempotently with server-side apply
+      ansible.builtin.command: >-
+        {{ k3s_kubectl_binary | default('k3s kubectl') }}
+        apply --server-side --force-conflicts
+        -f /tmp/k3s/tigera-operator.yaml
+      register: apply_operator
+      changed_when: >-
+        'created' in apply_operator.stdout or
+        'configured' in apply_operator.stdout or
+        'applied' in apply_operator.stdout
+      failed_when: apply_operator.rc != 0
+
+    - name: Wait for Tigera Operator and managed CRDs to become available
+      ansible.builtin.command: >-
+        {{ k3s_kubectl_binary | default('k3s kubectl') }} wait {{ item.type }}/{{ item.name }}
+        {% if item.namespace is defined %}--namespace='{{ item.namespace }}'{% endif %}
+        --for=condition={{ item.condition }}
+        --timeout=30s
+      register: tigera_result
+      changed_when: false
+      until: tigera_result is succeeded
+      retries: 7
+      delay: 7
+      with_items:
+        - { name: tigera-operator, type: deployment, namespace: tigera-operator, condition: Available=True }
+        - { name: installations.operator.tigera.io, type: crd, condition: Established }
+        - { name: apiservers.operator.tigera.io, type: crd, condition: Established }
+      loop_control:
+        label: "{{ item.type }}/{{ item.name }}"
+
     - name: Copy Calico custom resources manifest to first master
       ansible.builtin.template:
         src: calico.crs.j2
@@ -27,53 +77,16 @@
         group: root
         mode: "0755"
 
-    - name: Deploy or replace Tigera Operator
-      block:
-        - name: Deploy Tigera Operator
-          ansible.builtin.command:
-            cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} create -f /tmp/k3s/tigera-operator.yaml"
-          register: create_operator
-          changed_when: "'created' in create_operator.stdout"
-          failed_when: "'Error' in create_operator.stderr and 'already exists' not in create_operator.stderr"
-      rescue:
-        - name: Replace existing Tigera Operator
-          ansible.builtin.command:
-            cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} replace -f /tmp/k3s/tigera-operator.yaml"
-          register: replace_operator
-          changed_when: "'replaced' in replace_operator.stdout"
-          failed_when: "'Error' in replace_operator.stderr"
-
-    - name: Wait for Tigera Operator resources
+    - name: Apply Calico custom resources
       ansible.builtin.command: >-
-        {{ k3s_kubectl_binary | default('k3s kubectl') }} wait {{ item.type }}/{{ item.name }}
-        --namespace='tigera-operator'
-        --for=condition=Available=True
-        --timeout=30s
-      register: tigera_result
-      changed_when: false
-      until: tigera_result is succeeded
-      retries: 7
-      delay: 7
-      with_items:
-        - { name: tigera-operator, type: deployment }
-      loop_control:
-        label: "{{ item.type }}/{{ item.name }}"
-
-    - name: Deploy Calico custom resources
-      block:
-        - name: Deploy custom resources for Calico
-          ansible.builtin.command:
-            cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} create -f /tmp/k3s/custom-resources.yaml"
-          register: create_cr
-          changed_when: "'created' in create_cr.stdout"
-          failed_when: "'Error' in create_cr.stderr and 'already exists' not in create_cr.stderr"
-      rescue:
-        - name: Apply new Calico custom resource manifest
-          ansible.builtin.command:
-            cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} apply -f /tmp/k3s/custom-resources.yaml"
-          register: apply_cr
-          changed_when: "'configured' in apply_cr.stdout or 'created' in apply_cr.stdout"
-          failed_when: "'Error' in apply_cr.stderr"
+        {{ k3s_kubectl_binary | default('k3s kubectl') }}
+        apply -f /tmp/k3s/custom-resources.yaml
+      register: apply_cr
+      changed_when: >-
+        'configured' in apply_cr.stdout or
+        'created' in apply_cr.stdout or
+        'unchanged' in apply_cr.stdout
+      failed_when: apply_cr.rc != 0
 
     - name: Wait for Calico system resources to be available
       ansible.builtin.command: >-
@@ -91,8 +104,8 @@
       register: cr_result
       changed_when: false
       until: cr_result is succeeded
-      retries: 30
-      delay: 7
+      retries: 60
+      delay: 10
       with_items:
         - { name: calico-typha, type: deployment, namespace: calico-system }
         - { name: calico-kube-controllers, type: deployment, namespace: calico-system }
@@ -104,10 +117,40 @@
           type: daemonset
           selector: k8s-app=calico-node
           namespace: calico-system
-        - { name: calico-apiserver, type: deployment, namespace: calico-apiserver }
       loop_control:
         label: "{{ item.type }}/{{ item.name }}"
 
+    # The Calico API server is an optional add-on for managing Calico policy via
+    # kubectl (the projectcalico.org/v3 Kubernetes API). It is not required for
+    # Calico CNI data plane operation and does not reconcile on K3s with recent
+    # Calico releases, so it is treated as best-effort here.
+    - name: Wait for the optional Calico API server (best effort)
+      ansible.builtin.command: >-
+        {{ k3s_kubectl_binary | default('k3s kubectl') }} wait deployment/calico-apiserver
+        --namespace=calico-apiserver --for=condition=Available --timeout=30s
+      register: cr_apiserver
+      changed_when: false
+      until: cr_apiserver is succeeded
+      retries: 30
+      delay: 10
+      ignore_errors: true
+
+    - name: Verify Calico TigeraStatus reports Available
+      ansible.builtin.command: >-
+        {{ k3s_kubectl_binary | default('k3s kubectl') }}
+        get tigerastatus {{ item }}
+        -o jsonpath='{.status.conditions[?(@.type=="Available")].status}'
+      register: tigera_status
+      changed_when: false
+      until: tigera_status.stdout | trim == 'True'
+      retries: 30
+      delay: 7
+      failed_when: tigera_status.rc != 0 or (tigera_status.stdout | trim) != 'True'
+      with_items:
+        - calico
+      loop_control:
+        label: "tigerastatus/{{ item }}"
+
     - name: Patch Felix configuration for eBPF mode
       ansible.builtin.command:
         cmd: >

roles/k3s_server_post/tasks/cilium.yml

@@ -30,26 +30,13 @@
           }}
       when: cilium_cli_installed.rc == 0
 
-    - name: Get latest stable Cilium CLI version file
-      ansible.builtin.get_url:
-        url: https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt
-        dest: /tmp/k3s/cilium-cli-stable.txt
-        owner: root
-        group: root
-        mode: "0755"
-
-    - name: Read Cilium CLI stable version from file
-      ansible.builtin.command: cat /tmp/k3s/cilium-cli-stable.txt
-      register: cli_ver
-      changed_when: false
-
     - name: Log installed Cilium CLI version
       ansible.builtin.debug:
         msg: "Installed Cilium CLI version: {{ installed_cli_version | default('Not installed') }}"
 
-    - name: Log latest stable Cilium CLI version
+    - name: Log pinned Cilium CLI version
       ansible.builtin.debug:
-        msg: "Latest Cilium CLI version: {{ cli_ver.stdout }}"
+        msg: "Pinned Cilium CLI version: {{ cilium_cli_tag }}"
 
     - name: Determine if Cilium CLI needs installation or update
       ansible.builtin.set_fact:
@@ -57,7 +44,7 @@
           {{
             cilium_cli_installed.rc != 0 or
             (cilium_cli_installed.rc == 0 and
-            installed_cli_version != cli_ver.stdout)
+            installed_cli_version != cilium_cli_tag)
           }}
 
     - name: Install or update Cilium CLI
@@ -78,7 +65,7 @@
             - .tar.gz
             - .tar.gz.sha256sum
           vars:
-            cilium_base_url: https://github.com/cilium/cilium-cli/releases/download/{{ cli_ver.stdout }}
+            cilium_base_url: https://github.com/cilium/cilium-cli/releases/download/{{ cilium_cli_tag }}
 
         - name: Verify the downloaded tarball
           ansible.builtin.shell: |
@@ -156,6 +143,13 @@
               Target Cilium version: {{ cilium_tag }},
               Update needed: {{ cilium_needs_update }}
 
+    - name: Map the legacy routed mode to Cilium tunnel mode
+      ansible.builtin.set_fact:
+        # Cilium 1.20 accepts `native` or `tunnel`. `routed` was the name used
+        # by older releases and is kept only as a backward-compatible alias.
+        cilium_routing_mode: >-
+          {{ 'tunnel' if cilium_mode == 'routed' else cilium_mode }}
+
     - name: Install Cilium
       ansible.builtin.command: >-
         {% if cilium_installed.rc != 0 %}
@@ -167,13 +161,13 @@
         --helm-set operator.replicas="1"
         {{ '--helm-set devices=' + cilium_iface if cilium_iface != 'auto' else '' }}
         --helm-set ipam.operator.clusterPoolIPv4PodCIDRList={{ cluster_cidr }}
-        {% if cilium_mode == "native" or (cilium_bgp and cilium_exportPodCIDR != 'false') %}
+        {% if cilium_routing_mode == "native" or (cilium_bgp and cilium_exportPodCIDR != 'false') %}
         --helm-set ipv4NativeRoutingCIDR={{ cluster_cidr }}
         {% endif %}
         --helm-set k8sServiceHost="127.0.0.1"
         --helm-set k8sServicePort="6444"
-        --helm-set routingMode={{ cilium_mode }}
-        --helm-set autoDirectNodeRoutes={{ "true" if cilium_mode == "native" else "false" }}
+        --helm-set routingMode={{ cilium_routing_mode }}
+        --helm-set autoDirectNodeRoutes={{ "true" if cilium_routing_mode == "native" else "false" }}
         --helm-set kubeProxyReplacement={{ kube_proxy_replacement }}
         --helm-set bpf.masquerade={{ enable_bpf_masquerade }}
         --helm-set bgpControlPlane.enabled={{ cilium_bgp | default("false") }}
@@ -181,8 +175,8 @@
         --helm-set hubble.relay.enabled={{ "true" if cilium_hubble else "false" }}
         --helm-set hubble.ui.enabled={{ "true" if cilium_hubble else "false" }}
         {% if kube_proxy_replacement is not false %}
-        --helm-set bpf.loadBalancer.algorithm={{ bpf_lb_algorithm }}
-        --helm-set bpf.loadBalancer.mode={{ bpf_lb_mode }}
+        --helm-set loadBalancer.algorithm={{ bpf_lb_algorithm }}
+        --helm-set loadBalancer.mode={{ bpf_lb_mode }}
         {% endif %}
       environment:
         KUBECONFIG: "{{ ansible_user_dir }}/.kube/config"
@@ -218,6 +212,16 @@
       when: >-
         not item.check_hubble | default(false) or (item.check_hubble | default(false) and cilium_hubble)
 
+    - name: Wait for Cilium status to be healthy
+      ansible.builtin.command: cilium status --wait
+      environment:
+        KUBECONFIG: "{{ ansible_user_dir }}/.kube/config"
+      register: cilium_status
+      changed_when: false
+      until: cilium_status.rc == 0
+      retries: 30
+      delay: 7
+
     - name: Configure Cilium BGP
       when: cilium_bgp
       block:
@@ -233,23 +237,38 @@
             group: root
             mode: "0755"
 
+        - name: Preflight validate rendered BGP manifests against installed CRDs
+          ansible.builtin.command: >-
+            {{ k3s_kubectl_binary | default('k3s kubectl') }}
+            apply --dry-run=server -f /tmp/k3s/cilium-bgp.yaml
+          register: preflight_cr
+          changed_when: false
+          failed_when: preflight_cr.rc != 0
+
         - name: Apply BGP manifests
-          ansible.builtin.command:
-            cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} apply -f /tmp/k3s/cilium-bgp.yaml"
+          ansible.builtin.command: >-
+            {{ k3s_kubectl_binary | default('k3s kubectl') }}
+            apply -f /tmp/k3s/cilium-bgp.yaml
           register: apply_cr
           changed_when: "'configured' in apply_cr.stdout or 'created' in apply_cr.stdout"
-          failed_when: "'is invalid' in apply_cr.stderr"
-          ignore_errors: true
+          failed_when: apply_cr.rc != 0
 
-        - name: Print error message if BGP manifests application fails
-          ansible.builtin.debug:
-            msg: "{{ apply_cr.stderr }}"
-          when: "'is invalid' in apply_cr.stderr"
+        - name: Remove deprecated CiliumBGPPeeringPolicy after v2 resources are accepted
+          ansible.builtin.command: >-
+            {{ k3s_kubectl_binary | default('k3s kubectl') }}
+            delete CiliumBGPPeeringPolicy.cilium.io 01-bgp-peering-policy
+          register: delete_old_policy
+          changed_when: "'deleted' in delete_old_policy.stdout"
+          # The policy (and possibly its CRD) may already be absent; this is
+          # intentionally tolerated, not a command whose failure must be hidden.
+          failed_when: false
 
         - name: Test for BGP config resources
           ansible.builtin.command: "{{ item }}"
           loop:
-            - "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumBGPPeeringPolicy.cilium.io"
+            - "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumBGPClusterConfig.cilium.io"
+            - "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumBGPPeerConfig.cilium.io"
+            - "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumBGPAdvertisement.cilium.io"
             - "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumLoadBalancerIPPool.cilium.io"
           changed_when: false
           loop_control:

roles/k3s_server_post/tasks/metallb.yml

@@ -85,17 +85,9 @@
 
 - name: Set metallb webhook service name
   ansible.builtin.set_fact:
-    metallb_webhook_service_name: >-
-      {{
-        (
-          (metal_lb_controller_tag_version | regex_replace('^v', ''))
-          is
-          version('0.14.4', '<', version_type='semver')
-        ) | ternary(
-          'webhook-service',
-          'metallb-webhook-service'
-        )
-      }}
+    # Inspected the v0.16.0 manifest: the newer webhook Service name is used
+    # by every supported MetalLB release, so the old pre-0.14.4 branch is gone.
+    metallb_webhook_service_name: metallb-webhook-service
 
 - name: Test metallb-system webhook-service endpoint
   ansible.builtin.command: >-

roles/k3s_server_post/templates/cilium.crs.j2

@@ -1,40 +1,84 @@
-apiVersion: "cilium.io/v2alpha1"
-kind: CiliumBGPPeeringPolicy
+# Cilium BGP Control Plane v2 resources.
+# Replace the deprecated v2alpha1 CiliumBGPPeeringPolicy that was removed
+# in Cilium 1.19.
+{% set _cilium_default_peer = {'peer_address': cilium_bgp_peer_address, 'peer_asn': cilium_bgp_peer_asn} %}
+{% set _cilium_peers = _cilium_bgp_neighbors if _cilium_bgp_neighbors | length > 0 else [_cilium_default_peer] %}
+apiVersion: cilium.io/v2
+kind: CiliumBGPPeerConfig
 metadata:
- name: 01-bgp-peering-policy
-spec: # CiliumBGPPeeringPolicySpec
- virtualRouters: # []CiliumBGPVirtualRouter
- - localASN: {{ cilium_bgp_my_asn }}
-   exportPodCIDR: {{ cilium_exportPodCIDR | default('true') }}
-   neighbors: # []CiliumBGPNeighbor
-{% if _cilium_bgp_neighbors | length > 0 %}
-{% for item in _cilium_bgp_neighbors %}
-    - peerAddress: '{{ item.peer_address + "/32"}}'
-      peerASN: {{ item.peer_asn }}
-      eBGPMultihopTTL: 10
-      connectRetryTimeSeconds: 120
-      holdTimeSeconds: 90
-      keepAliveTimeSeconds: 30
-      gracefulRestart:
-        enabled: true
-        restartTimeSeconds: 120
-{% endfor %}
-{% else %}
-    - peerAddress: '{{ cilium_bgp_peer_address + "/32"}}'
-      peerASN: {{ cilium_bgp_peer_asn }}
-      eBGPMultihopTTL: 10
-      connectRetryTimeSeconds: 120
-      holdTimeSeconds: 90
-      keepAliveTimeSeconds: 30
-      gracefulRestart:
-        enabled: true
-        restartTimeSeconds: 120
-{% endif %}
-   serviceSelector:
-      matchExpressions:
-         - {key: somekey, operator: NotIn, values: ['never-used-value']}
+  name: cilium-peer
+spec:
+  # Matches the timers and multihop used by the previous v2alpha1 policy.
+  ebgpMultihop: 10
+  timers:
+    connectRetryTimeSeconds: 120
+    holdTimeSeconds: 90
+    keepAliveTimeSeconds: 30
+  gracefulRestart:
+    enabled: true
+    restartTimeSeconds: 120
+  families:
+    - afi: ipv4
+      safi: unicast
+      advertisements:
+        matchLabels:
+          advertise: "bgp"
 ---
-apiVersion: "cilium.io/v2alpha1"
+apiVersion: cilium.io/v2
+kind: CiliumBGPClusterConfig
+metadata:
+  name: cilium-bgp
+spec:
+  # Explicitly select every node so the BGP instance runs across the cluster.
+  nodeSelector:
+    matchExpressions:
+      - key: somekey
+        operator: NotIn
+        values: ['never-used-value']
+  bgpInstances:
+    - name: "instance-{{ cilium_bgp_my_asn }}"
+      localASN: {{ cilium_bgp_my_asn }}
+      peers:
+{% for peer in _cilium_peers %}
+        - name: "peer-{{ peer.peer_asn }}-{{ loop.index }}"
+          peerASN: {{ peer.peer_asn }}
+          peerAddress: {{ peer.peer_address }}
+          peerConfigRef:
+            name: cilium-peer
+{% endfor %}
+{% if cilium_exportPodCIDR %}
+---
+apiVersion: cilium.io/v2
+kind: CiliumBGPAdvertisement
+metadata:
+  name: cilium-pod-cidrs
+  labels:
+    advertise: "bgp"
+spec:
+  advertisements:
+    - advertisementType: "PodCIDR"
+{% endif %}
+---
+apiVersion: cilium.io/v2
+kind: CiliumBGPAdvertisement
+metadata:
+  name: cilium-lb-services
+  labels:
+    advertise: "bgp"
+spec:
+  advertisements:
+    - advertisementType: "Service"
+      service:
+        addresses:
+          - LoadBalancerIP
+      # Advertise all Services carrying an ingress address from the pool.
+      selector:
+        matchExpressions:
+          - key: somekey
+            operator: NotIn
+            values: ['never-used-value']
+---
+apiVersion: cilium.io/v2
 kind: CiliumLoadBalancerIPPool
 metadata:
   name: "01-lb-pool"

site.yml

@@ -6,7 +6,7 @@
       ansible.builtin.assert:
         that: ansible_version.full is version_compare('2.11', '>=')
         msg: >
-          "Ansible is out of date. See here for more info: https://docs.technotim.live/posts/ansible-automation/"
+          "Ansible is out of date. See here for more info: https://docs.technotim.com/posts/ansible-automation/"
 
 - name: Prepare Proxmox cluster
   hosts: proxmox