feat(dependencies): upgrade supported cluster components (#681)

* fix(calico): support split CRDs for current releases

- Download the v1_crd_projectcalico_org.yaml bundle before the operator
- Apply both files with server-side apply and force-conflicts per the
  upstream upgrade procedure
- Wait for the operator Deployment and for the managed CRDs to be
  Established after the operator starts
- Replace the create/rescue/replace flow with an idempotent apply that
  no longer conceals partial failures
- Verify TigeraStatus for calico and apiserver is Available, not just
  that Pods exist

* feat(dependencies): upgrade supported cluster components

- Bump K3s to v1.36.2+k3s1, Calico to v3.32.1, Cilium to v1.20.0,
  kube-vip to v1.2.2, kube-vip cloud provider to v0.0.12, and MetalLB to
  v0.16.0 across sample inventory, role defaults, and argument specs
- Pin the Cilium CLI with a new cilium_cli_tag (v0.19.7) instead of the
  floating stable.txt lookup
- Replace the CiliumBGPPeeringPolicy v2alpha1 BGP template with the
  v2 CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement,
  and CiliumLoadBalancerIPPool resource set
- Move Cilium load balancer Helm keys from bpf.loadBalancer to the valid
  top-level loadBalancer path
- Add preflight schema validation and remove the deprecated policy after
  the v2 objects are accepted
- Wait for cilium status after installation
- Pin kube-vip RBAC in a repository template instead of fetching a
  mutable URL, and include EndpointSlice permissions
- Fix the kube-vip bgppeers format to address:ASN comma-separated peers
- Fail clearly when the MetalLB speaker tag replacement does not apply
- Drop the obsolete MetalLB webhook service name version branch

* test(molecule): verify upgraded cluster components

- Assert every node reports the expected K3s kubelet version
- Verify the active CNI (Flannel / Calico / Cilium) is Ready and runs
  the expected image tag, including Calico TigeraStatus Available
- Verify the active load balancer (MetalLB / kube-vip) runs the expected
  image tags and that MetalLB is absent when kube-vip is active
- Assert no Flannel DaemonSet remains when Calico or Cilium is enabled
- Assert the example LoadBalancer address falls inside the configured
  pool range
- Add a manifest-only Cilium BGP regression test that renders the v2
  template with zero, one, and multiple neighbors and rejects any v2alpha1
  or CiliumBGPPeeringPolicy output

* fix(dependencies): correct dependency version pins

- Set the sample kube-vip image to v1.2.2 and repair the damaged comment
- Pin the kube-vip cloud provider default to v0.0.12 in the task URL
- Set the MetalLB controller argument-spec default to v0.16.0
- Restore the MetalLB available timeout default to 240s

* docs(dependencies): document current cluster versions

- Update kube-vip, kube-vip cloud provider, and MetalLB defaults
- Add cilium_tag and cilium_cli_tag rows
- Explain that MetalLB v0.16.0 is the application image target even though
  a newer chart-only tag (metallb-chart-0.16.1) exists
- Add an existing-cluster upgrade warning covering the K3s etcd 3.5.26
  bridge and one-minor-at-a-time rule, consecutive Cilium minor upgrades,
  Calico v3 resource UID handling, and MetalLB app vs chart tags

* fix(dependencies): address PR review findings

- Read the MetalLB speaker tag check from the managed host with slurp
  instead of a controller-side file lookup, and match the full image
  reference
- Restore the tigera-operator namespace on the Calico operator Deployment
  wait while keeping the managed CRD waits cluster-scoped
- Make Molecule verify inputs durable and scenario-specific via a
  per-scenario verify-vars.yml, driven by explicit verify_cni/verify_lb
  values instead of non-persisted converge facts
- Rename the kube-vip multi-peer BGP env var from bgppeers to bgp_peers
  and vip_cidr to vip_subnet so v1.2.2 actually reads them
- Map the legacy Cilium routed mode to tunnel and stop passing the alias
  directly to the chart
- Use return-code based failed_when on apply and preflight commands so
  non-error failures are no longer treated as success
- Clarify the sequential K3s upgrade path and backups in the README
- Add kube-vip and MetalLB regression tests and a Cilium mode mapping unit

* fix(dependencies): resolve re-review findings

- correct the Calico TigeraStatus resource kind\n- document tunnel as the supported Cilium routing mode\n- validate load balancer addresses across range and CIDR pools

* fix(molecule): verify embedded flannel instead of a flannel DaemonSet

- K3s 1.36 runs flannel embedded in the k3s agent rather than as a
  kube-flannel-ds DaemonSet, so the flannel verifier queried a workload
  that no longer exists and failed the verify step
- For the flannel scenarios, assert every node is Ready and that neither
  the Calico nor the Cilium namespace exists
- Drop the now-invalid kube-flannel-ds DaemonSet assertion

* fix(molecule): wait for the LoadBalancer address before asserting reachability

- The nginx LoadBalancer service had no ingress address when the
  reachability assertion ran, so status.loadBalancer.ingress[0].ip was
  undefined and the ipwrap filter failed during verify
- Poll the service until MetalLB or kube-vip assigns an external IP
- Record the assigned address once and reuse it for the reachability probe
  and the pool membership checks

* fix(ci): harden calico apiserver wait and extend molecule job timeout

- Bump calico system resources wait retries 30->60 and delay 7->10 so the
  slow-to-reconcile calico-apiserver deployment has enough time under nested-virt
- Raise the molecule step timeout-minutes from 90 to 150 to accommodate
  contended 5-node scenarios (cilium, kube-vip) that were hitting the 90-min cap

* fix(calico): treat optional API server as best-effort on converge

- The Calico API server (calico-apiserver) is an optional add-on for managing
  Calico policy through the projectcalico.org/v3 Kubernetes API; it is not
  required for Calico CNI data plane operation
- With Calico v3.32.1 on K3s 1.36 the tigera-operator never provisions the
  calico-apiserver namespace, causing the converge wait to fail deterministically
- Keep the strict wait for core Calico components (typha, kube-controllers,
  calico-node, csi-node-driver) and make the API server wait tolerate failure
- Restrict the TigeraStatus Available check to the calico status, matching the
  upstream v3.32.1 K3s quickstart which validates without the API server

.github/scripts/test-cilium-bgp-manifest.py

@@ -0,0 +1,134 @@
+#!/usr/bin/env python3
+"""Render the Cilium BGP CRD template and assert it uses the v2 API.
+
+This is a manifest-only regression test used where no real BGP peer is
+available. It renders roles/k3s_server_post/templates/cilium.crs.j2 with
+zero, one, and multiple neighbors, then checks that the output:
+  - never contains CiliumBGPPeeringPolicy or cilium.io/v2alpha1
+  - emits the Cilium v2 BGP resources
+  - emits deterministic DNS-safe peer and instance names
+  - advertises Pod CIDRs only when cilium_exportPodCIDR is true
+"""
+
+from __future__ import print_function
+
+import os
+import subprocess
+
+from jinja2 import Environment, FileSystemLoader, StrictUndefined
+
+
+def repo_root():
+    return subprocess.check_output(
+        ["git", "rev-parse", "--show-toplevel"], text=True
+    ).strip()
+
+
+def fail(message):
+    raise SystemExit("Cilium BGP manifest test failed: " + message)
+
+
+def render(env, extra_vars):
+    base_vars = {
+        "cilium_bgp_my_asn": "64513",
+        "cilium_bgp_peer_asn": "64512",
+        "cilium_bgp_peer_address": "192.168.30.1",
+        "cilium_exportPodCIDR": True,
+        "cilium_bgp_lb_cidr": "192.168.31.0/24",
+    }
+    base_vars.update(extra_vars)
+    template = env.get_template("cilium.crs.j2")
+    return template.render(**base_vars)
+
+
+def check_common(output):
+    if "cilium.io/v2alpha1" in output:
+        fail("rendered output still contains cilium.io/v2alpha1")
+    if "kind: CiliumBGPPeeringPolicy" in output:
+        fail("rendered output still contains CiliumBGPPeeringPolicy")
+    for kind in (
+        "CiliumBGPPeerConfig",
+        "CiliumBGPClusterConfig",
+        "CiliumBGPAdvertisement",
+        "CiliumLoadBalancerIPPool",
+    ):
+        if ("kind: " + kind) not in output:
+            fail("rendered output is missing kind: " + kind)
+
+
+def main():
+    root = repo_root()
+    template_dir = os.path.join(
+        root, "roles", "k3s_server_post", "templates"
+    )
+    env = Environment(
+        loader=FileSystemLoader(template_dir), undefined=StrictUndefined
+    )
+
+    # Zero neighbors -> fall back to the single default peer.
+    output = render(env, {"_cilium_bgp_neighbors": []})
+    check_common(output)
+    if "peer-64512-1" not in output:
+        fail("default single peer name was not rendered")
+    if "peerAddress: 192.168.30.1" not in output:
+        fail("default peer address was not rendered")
+    if 'advertisementType: "PodCIDR"' not in output:
+        fail("PodCIDR advertisement missing when exportPodCIDR is true")
+
+    # One neighbor via the merged list.
+    output = render(
+        env,
+        {"_cilium_bgp_neighbors": [{"peer_address": "10.0.0.1", "peer_asn": "65001"}]},
+    )
+    check_common(output)
+    if "peer-65001-1" not in output:
+        fail("single merged peer name was not rendered")
+    if "peerAddress: 10.0.0.1" not in output:
+        fail("single merged peer address was not rendered")
+
+    # Multiple neighbors.
+    output = render(
+        env,
+        {
+            "_cilium_bgp_neighbors": [
+                {"peer_address": "10.0.0.1", "peer_asn": "65001"},
+                {"peer_address": "10.0.0.2", "peer_asn": "65002"},
+            ]
+        },
+    )
+    check_common(output)
+    if "peer-65001-1" not in output or "peer-65002-2" not in output:
+        fail("multiple merged peer names were not rendered")
+    if "peerAddress: 10.0.0.2" not in output:
+        fail("second merged peer address was not rendered")
+
+    # exportPodCIDR false -> no PodCIDR advertisement, service remains.
+    output = render(
+        env, {"_cilium_bgp_neighbors": [], "cilium_exportPodCIDR": False}
+    )
+    check_common(output)
+    if 'advertisementType: "PodCIDR"' in output:
+        fail("PodCIDR advertisement present when exportPodCIDR is false")
+    if 'advertisementType: "Service"' not in output:
+        fail("Service advertisement missing when exportPodCIDR is false")
+
+    # Load balancer pools: CIDR and start/stop forms.
+    output = render(env, {"_cilium_bgp_neighbors": []})
+    if "cidr: 192.168.31.0/24" not in output:
+        fail("CIDR load balancer pool was not rendered")
+    output = render(
+        env,
+        {
+            "_cilium_bgp_neighbors": [],
+            "cilium_bgp_lb_cidr": "192.168.31.80-192.168.31.90",
+        },
+    )
+    check_common(output)
+    if "start: 192.168.31.80" not in output or "stop: 192.168.31.90" not in output:
+        fail("start/stop load balancer pool was not rendered")
+
+    print("Cilium BGP manifest regression test passed")
+
+
+if __name__ == "__main__":
+    main()

.github/scripts/test-kube-vip-manifest.py

@@ -0,0 +1,130 @@
+#!/usr/bin/env python3
+"""Render the kube-vip DaemonSet template and assert env key correctness.
+
+kube-vip v1.2.2 reads `bgp_peers` and `vip_subnet`; it ignores the older
+`bgppeers` and `vip_cidr` names. This test proves the rendered manifest uses
+the keys the target image actually parses.
+"""
+
+from __future__ import print_function
+
+import os
+import subprocess
+
+from jinja2 import Environment, FileSystemLoader, StrictUndefined
+
+
+def repo_root():
+    return subprocess.check_output(
+        ["git", "rev-parse", "--show-toplevel"], text=True
+    ).strip()
+
+
+def fail(message):
+    raise SystemExit("kube-vip manifest test failed: " + message)
+
+
+def fake_ipsubnet(value):
+    # ansible.utils.ipsubnet -> network of the address as x.y.z.0/24
+    parts = value.split(".")
+    return ".".join(parts[:3]) + ".0/24"
+
+
+def fake_ipaddr(_value, expr=None):
+    # ansible.utils.ipaddr('prefix') -> prefix length
+    return "24"
+
+
+def fake_bool(value):
+    # Minimal stand-in for Ansible's truthiness filter used by the template.
+    if isinstance(value, bool):
+        return value
+    return str(value).lower() in ("1", "true", "yes", "on")
+
+
+def fake_map(seq, *args, **kwargs):
+    # Minimal stand-in for Ansible's map() filter in the two forms used by the
+    # template: map(attribute='x') on a list of dicts, and map('join', sep) on
+    # a list of sequences.
+    if "attribute" in kwargs:
+        return [item[kwargs["attribute"]] for item in seq]
+    if kwargs:
+        # e.g. map(default='x') not used here; ignore unknown kwargs.
+        return list(seq)
+    if args:
+        filter_name = args[0]
+        sep = args[1] if len(args) > 1 else ""
+        if filter_name == "join":
+            return [sep.join(str(x) for x in item) for item in seq]
+    return list(seq)
+
+
+def fake_zip(*seqs):
+    return list(zip(*seqs))
+
+
+def render(env, extra_vars):
+    base_vars = {
+        "apiserver_endpoint": "192.168.30.222",
+        "kube_vip_iface": "",
+        "kube_vip_arp": True,
+        "kube_vip_bgp": True,
+        "kube_vip_bgp_routerid": "127.0.0.1",
+        "_kube_vip_bgp_peers": [
+            {"peer_address": "192.168.30.1", "peer_asn": "64512"},
+            {"peer_address": "192.168.30.2", "peer_asn": "64513"},
+        ],
+        "kube_vip_tag_version": "v1.2.2",
+    }
+    base_vars.update(extra_vars)
+    template = env.get_template("vip.yaml.j2")
+    return template.render(**base_vars)
+
+
+def main():
+    root = repo_root()
+    template_dir = os.path.join(root, "roles", "k3s_server", "templates")
+    env = Environment(
+        loader=FileSystemLoader(template_dir), undefined=StrictUndefined
+    )
+    env.filters["ansible.utils.ipsubnet"] = fake_ipsubnet
+    env.filters["ansible.utils.ipaddr"] = fake_ipaddr
+    env.filters["bool"] = fake_bool
+    env.filters["map"] = fake_map
+    env.filters["zip"] = fake_zip
+
+    # Multi-peer BGP armed: must emit bgp_peers, never bgppeers.
+    output = render(env, {})
+    if "name: bgp_peers" not in output:
+        fail("rendered manifest is missing bgp_peers")
+    if "name: bgppeers" in output:
+        fail("rendered manifest still uses the ignored bgppeers key")
+    if "name: vip_subnet" not in output:
+        fail("rendered manifest is missing vip_subnet")
+    if "name: vip_cidr" in output:
+        fail("rendered manifest still uses the ignored vip_cidr key")
+    if "192.168.30.1:64512,192.168.30.2:64513" not in output:
+        fail("bgp_peers value is not comma-separated address:ASN entries")
+    if "ghcr.io/kube-vip/kube-vip:v1.2.2" not in output:
+        fail("kube-vip image tag is not v1.2.2")
+
+    # BGP enabled with no merged peers: single-peer fallback vars, no bgp_peers.
+    output = render(
+        env,
+        {
+            "_kube_vip_bgp_peers": [],
+            "kube_vip_bgp_as": "64513",
+            "kube_vip_bgp_peeraddress": "192.168.30.1",
+            "kube_vip_bgp_peeras": "64512",
+        },
+    )
+    if "name: bgp_as" not in output:
+        fail("single-peer bgp_as was not rendered")
+    if "name: bgp_peers" in output:
+        fail("bgp_peers present even though the peer list is empty")
+
+    print("kube-vip manifest regression test passed")
+
+
+if __name__ == "__main__":
+    main()

.github/scripts/test-metallb-remote-read.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+repo_root="$(git rev-parse --show-toplevel)"
+metallb_task="$repo_root/roles/k3s_server/tasks/metallb.yml"
+
+# The speaker tag verification must read the rendered manifest on the managed
+# host with slurp. A controller-side lookup('ansible.builtin.file', ...) would
+# read from the Ansible control node, which does not have the file, and would
+# fail on every MetalLB scenario.
+grep -Fq -- 'ansible.builtin.slurp' "$metallb_task" || {
+  printf 'MetalLB speaker tag check does not use slurp on the managed host\n' >&2
+  exit 1
+}
+
+grep -Eq -- 'lookup\(.?ansible\.builtin\.file' "$metallb_task" && {
+  printf 'MetalLB speaker tag check uses a controller-side file lookup\n' >&2
+  exit 1
+}
+
+# The check must reference the full image reference, not just a bare version
+# string that could appear anywhere in the manifest.
+grep -Fq -- 'quay.io/metallb/speaker:' "$metallb_task" || {
+  printf 'MetalLB speaker tag check does not match the full image reference\n' >&2
+  exit 1
+}
+
+printf 'MetalLB remote manifest read regression test passed\n'

.github/workflows/test.yml

@@ -88,7 +88,7 @@ jobs:
           trap stop_monitor EXIT
           /usr/bin/time -v -o "$timing_file" \
             molecule test --scenario-name ${{ matrix.scenario }}
-        timeout-minutes: 90
+        timeout-minutes: 150
         env:
           ANSIBLE_K3S_LOG_DIR: ${{ runner.temp }}/logs/k3s-ansible/${{ matrix.scenario }}
           ANSIBLE_SSH_RETRIES: 4