feat(dependencies): upgrade supported cluster components (#681)

* fix(calico): support split CRDs for current releases

- Download the v1_crd_projectcalico_org.yaml bundle before the operator
- Apply both files with server-side apply and force-conflicts per the
  upstream upgrade procedure
- Wait for the operator Deployment and for the managed CRDs to be
  Established after the operator starts
- Replace the create/rescue/replace flow with an idempotent apply that
  no longer conceals partial failures
- Verify TigeraStatus for calico and apiserver is Available, not just
  that Pods exist

* feat(dependencies): upgrade supported cluster components

- Bump K3s to v1.36.2+k3s1, Calico to v3.32.1, Cilium to v1.20.0,
  kube-vip to v1.2.2, kube-vip cloud provider to v0.0.12, and MetalLB to
  v0.16.0 across sample inventory, role defaults, and argument specs
- Pin the Cilium CLI with a new cilium_cli_tag (v0.19.7) instead of the
  floating stable.txt lookup
- Replace the CiliumBGPPeeringPolicy v2alpha1 BGP template with the
  v2 CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement,
  and CiliumLoadBalancerIPPool resource set
- Move Cilium load balancer Helm keys from bpf.loadBalancer to the valid
  top-level loadBalancer path
- Add preflight schema validation and remove the deprecated policy after
  the v2 objects are accepted
- Wait for cilium status after installation
- Pin kube-vip RBAC in a repository template instead of fetching a
  mutable URL, and include EndpointSlice permissions
- Fix the kube-vip bgppeers format to address:ASN comma-separated peers
- Fail clearly when the MetalLB speaker tag replacement does not apply
- Drop the obsolete MetalLB webhook service name version branch

* test(molecule): verify upgraded cluster components

- Assert every node reports the expected K3s kubelet version
- Verify the active CNI (Flannel / Calico / Cilium) is Ready and runs
  the expected image tag, including Calico TigeraStatus Available
- Verify the active load balancer (MetalLB / kube-vip) runs the expected
  image tags and that MetalLB is absent when kube-vip is active
- Assert no Flannel DaemonSet remains when Calico or Cilium is enabled
- Assert the example LoadBalancer address falls inside the configured
  pool range
- Add a manifest-only Cilium BGP regression test that renders the v2
  template with zero, one, and multiple neighbors and rejects any v2alpha1
  or CiliumBGPPeeringPolicy output

* fix(dependencies): correct dependency version pins

- Set the sample kube-vip image to v1.2.2 and repair the damaged comment
- Pin the kube-vip cloud provider default to v0.0.12 in the task URL
- Set the MetalLB controller argument-spec default to v0.16.0
- Restore the MetalLB available timeout default to 240s

* docs(dependencies): document current cluster versions

- Update kube-vip, kube-vip cloud provider, and MetalLB defaults
- Add cilium_tag and cilium_cli_tag rows
- Explain that MetalLB v0.16.0 is the application image target even though
  a newer chart-only tag (metallb-chart-0.16.1) exists
- Add an existing-cluster upgrade warning covering the K3s etcd 3.5.26
  bridge and one-minor-at-a-time rule, consecutive Cilium minor upgrades,
  Calico v3 resource UID handling, and MetalLB app vs chart tags

* fix(dependencies): address PR review findings

- Read the MetalLB speaker tag check from the managed host with slurp
  instead of a controller-side file lookup, and match the full image
  reference
- Restore the tigera-operator namespace on the Calico operator Deployment
  wait while keeping the managed CRD waits cluster-scoped
- Make Molecule verify inputs durable and scenario-specific via a
  per-scenario verify-vars.yml, driven by explicit verify_cni/verify_lb
  values instead of non-persisted converge facts
- Rename the kube-vip multi-peer BGP env var from bgppeers to bgp_peers
  and vip_cidr to vip_subnet so v1.2.2 actually reads them
- Map the legacy Cilium routed mode to tunnel and stop passing the alias
  directly to the chart
- Use return-code based failed_when on apply and preflight commands so
  non-error failures are no longer treated as success
- Clarify the sequential K3s upgrade path and backups in the README
- Add kube-vip and MetalLB regression tests and a Cilium mode mapping unit

* fix(dependencies): resolve re-review findings

- correct the Calico TigeraStatus resource kind\n- document tunnel as the supported Cilium routing mode\n- validate load balancer addresses across range and CIDR pools

* fix(molecule): verify embedded flannel instead of a flannel DaemonSet

- K3s 1.36 runs flannel embedded in the k3s agent rather than as a
  kube-flannel-ds DaemonSet, so the flannel verifier queried a workload
  that no longer exists and failed the verify step
- For the flannel scenarios, assert every node is Ready and that neither
  the Calico nor the Cilium namespace exists
- Drop the now-invalid kube-flannel-ds DaemonSet assertion

* fix(molecule): wait for the LoadBalancer address before asserting reachability

- The nginx LoadBalancer service had no ingress address when the
  reachability assertion ran, so status.loadBalancer.ingress[0].ip was
  undefined and the ipwrap filter failed during verify
- Poll the service until MetalLB or kube-vip assigns an external IP
- Record the assigned address once and reuse it for the reachability probe
  and the pool membership checks

* fix(ci): harden calico apiserver wait and extend molecule job timeout

- Bump calico system resources wait retries 30->60 and delay 7->10 so the
  slow-to-reconcile calico-apiserver deployment has enough time under nested-virt
- Raise the molecule step timeout-minutes from 90 to 150 to accommodate
  contended 5-node scenarios (cilium, kube-vip) that were hitting the 90-min cap

* fix(calico): treat optional API server as best-effort on converge

- The Calico API server (calico-apiserver) is an optional add-on for managing
  Calico policy through the projectcalico.org/v3 Kubernetes API; it is not
  required for Calico CNI data plane operation
- With Calico v3.32.1 on K3s 1.36 the tigera-operator never provisions the
  calico-apiserver namespace, causing the converge wait to fail deterministically
- Keep the strict wait for core Calico components (typha, kube-controllers,
  calico-node, csi-node-driver) and make the API server wait tolerate failure
- Restrict the TigeraStatus Available check to the calico status, matching the
  upstream v3.32.1 K3s quickstart which validates without the API server

roles/k3s_server_post/tasks/cilium.yml

@@ -30,26 +30,13 @@
           }}
       when: cilium_cli_installed.rc == 0
 
-    - name: Get latest stable Cilium CLI version file
-      ansible.builtin.get_url:
-        url: https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt
-        dest: /tmp/k3s/cilium-cli-stable.txt
-        owner: root
-        group: root
-        mode: "0755"
-
-    - name: Read Cilium CLI stable version from file
-      ansible.builtin.command: cat /tmp/k3s/cilium-cli-stable.txt
-      register: cli_ver
-      changed_when: false
-
     - name: Log installed Cilium CLI version
       ansible.builtin.debug:
         msg: "Installed Cilium CLI version: {{ installed_cli_version | default('Not installed') }}"
 
-    - name: Log latest stable Cilium CLI version
+    - name: Log pinned Cilium CLI version
       ansible.builtin.debug:
-        msg: "Latest Cilium CLI version: {{ cli_ver.stdout }}"
+        msg: "Pinned Cilium CLI version: {{ cilium_cli_tag }}"
 
     - name: Determine if Cilium CLI needs installation or update
       ansible.builtin.set_fact:
@@ -57,7 +44,7 @@
           {{
             cilium_cli_installed.rc != 0 or
             (cilium_cli_installed.rc == 0 and
-            installed_cli_version != cli_ver.stdout)
+            installed_cli_version != cilium_cli_tag)
           }}
 
     - name: Install or update Cilium CLI
@@ -78,7 +65,7 @@
             - .tar.gz
             - .tar.gz.sha256sum
           vars:
-            cilium_base_url: https://github.com/cilium/cilium-cli/releases/download/{{ cli_ver.stdout }}
+            cilium_base_url: https://github.com/cilium/cilium-cli/releases/download/{{ cilium_cli_tag }}
 
         - name: Verify the downloaded tarball
           ansible.builtin.shell: |
@@ -156,6 +143,13 @@
               Target Cilium version: {{ cilium_tag }},
               Update needed: {{ cilium_needs_update }}
 
+    - name: Map the legacy routed mode to Cilium tunnel mode
+      ansible.builtin.set_fact:
+        # Cilium 1.20 accepts `native` or `tunnel`. `routed` was the name used
+        # by older releases and is kept only as a backward-compatible alias.
+        cilium_routing_mode: >-
+          {{ 'tunnel' if cilium_mode == 'routed' else cilium_mode }}
+
     - name: Install Cilium
       ansible.builtin.command: >-
         {% if cilium_installed.rc != 0 %}
@@ -167,13 +161,13 @@
         --helm-set operator.replicas="1"
         {{ '--helm-set devices=' + cilium_iface if cilium_iface != 'auto' else '' }}
         --helm-set ipam.operator.clusterPoolIPv4PodCIDRList={{ cluster_cidr }}
-        {% if cilium_mode == "native" or (cilium_bgp and cilium_exportPodCIDR != 'false') %}
+        {% if cilium_routing_mode == "native" or (cilium_bgp and cilium_exportPodCIDR != 'false') %}
         --helm-set ipv4NativeRoutingCIDR={{ cluster_cidr }}
         {% endif %}
         --helm-set k8sServiceHost="127.0.0.1"
         --helm-set k8sServicePort="6444"
-        --helm-set routingMode={{ cilium_mode }}
-        --helm-set autoDirectNodeRoutes={{ "true" if cilium_mode == "native" else "false" }}
+        --helm-set routingMode={{ cilium_routing_mode }}
+        --helm-set autoDirectNodeRoutes={{ "true" if cilium_routing_mode == "native" else "false" }}
         --helm-set kubeProxyReplacement={{ kube_proxy_replacement }}
         --helm-set bpf.masquerade={{ enable_bpf_masquerade }}
         --helm-set bgpControlPlane.enabled={{ cilium_bgp | default("false") }}
@@ -181,8 +175,8 @@
         --helm-set hubble.relay.enabled={{ "true" if cilium_hubble else "false" }}
         --helm-set hubble.ui.enabled={{ "true" if cilium_hubble else "false" }}
         {% if kube_proxy_replacement is not false %}
-        --helm-set bpf.loadBalancer.algorithm={{ bpf_lb_algorithm }}
-        --helm-set bpf.loadBalancer.mode={{ bpf_lb_mode }}
+        --helm-set loadBalancer.algorithm={{ bpf_lb_algorithm }}
+        --helm-set loadBalancer.mode={{ bpf_lb_mode }}
         {% endif %}
       environment:
         KUBECONFIG: "{{ ansible_user_dir }}/.kube/config"
@@ -218,6 +212,16 @@
       when: >-
         not item.check_hubble | default(false) or (item.check_hubble | default(false) and cilium_hubble)
 
+    - name: Wait for Cilium status to be healthy
+      ansible.builtin.command: cilium status --wait
+      environment:
+        KUBECONFIG: "{{ ansible_user_dir }}/.kube/config"
+      register: cilium_status
+      changed_when: false
+      until: cilium_status.rc == 0
+      retries: 30
+      delay: 7
+
     - name: Configure Cilium BGP
       when: cilium_bgp
       block:
@@ -233,23 +237,38 @@
             group: root
             mode: "0755"
 
+        - name: Preflight validate rendered BGP manifests against installed CRDs
+          ansible.builtin.command: >-
+            {{ k3s_kubectl_binary | default('k3s kubectl') }}
+            apply --dry-run=server -f /tmp/k3s/cilium-bgp.yaml
+          register: preflight_cr
+          changed_when: false
+          failed_when: preflight_cr.rc != 0
+
         - name: Apply BGP manifests
-          ansible.builtin.command:
-            cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} apply -f /tmp/k3s/cilium-bgp.yaml"
+          ansible.builtin.command: >-
+            {{ k3s_kubectl_binary | default('k3s kubectl') }}
+            apply -f /tmp/k3s/cilium-bgp.yaml
           register: apply_cr
           changed_when: "'configured' in apply_cr.stdout or 'created' in apply_cr.stdout"
-          failed_when: "'is invalid' in apply_cr.stderr"
-          ignore_errors: true
+          failed_when: apply_cr.rc != 0
 
-        - name: Print error message if BGP manifests application fails
-          ansible.builtin.debug:
-            msg: "{{ apply_cr.stderr }}"
-          when: "'is invalid' in apply_cr.stderr"
+        - name: Remove deprecated CiliumBGPPeeringPolicy after v2 resources are accepted
+          ansible.builtin.command: >-
+            {{ k3s_kubectl_binary | default('k3s kubectl') }}
+            delete CiliumBGPPeeringPolicy.cilium.io 01-bgp-peering-policy
+          register: delete_old_policy
+          changed_when: "'deleted' in delete_old_policy.stdout"
+          # The policy (and possibly its CRD) may already be absent; this is
+          # intentionally tolerated, not a command whose failure must be hidden.
+          failed_when: false
 
         - name: Test for BGP config resources
           ansible.builtin.command: "{{ item }}"
           loop:
-            - "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumBGPPeeringPolicy.cilium.io"
+            - "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumBGPClusterConfig.cilium.io"
+            - "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumBGPPeerConfig.cilium.io"
+            - "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumBGPAdvertisement.cilium.io"
             - "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumLoadBalancerIPPool.cilium.io"
           changed_when: false
           loop_control: