feat(dependencies): upgrade supported cluster components (#681)

* fix(calico): support split CRDs for current releases

- Download the v1_crd_projectcalico_org.yaml bundle before the operator
- Apply both files with server-side apply and force-conflicts per the
  upstream upgrade procedure
- Wait for the operator Deployment and for the managed CRDs to be
  Established after the operator starts
- Replace the create/rescue/replace flow with an idempotent apply that
  no longer conceals partial failures
- Verify TigeraStatus for calico and apiserver is Available, not just
  that Pods exist

* feat(dependencies): upgrade supported cluster components

- Bump K3s to v1.36.2+k3s1, Calico to v3.32.1, Cilium to v1.20.0,
  kube-vip to v1.2.2, kube-vip cloud provider to v0.0.12, and MetalLB to
  v0.16.0 across sample inventory, role defaults, and argument specs
- Pin the Cilium CLI with a new cilium_cli_tag (v0.19.7) instead of the
  floating stable.txt lookup
- Replace the CiliumBGPPeeringPolicy v2alpha1 BGP template with the
  v2 CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement,
  and CiliumLoadBalancerIPPool resource set
- Move Cilium load balancer Helm keys from bpf.loadBalancer to the valid
  top-level loadBalancer path
- Add preflight schema validation and remove the deprecated policy after
  the v2 objects are accepted
- Wait for cilium status after installation
- Pin kube-vip RBAC in a repository template instead of fetching a
  mutable URL, and include EndpointSlice permissions
- Fix the kube-vip bgppeers format to address:ASN comma-separated peers
- Fail clearly when the MetalLB speaker tag replacement does not apply
- Drop the obsolete MetalLB webhook service name version branch

* test(molecule): verify upgraded cluster components

- Assert every node reports the expected K3s kubelet version
- Verify the active CNI (Flannel / Calico / Cilium) is Ready and runs
  the expected image tag, including Calico TigeraStatus Available
- Verify the active load balancer (MetalLB / kube-vip) runs the expected
  image tags and that MetalLB is absent when kube-vip is active
- Assert no Flannel DaemonSet remains when Calico or Cilium is enabled
- Assert the example LoadBalancer address falls inside the configured
  pool range
- Add a manifest-only Cilium BGP regression test that renders the v2
  template with zero, one, and multiple neighbors and rejects any v2alpha1
  or CiliumBGPPeeringPolicy output

* fix(dependencies): correct dependency version pins

- Set the sample kube-vip image to v1.2.2 and repair the damaged comment
- Pin the kube-vip cloud provider default to v0.0.12 in the task URL
- Set the MetalLB controller argument-spec default to v0.16.0
- Restore the MetalLB available timeout default to 240s

* docs(dependencies): document current cluster versions

- Update kube-vip, kube-vip cloud provider, and MetalLB defaults
- Add cilium_tag and cilium_cli_tag rows
- Explain that MetalLB v0.16.0 is the application image target even though
  a newer chart-only tag (metallb-chart-0.16.1) exists
- Add an existing-cluster upgrade warning covering the K3s etcd 3.5.26
  bridge and one-minor-at-a-time rule, consecutive Cilium minor upgrades,
  Calico v3 resource UID handling, and MetalLB app vs chart tags

* fix(dependencies): address PR review findings

- Read the MetalLB speaker tag check from the managed host with slurp
  instead of a controller-side file lookup, and match the full image
  reference
- Restore the tigera-operator namespace on the Calico operator Deployment
  wait while keeping the managed CRD waits cluster-scoped
- Make Molecule verify inputs durable and scenario-specific via a
  per-scenario verify-vars.yml, driven by explicit verify_cni/verify_lb
  values instead of non-persisted converge facts
- Rename the kube-vip multi-peer BGP env var from bgppeers to bgp_peers
  and vip_cidr to vip_subnet so v1.2.2 actually reads them
- Map the legacy Cilium routed mode to tunnel and stop passing the alias
  directly to the chart
- Use return-code based failed_when on apply and preflight commands so
  non-error failures are no longer treated as success
- Clarify the sequential K3s upgrade path and backups in the README
- Add kube-vip and MetalLB regression tests and a Cilium mode mapping unit

* fix(dependencies): resolve re-review findings

- correct the Calico TigeraStatus resource kind\n- document tunnel as the supported Cilium routing mode\n- validate load balancer addresses across range and CIDR pools

* fix(molecule): verify embedded flannel instead of a flannel DaemonSet

- K3s 1.36 runs flannel embedded in the k3s agent rather than as a
  kube-flannel-ds DaemonSet, so the flannel verifier queried a workload
  that no longer exists and failed the verify step
- For the flannel scenarios, assert every node is Ready and that neither
  the Calico nor the Cilium namespace exists
- Drop the now-invalid kube-flannel-ds DaemonSet assertion

* fix(molecule): wait for the LoadBalancer address before asserting reachability

- The nginx LoadBalancer service had no ingress address when the
  reachability assertion ran, so status.loadBalancer.ingress[0].ip was
  undefined and the ipwrap filter failed during verify
- Poll the service until MetalLB or kube-vip assigns an external IP
- Record the assigned address once and reuse it for the reachability probe
  and the pool membership checks

* fix(ci): harden calico apiserver wait and extend molecule job timeout

- Bump calico system resources wait retries 30->60 and delay 7->10 so the
  slow-to-reconcile calico-apiserver deployment has enough time under nested-virt
- Raise the molecule step timeout-minutes from 90 to 150 to accommodate
  contended 5-node scenarios (cilium, kube-vip) that were hitting the 90-min cap

* fix(calico): treat optional API server as best-effort on converge

- The Calico API server (calico-apiserver) is an optional add-on for managing
  Calico policy through the projectcalico.org/v3 Kubernetes API; it is not
  required for Calico CNI data plane operation
- With Calico v3.32.1 on K3s 1.36 the tigera-operator never provisions the
  calico-apiserver namespace, causing the converge wait to fail deterministically
- Keep the strict wait for core Calico components (typha, kube-controllers,
  calico-node, csi-node-driver) and make the API server wait tolerate failure
- Restrict the TigeraStatus Available check to the calico status, matching the
  upstream v3.32.1 K3s quickstart which validates without the API server

roles/k3s_server_post/tasks/calico.yml

@@ -11,6 +11,14 @@
         group: root
         mode: "0755"
 
+    - name: "Download to first master: Calico CRD bundle for {{ calico_tag }}"
+      ansible.builtin.get_url:
+        url: https://raw.githubusercontent.com/projectcalico/calico/{{ calico_tag }}/manifests/v1_crd_projectcalico_org.yaml # noqa yaml[line-length]
+        dest: /tmp/k3s/v1_crd_projectcalico_org.yaml
+        owner: root
+        group: root
+        mode: "0755"
+
     - name: "Download to first master: manifest for Tigera Operator and Calico CRDs"
       ansible.builtin.get_url:
         url: https://raw.githubusercontent.com/projectcalico/calico/{{ calico_tag }}/manifests/tigera-operator.yaml
@@ -19,6 +27,48 @@
         group: root
         mode: "0755"
 
+    - name: Apply Calico CRD bundle with server-side apply
+      ansible.builtin.command: >-
+        {{ k3s_kubectl_binary | default('k3s kubectl') }}
+        apply --server-side --force-conflicts
+        -f /tmp/k3s/v1_crd_projectcalico_org.yaml
+      register: apply_crds
+      changed_when: >-
+        'created' in apply_crds.stdout or
+        'configured' in apply_crds.stdout or
+        'applied' in apply_crds.stdout
+      failed_when: apply_crds.rc != 0
+
+    - name: Apply Tigera Operator manifest idempotently with server-side apply
+      ansible.builtin.command: >-
+        {{ k3s_kubectl_binary | default('k3s kubectl') }}
+        apply --server-side --force-conflicts
+        -f /tmp/k3s/tigera-operator.yaml
+      register: apply_operator
+      changed_when: >-
+        'created' in apply_operator.stdout or
+        'configured' in apply_operator.stdout or
+        'applied' in apply_operator.stdout
+      failed_when: apply_operator.rc != 0
+
+    - name: Wait for Tigera Operator and managed CRDs to become available
+      ansible.builtin.command: >-
+        {{ k3s_kubectl_binary | default('k3s kubectl') }} wait {{ item.type }}/{{ item.name }}
+        {% if item.namespace is defined %}--namespace='{{ item.namespace }}'{% endif %}
+        --for=condition={{ item.condition }}
+        --timeout=30s
+      register: tigera_result
+      changed_when: false
+      until: tigera_result is succeeded
+      retries: 7
+      delay: 7
+      with_items:
+        - { name: tigera-operator, type: deployment, namespace: tigera-operator, condition: Available=True }
+        - { name: installations.operator.tigera.io, type: crd, condition: Established }
+        - { name: apiservers.operator.tigera.io, type: crd, condition: Established }
+      loop_control:
+        label: "{{ item.type }}/{{ item.name }}"
+
     - name: Copy Calico custom resources manifest to first master
       ansible.builtin.template:
         src: calico.crs.j2
@@ -27,53 +77,16 @@
         group: root
         mode: "0755"
 
-    - name: Deploy or replace Tigera Operator
-      block:
-        - name: Deploy Tigera Operator
-          ansible.builtin.command:
-            cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} create -f /tmp/k3s/tigera-operator.yaml"
-          register: create_operator
-          changed_when: "'created' in create_operator.stdout"
-          failed_when: "'Error' in create_operator.stderr and 'already exists' not in create_operator.stderr"
-      rescue:
-        - name: Replace existing Tigera Operator
-          ansible.builtin.command:
-            cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} replace -f /tmp/k3s/tigera-operator.yaml"
-          register: replace_operator
-          changed_when: "'replaced' in replace_operator.stdout"
-          failed_when: "'Error' in replace_operator.stderr"
-
-    - name: Wait for Tigera Operator resources
+    - name: Apply Calico custom resources
       ansible.builtin.command: >-
-        {{ k3s_kubectl_binary | default('k3s kubectl') }} wait {{ item.type }}/{{ item.name }}
-        --namespace='tigera-operator'
-        --for=condition=Available=True
-        --timeout=30s
-      register: tigera_result
-      changed_when: false
-      until: tigera_result is succeeded
-      retries: 7
-      delay: 7
-      with_items:
-        - { name: tigera-operator, type: deployment }
-      loop_control:
-        label: "{{ item.type }}/{{ item.name }}"
-
-    - name: Deploy Calico custom resources
-      block:
-        - name: Deploy custom resources for Calico
-          ansible.builtin.command:
-            cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} create -f /tmp/k3s/custom-resources.yaml"
-          register: create_cr
-          changed_when: "'created' in create_cr.stdout"
-          failed_when: "'Error' in create_cr.stderr and 'already exists' not in create_cr.stderr"
-      rescue:
-        - name: Apply new Calico custom resource manifest
-          ansible.builtin.command:
-            cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} apply -f /tmp/k3s/custom-resources.yaml"
-          register: apply_cr
-          changed_when: "'configured' in apply_cr.stdout or 'created' in apply_cr.stdout"
-          failed_when: "'Error' in apply_cr.stderr"
+        {{ k3s_kubectl_binary | default('k3s kubectl') }}
+        apply -f /tmp/k3s/custom-resources.yaml
+      register: apply_cr
+      changed_when: >-
+        'configured' in apply_cr.stdout or
+        'created' in apply_cr.stdout or
+        'unchanged' in apply_cr.stdout
+      failed_when: apply_cr.rc != 0
 
     - name: Wait for Calico system resources to be available
       ansible.builtin.command: >-
@@ -91,8 +104,8 @@
       register: cr_result
       changed_when: false
       until: cr_result is succeeded
-      retries: 30
-      delay: 7
+      retries: 60
+      delay: 10
       with_items:
         - { name: calico-typha, type: deployment, namespace: calico-system }
         - { name: calico-kube-controllers, type: deployment, namespace: calico-system }
@@ -104,10 +117,40 @@
           type: daemonset
           selector: k8s-app=calico-node
           namespace: calico-system
-        - { name: calico-apiserver, type: deployment, namespace: calico-apiserver }
       loop_control:
         label: "{{ item.type }}/{{ item.name }}"
 
+    # The Calico API server is an optional add-on for managing Calico policy via
+    # kubectl (the projectcalico.org/v3 Kubernetes API). It is not required for
+    # Calico CNI data plane operation and does not reconcile on K3s with recent
+    # Calico releases, so it is treated as best-effort here.
+    - name: Wait for the optional Calico API server (best effort)
+      ansible.builtin.command: >-
+        {{ k3s_kubectl_binary | default('k3s kubectl') }} wait deployment/calico-apiserver
+        --namespace=calico-apiserver --for=condition=Available --timeout=30s
+      register: cr_apiserver
+      changed_when: false
+      until: cr_apiserver is succeeded
+      retries: 30
+      delay: 10
+      ignore_errors: true
+
+    - name: Verify Calico TigeraStatus reports Available
+      ansible.builtin.command: >-
+        {{ k3s_kubectl_binary | default('k3s kubectl') }}
+        get tigerastatus {{ item }}
+        -o jsonpath='{.status.conditions[?(@.type=="Available")].status}'
+      register: tigera_status
+      changed_when: false
+      until: tigera_status.stdout | trim == 'True'
+      retries: 30
+      delay: 7
+      failed_when: tigera_status.rc != 0 or (tigera_status.stdout | trim) != 'True'
+      with_items:
+        - calico
+      loop_control:
+        label: "tigerastatus/{{ item }}"
+
     - name: Patch Felix configuration for eBPF mode
       ansible.builtin.command:
         cmd: >