feat(dependencies): upgrade supported cluster components (#681)

* fix(calico): support split CRDs for current releases

- Download the v1_crd_projectcalico_org.yaml bundle before the operator
- Apply both files with server-side apply and force-conflicts per the
  upstream upgrade procedure
- Wait for the operator Deployment and for the managed CRDs to be
  Established after the operator starts
- Replace the create/rescue/replace flow with an idempotent apply that
  no longer conceals partial failures
- Verify TigeraStatus for calico and apiserver is Available, not just
  that Pods exist

* feat(dependencies): upgrade supported cluster components

- Bump K3s to v1.36.2+k3s1, Calico to v3.32.1, Cilium to v1.20.0,
  kube-vip to v1.2.2, kube-vip cloud provider to v0.0.12, and MetalLB to
  v0.16.0 across sample inventory, role defaults, and argument specs
- Pin the Cilium CLI with a new cilium_cli_tag (v0.19.7) instead of the
  floating stable.txt lookup
- Replace the CiliumBGPPeeringPolicy v2alpha1 BGP template with the
  v2 CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement,
  and CiliumLoadBalancerIPPool resource set
- Move Cilium load balancer Helm keys from bpf.loadBalancer to the valid
  top-level loadBalancer path
- Add preflight schema validation and remove the deprecated policy after
  the v2 objects are accepted
- Wait for cilium status after installation
- Pin kube-vip RBAC in a repository template instead of fetching a
  mutable URL, and include EndpointSlice permissions
- Fix the kube-vip bgppeers format to address:ASN comma-separated peers
- Fail clearly when the MetalLB speaker tag replacement does not apply
- Drop the obsolete MetalLB webhook service name version branch

* test(molecule): verify upgraded cluster components

- Assert every node reports the expected K3s kubelet version
- Verify the active CNI (Flannel / Calico / Cilium) is Ready and runs
  the expected image tag, including Calico TigeraStatus Available
- Verify the active load balancer (MetalLB / kube-vip) runs the expected
  image tags and that MetalLB is absent when kube-vip is active
- Assert no Flannel DaemonSet remains when Calico or Cilium is enabled
- Assert the example LoadBalancer address falls inside the configured
  pool range
- Add a manifest-only Cilium BGP regression test that renders the v2
  template with zero, one, and multiple neighbors and rejects any v2alpha1
  or CiliumBGPPeeringPolicy output

* fix(dependencies): correct dependency version pins

- Set the sample kube-vip image to v1.2.2 and repair the damaged comment
- Pin the kube-vip cloud provider default to v0.0.12 in the task URL
- Set the MetalLB controller argument-spec default to v0.16.0
- Restore the MetalLB available timeout default to 240s

* docs(dependencies): document current cluster versions

- Update kube-vip, kube-vip cloud provider, and MetalLB defaults
- Add cilium_tag and cilium_cli_tag rows
- Explain that MetalLB v0.16.0 is the application image target even though
  a newer chart-only tag (metallb-chart-0.16.1) exists
- Add an existing-cluster upgrade warning covering the K3s etcd 3.5.26
  bridge and one-minor-at-a-time rule, consecutive Cilium minor upgrades,
  Calico v3 resource UID handling, and MetalLB app vs chart tags

* fix(dependencies): address PR review findings

- Read the MetalLB speaker tag check from the managed host with slurp
  instead of a controller-side file lookup, and match the full image
  reference
- Restore the tigera-operator namespace on the Calico operator Deployment
  wait while keeping the managed CRD waits cluster-scoped
- Make Molecule verify inputs durable and scenario-specific via a
  per-scenario verify-vars.yml, driven by explicit verify_cni/verify_lb
  values instead of non-persisted converge facts
- Rename the kube-vip multi-peer BGP env var from bgppeers to bgp_peers
  and vip_cidr to vip_subnet so v1.2.2 actually reads them
- Map the legacy Cilium routed mode to tunnel and stop passing the alias
  directly to the chart
- Use return-code based failed_when on apply and preflight commands so
  non-error failures are no longer treated as success
- Clarify the sequential K3s upgrade path and backups in the README
- Add kube-vip and MetalLB regression tests and a Cilium mode mapping unit

* fix(dependencies): resolve re-review findings

- correct the Calico TigeraStatus resource kind\n- document tunnel as the supported Cilium routing mode\n- validate load balancer addresses across range and CIDR pools

* fix(molecule): verify embedded flannel instead of a flannel DaemonSet

- K3s 1.36 runs flannel embedded in the k3s agent rather than as a
  kube-flannel-ds DaemonSet, so the flannel verifier queried a workload
  that no longer exists and failed the verify step
- For the flannel scenarios, assert every node is Ready and that neither
  the Calico nor the Cilium namespace exists
- Drop the now-invalid kube-flannel-ds DaemonSet assertion

* fix(molecule): wait for the LoadBalancer address before asserting reachability

- The nginx LoadBalancer service had no ingress address when the
  reachability assertion ran, so status.loadBalancer.ingress[0].ip was
  undefined and the ipwrap filter failed during verify
- Poll the service until MetalLB or kube-vip assigns an external IP
- Record the assigned address once and reuse it for the reachability probe
  and the pool membership checks

* fix(ci): harden calico apiserver wait and extend molecule job timeout

- Bump calico system resources wait retries 30->60 and delay 7->10 so the
  slow-to-reconcile calico-apiserver deployment has enough time under nested-virt
- Raise the molecule step timeout-minutes from 90 to 150 to accommodate
  contended 5-node scenarios (cilium, kube-vip) that were hitting the 90-min cap

* fix(calico): treat optional API server as best-effort on converge

- The Calico API server (calico-apiserver) is an optional add-on for managing
  Calico policy through the projectcalico.org/v3 Kubernetes API; it is not
  required for Calico CNI data plane operation
- With Calico v3.32.1 on K3s 1.36 the tigera-operator never provisions the
  calico-apiserver namespace, causing the converge wait to fail deterministically
- Keep the strict wait for core Calico components (typha, kube-controllers,
  calico-node, csi-node-driver) and make the API server wait tolerate failure
- Restrict the TigeraStatus Available check to the calico status, matching the
  upstream v3.32.1 K3s quickstart which validates without the API server
This commit is contained in:
Techno Tim
2026-08-02 12:44:50 -05:00
committed by GitHub
parent b6363cdfc5
commit bb006cf157
31 changed files with 1129 additions and 166 deletions
+3
View File
@@ -1,5 +1,8 @@
---
- name: Verify
hosts: all
vars_files:
- >-
{{ lookup("ansible.builtin.env", "MOLECULE_SCENARIO_DIRECTORY") }}/verify-vars.yml
roles:
- verify_from_outside
@@ -7,6 +7,8 @@
ansible.builtin.import_tasks: kubecfg-fetch.yml
- name: "TEST CASE: Get nodes"
ansible.builtin.include_tasks: test/get-nodes.yml
- name: "TEST CASE: Verify components"
ansible.builtin.include_tasks: test/verify-components.yml
- name: "TEST CASE: Deploy example"
ansible.builtin.include_tasks: test/deploy-example.yml
always:
@@ -32,17 +32,86 @@
metallb_port: spec.ports[0].port
register: nginx_services
- name: Wait for the load balancer address to be assigned
ansible.builtin.set_fact:
nginx_lb_ip: >-
{{
nginx_services.resources[0].status.loadBalancer.ingress[0].ip
if (nginx_services.resources | length > 0) and
(nginx_services.resources[0].status.loadBalancer.ingress is defined) and
(nginx_services.resources[0].status.loadBalancer.ingress | length > 0)
else ''
}}
- name: Retry until the load balancer service has an external IP
block:
- name: Refresh nginx service until it has an assigned address
kubernetes.core.k8s_info:
kind: service
name: nginx
namespace: "{{ testing_namespace }}"
kubeconfig: "{{ kubecfg_path }}"
register: nginx_lb_wait
until: >-
(nginx_lb_wait.resources | length > 0) and
(nginx_lb_wait.resources[0].status.loadBalancer.ingress is defined) and
(nginx_lb_wait.resources[0].status.loadBalancer.ingress | length > 0)
retries: 30
delay: 5
- name: Record the assigned load balancer address
ansible.builtin.set_fact:
nginx_lb_ip: >-
{{ nginx_lb_wait.resources[0].status.loadBalancer.ingress[0].ip }}
- name: Assert that the nginx welcome page is available
ansible.builtin.uri:
url: http://{{ ip | ansible.utils.ipwrap }}:{{ port_ }}/
url: http://{{ nginx_lb_ip | ansible.utils.ipwrap }}:{{ port_ }}/
return_content: true
register: result
failed_when: "'Welcome to nginx!' not in result.content"
vars:
ip: >-
{{ nginx_services.resources[0].status.loadBalancer.ingress[0].ip }}
port_: >-
{{ nginx_services.resources[0].spec.ports[0].port }}
- name: Initialize load balancer address range check
ansible.builtin.set_fact:
lb_addr_in_range: false
lb_ip_value: "{{ nginx_lb_ip }}"
- name: Check load balancer address against start-end pools
ansible.builtin.set_fact:
lb_addr_in_range: true
loop: "{{ verify_lb_ip_range }}"
loop_control:
label: "{{ item }}"
when:
- "'-' in item"
- "'/' not in item"
- >-
(lb_ip_value | ansible.utils.ipaddr('int') | int) >=
(item.split('-')[0] | ansible.utils.ipaddr('int') | int)
- >-
(lb_ip_value | ansible.utils.ipaddr('int') | int) <=
(item.split('-')[1] | ansible.utils.ipaddr('int') | int)
- name: Check load balancer address against CIDR pools
ansible.builtin.set_fact:
lb_addr_in_range: true
loop: "{{ verify_lb_ip_range }}"
loop_control:
label: "{{ item }}"
when:
- "'/' in item"
- (lb_ip_value | ansible.utils.ipaddr(item)) is string
- name: Assert that the load balancer address is within a configured pool
ansible.builtin.assert:
that: lb_addr_in_range
success_msg: "LoadBalancer address {{ lb_ip_value }} is in a configured range"
fail_msg: >-
LoadBalancer address {{ lb_ip_value }} is not in a configured
range {{ verify_lb_ip_range }}
# Deactivated linter rules:
# - jinja[invalid]: As of version 6.6.0, ansible-lint complains that the input to ipwrap
# would be undefined. This will not be the case during playbook execution.
@@ -0,0 +1,313 @@
---
# Scenario-aware verification of cluster components and their live image tags.
# Scenario identity (verify_cni / verify_lb) and expected address range come
# from each scenario's verify-vars.yml, which is plain inventory data available
# to the verify play. Converge-time set_fact values are not persisted between
# the two Ansible processes, so they are never used here.
- name: Verify cluster components report expected versions
block:
- name: Get all nodes with their kubelet versions
kubernetes.core.k8s_info:
kind: node
kubeconfig: "{{ kubecfg_path }}"
register: verify_nodes
- name: Assert each node reports the expected Kubernetes version
ansible.builtin.assert:
that: item.status.nodeInfo.kubeletVersion == k3s_version
success_msg: "{{ item.metadata.name }} reports {{ k3s_version }}"
fail_msg: >-
{{ item.metadata.name }} reports
{{ item.status.nodeInfo.kubeletVersion }},
expected {{ k3s_version }}
loop: "{{ verify_nodes.resources }}"
loop_control:
label: "{{ item.metadata.name }}"
- name: Verify Flannel is the active CNI
when: verify_cni == 'flannel'
block:
- name: Assert every node reports Ready
ansible.builtin.assert:
that: item.status.conditions
| selectattr('type', 'equalto', 'Ready')
| map(attribute='status') | first | default('') == 'True'
success_msg: "{{ item.metadata.name }} is Ready"
fail_msg: "{{ item.metadata.name }} is not Ready"
loop: "{{ verify_nodes.resources }}"
loop_control:
label: "{{ item.metadata.name }} ready"
- name: Get any Calico namespaces with Flannel enabled
kubernetes.core.k8s_info:
kind: Namespace
name: calico-system
kubeconfig: "{{ kubecfg_path }}"
register: flannel_calico_absent
- name: Assert there is no Calico system namespace
ansible.builtin.assert:
that: flannel_calico_absent.resources | length == 0
success_msg: "No Calico present with Flannel"
fail_msg: "A Calico namespace exists alongside Flannel"
- name: Get the Cilium namespace with Flannel enabled
kubernetes.core.k8s_info:
kind: Namespace
name: cilium
kubeconfig: "{{ kubecfg_path }}"
register: flannel_cilium
- name: Assert the Cilium namespace is absent
ansible.builtin.assert:
that: flannel_cilium.resources | length == 0
success_msg: "No Cilium present with Flannel"
fail_msg: "A Cilium namespace exists alongside Flannel"
- name: Verify Calico is the active CNI
when: verify_cni == 'calico'
block:
- name: Get the Calico node DaemonSet image
kubernetes.core.k8s_info:
kind: DaemonSet
name: calico-node
namespace: calico-system
kubeconfig: "{{ kubecfg_path }}"
register: calico_node_ds
- name: Assert the Calico node image uses the expected tag
ansible.builtin.assert:
that:
- calico_node_ds.resources | length == 1
- calico_node_image | regex_search(':' ~ calico_tag)
success_msg: "Calico node image uses tag {{ calico_tag }}"
fail_msg: >-
Calico node image {{ calico_node_image }},
expected {{ calico_tag }}
vars:
calico_node_image: "{{ calico_node_ds.resources[0].spec.template.spec.containers[0].image }}"
- name: Get Calico TigeraStatus for calico and apiserver
kubernetes.core.k8s_info:
api_version: operator.tigera.io/v1
kind: TigeraStatus
name: "{{ item }}"
kubeconfig: "{{ kubecfg_path }}"
register: calico_tigerastatus
loop:
- calico
- apiserver
loop_control:
label: "Tigerastatus/{{ item }}"
- name: Assert Calico TigeraStatus reports Available
ansible.builtin.assert:
that: >-
item.resources | length == 1 and
(item.resources[0].status.conditions
| selectattr('type', 'equalto', 'Available')
| map(attribute='status') | first | default('')) == 'True'
success_msg: "Tigerastatus {{ item.resources[0].metadata.name }} is Available"
fail_msg: "Tigerastatus is not Available"
loop: "{{ calico_tigerastatus.results }}"
loop_control:
label: "Tigerastatus Available"
- name: Get any Flannel DaemonSets with Calico enabled
kubernetes.core.k8s_info:
kind: DaemonSet
namespace: kube-flannel
kubeconfig: "{{ kubecfg_path }}"
register: no_flannel_ds
- name: Assert there are no Flannel DaemonSets
ansible.builtin.assert:
that: no_flannel_ds.resources | length == 0
success_msg: "No Flannel DaemonSet present with Calico"
fail_msg: "A Flannel DaemonSet exists alongside Calico"
- name: Verify Cilium is the active CNI
when: verify_cni == 'cilium'
block:
- name: Get the Cilium agent and operator images
kubernetes.core.k8s_info:
kind: "{{ item.kind }}"
name: "{{ item.name }}"
namespace: kube-system
kubeconfig: "{{ kubecfg_path }}"
register: cilium_info
loop:
- { kind: DaemonSet, name: cilium }
- { kind: Deployment, name: cilium-operator }
loop_control:
label: "{{ item.kind }}/{{ item.name }}"
- name: Assert Cilium agent and operator use the expected image tag
ansible.builtin.assert:
that:
- cilium_agent_image | regex_search(':' ~ cilium_tag)
- cilium_operator_image | regex_search(':' ~ cilium_tag)
success_msg: "Cilium agent and operator use {{ cilium_tag }}"
fail_msg: >-
Cilium agent {{ cilium_agent_image }},
operator {{ cilium_operator_image }},
expected {{ cilium_tag }}
vars:
cilium_agent_image: >-
{{ (cilium_info.results
| selectattr('resources', 'defined')
| map(attribute='resources')
| list
| map(attribute='0')
| selectattr('kind', 'equalto', 'DaemonSet')
| list)[0].spec.template.spec.containers[0].image }}
cilium_operator_image: >-
{{ (cilium_info.results
| selectattr('resources', 'defined')
| map(attribute='resources')
| list
| map(attribute='0')
| selectattr('kind', 'equalto', 'Deployment')
| list)[0].spec.template.spec.containers[0].image }}
- name: Get Hubble relay and UI deployments when enabled
kubernetes.core.k8s_info:
kind: Deployment
name: "{{ item }}"
namespace: kube-system
kubeconfig: "{{ kubecfg_path }}"
register: hubble_info
loop:
- hubble-relay
- hubble-ui
loop_control:
label: "Deployment/{{ item }}"
when: cilium_hubble | bool
- name: Assert Hubble components are Ready when enabled
ansible.builtin.assert:
that:
- item.resources | length == 1
- item.resources[0].status.readyReplicas | default(0) >= 1
success_msg: "Hubble deployment {{ item.resources[0].metadata.name }} is Ready"
fail_msg: "Hubble deployment is not Ready"
loop: "{{ hubble_info.results }}"
loop_control:
label: "Hubble deployment"
when: cilium_hubble | bool
- name: Get any Flannel DaemonSets with Cilium enabled
kubernetes.core.k8s_info:
kind: DaemonSet
namespace: kube-flannel
kubeconfig: "{{ kubecfg_path }}"
register: no_flannel_ds_cilium
- name: Assert there are no Flannel DaemonSets
ansible.builtin.assert:
that: no_flannel_ds_cilium.resources | length == 0
success_msg: "No Flannel DaemonSet present with Cilium"
fail_msg: "A Flannel DaemonSet exists alongside Cilium"
- name: Verify MetalLB is the active load balancer
when: verify_lb == 'metallb'
block:
- name: Get the MetalLB controller and speaker images
kubernetes.core.k8s_info:
kind: "{{ item.kind }}"
name: "{{ item.name }}"
namespace: metallb-system
kubeconfig: "{{ kubecfg_path }}"
register: metallb_info
loop:
- { kind: Deployment, name: controller }
- { kind: DaemonSet, name: speaker }
loop_control:
label: "{{ item.kind }}/{{ item.name }}"
- name: Assert MetalLB controller and speaker use the expected image tags
ansible.builtin.assert:
that:
- controller_image | regex_search(metal_lb_controller_tag_version)
- speaker_image | regex_search(metal_lb_speaker_tag_version)
success_msg: >-
MetalLB controller {{ metal_lb_controller_tag_version }},
speaker {{ metal_lb_speaker_tag_version }}
fail_msg: >-
MetalLB controller {{ controller_image }},
speaker {{ speaker_image }}
vars:
controller_image: >-
{{ (metallb_info.results
| selectattr('resources', 'defined')
| map(attribute='resources')
| list
| map(attribute='0')
| selectattr('kind', 'equalto', 'Deployment')
| list)[0].spec.template.spec.containers[0].image }}
speaker_image: >-
{{ (metallb_info.results
| selectattr('resources', 'defined')
| map(attribute='resources')
| list
| map(attribute='0')
| selectattr('kind', 'equalto', 'DaemonSet')
| list)[0].spec.template.spec.containers[0].image }}
- name: Verify kube-vip is the active load balancer
when: verify_lb == 'kube-vip'
block:
- name: Get the kube-vip and cloud provider images
kubernetes.core.k8s_info:
kind: "{{ item.kind }}"
name: "{{ item.name }}"
namespace: kube-system
kubeconfig: "{{ kubecfg_path }}"
register: kubevip_info
loop:
- { kind: DaemonSet, name: kube-vip-ds }
- { kind: Deployment, name: kube-vip-cloud-provider }
loop_control:
label: "{{ item.kind }}/{{ item.name }}"
- name: Assert the kube-vip and cloud provider image tags
ansible.builtin.assert:
that:
- kubevip_image | regex_search(':' ~ kube_vip_tag_version)
- cloud_provider_image | regex_search(verify_kube_vip_cloud_provider_tag)
success_msg: >-
kube-vip {{ kube_vip_tag_version }},
cloud provider {{ verify_kube_vip_cloud_provider_tag }}
fail_msg: >-
kube-vip {{ kubevip_image }},
cloud provider {{ cloud_provider_image }}
vars:
kubevip_image: >-
{{ (kubevip_info.results
| selectattr('resources', 'defined')
| map(attribute='resources')
| list
| map(attribute='0')
| selectattr('kind', 'equalto', 'DaemonSet')
| list)[0].spec.template.spec.containers[0].image }}
cloud_provider_image: >-
{{ (kubevip_info.results
| selectattr('resources', 'defined')
| map(attribute='resources')
| list
| map(attribute='0')
| selectattr('kind', 'equalto', 'Deployment')
| list)[0].spec.template.spec.containers[0].image }}
- name: Get the MetalLB namespace with kube-vip enabled
kubernetes.core.k8s_info:
kind: Namespace
name: metallb-system
kubeconfig: "{{ kubecfg_path }}"
register: metallb_absent
- name: Assert the MetalLB namespace does not exist
ansible.builtin.assert:
that: metallb_absent.resources | length == 0
success_msg: "MetalLB is not installed with kube-vip"
fail_msg: "MetalLB namespace exists alongside kube-vip"