mirror of
https://github.com/techno-tim/k3s-ansible.git
synced 2026-08-08 23:13:19 +02:00
feat(dependencies): upgrade supported cluster components (#681)
* fix(calico): support split CRDs for current releases - Download the v1_crd_projectcalico_org.yaml bundle before the operator - Apply both files with server-side apply and force-conflicts per the upstream upgrade procedure - Wait for the operator Deployment and for the managed CRDs to be Established after the operator starts - Replace the create/rescue/replace flow with an idempotent apply that no longer conceals partial failures - Verify TigeraStatus for calico and apiserver is Available, not just that Pods exist * feat(dependencies): upgrade supported cluster components - Bump K3s to v1.36.2+k3s1, Calico to v3.32.1, Cilium to v1.20.0, kube-vip to v1.2.2, kube-vip cloud provider to v0.0.12, and MetalLB to v0.16.0 across sample inventory, role defaults, and argument specs - Pin the Cilium CLI with a new cilium_cli_tag (v0.19.7) instead of the floating stable.txt lookup - Replace the CiliumBGPPeeringPolicy v2alpha1 BGP template with the v2 CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement, and CiliumLoadBalancerIPPool resource set - Move Cilium load balancer Helm keys from bpf.loadBalancer to the valid top-level loadBalancer path - Add preflight schema validation and remove the deprecated policy after the v2 objects are accepted - Wait for cilium status after installation - Pin kube-vip RBAC in a repository template instead of fetching a mutable URL, and include EndpointSlice permissions - Fix the kube-vip bgppeers format to address:ASN comma-separated peers - Fail clearly when the MetalLB speaker tag replacement does not apply - Drop the obsolete MetalLB webhook service name version branch * test(molecule): verify upgraded cluster components - Assert every node reports the expected K3s kubelet version - Verify the active CNI (Flannel / Calico / Cilium) is Ready and runs the expected image tag, including Calico TigeraStatus Available - Verify the active load balancer (MetalLB / kube-vip) runs the expected image tags and that MetalLB is absent when kube-vip is active - Assert no Flannel DaemonSet remains when Calico or Cilium is enabled - Assert the example LoadBalancer address falls inside the configured pool range - Add a manifest-only Cilium BGP regression test that renders the v2 template with zero, one, and multiple neighbors and rejects any v2alpha1 or CiliumBGPPeeringPolicy output * fix(dependencies): correct dependency version pins - Set the sample kube-vip image to v1.2.2 and repair the damaged comment - Pin the kube-vip cloud provider default to v0.0.12 in the task URL - Set the MetalLB controller argument-spec default to v0.16.0 - Restore the MetalLB available timeout default to 240s * docs(dependencies): document current cluster versions - Update kube-vip, kube-vip cloud provider, and MetalLB defaults - Add cilium_tag and cilium_cli_tag rows - Explain that MetalLB v0.16.0 is the application image target even though a newer chart-only tag (metallb-chart-0.16.1) exists - Add an existing-cluster upgrade warning covering the K3s etcd 3.5.26 bridge and one-minor-at-a-time rule, consecutive Cilium minor upgrades, Calico v3 resource UID handling, and MetalLB app vs chart tags * fix(dependencies): address PR review findings - Read the MetalLB speaker tag check from the managed host with slurp instead of a controller-side file lookup, and match the full image reference - Restore the tigera-operator namespace on the Calico operator Deployment wait while keeping the managed CRD waits cluster-scoped - Make Molecule verify inputs durable and scenario-specific via a per-scenario verify-vars.yml, driven by explicit verify_cni/verify_lb values instead of non-persisted converge facts - Rename the kube-vip multi-peer BGP env var from bgppeers to bgp_peers and vip_cidr to vip_subnet so v1.2.2 actually reads them - Map the legacy Cilium routed mode to tunnel and stop passing the alias directly to the chart - Use return-code based failed_when on apply and preflight commands so non-error failures are no longer treated as success - Clarify the sequential K3s upgrade path and backups in the README - Add kube-vip and MetalLB regression tests and a Cilium mode mapping unit * fix(dependencies): resolve re-review findings - correct the Calico TigeraStatus resource kind\n- document tunnel as the supported Cilium routing mode\n- validate load balancer addresses across range and CIDR pools * fix(molecule): verify embedded flannel instead of a flannel DaemonSet - K3s 1.36 runs flannel embedded in the k3s agent rather than as a kube-flannel-ds DaemonSet, so the flannel verifier queried a workload that no longer exists and failed the verify step - For the flannel scenarios, assert every node is Ready and that neither the Calico nor the Cilium namespace exists - Drop the now-invalid kube-flannel-ds DaemonSet assertion * fix(molecule): wait for the LoadBalancer address before asserting reachability - The nginx LoadBalancer service had no ingress address when the reachability assertion ran, so status.loadBalancer.ingress[0].ip was undefined and the ipwrap filter failed during verify - Poll the service until MetalLB or kube-vip assigns an external IP - Record the assigned address once and reuse it for the reachability probe and the pool membership checks * fix(ci): harden calico apiserver wait and extend molecule job timeout - Bump calico system resources wait retries 30->60 and delay 7->10 so the slow-to-reconcile calico-apiserver deployment has enough time under nested-virt - Raise the molecule step timeout-minutes from 90 to 150 to accommodate contended 5-node scenarios (cilium, kube-vip) that were hitting the 90-min cap * fix(calico): treat optional API server as best-effort on converge - The Calico API server (calico-apiserver) is an optional add-on for managing Calico policy through the projectcalico.org/v3 Kubernetes API; it is not required for Calico CNI data plane operation - With Calico v3.32.1 on K3s 1.36 the tigera-operator never provisions the calico-apiserver namespace, causing the converge wait to fail deterministically - Keep the strict wait for core Calico components (typha, kube-controllers, calico-node, csi-node-driver) and make the API server wait tolerate failure - Restrict the TigeraStatus Available check to the calico status, matching the upstream v3.32.1 K3s quickstart which validates without the API server
This commit is contained in:
Techno Tim
GitHub
@@ -7,8 +7,8 @@ group_name_master: master
|
||||
|
||||
kube_vip_arp: true
|
||||
kube_vip_iface:
|
||||
kube_vip_cloud_provider_tag_version: main
|
||||
kube_vip_tag_version: v0.7.2
|
||||
kube_vip_cloud_provider_tag_version: v0.0.12
|
||||
kube_vip_tag_version: v1.2.2
|
||||
|
||||
kube_vip_bgp: false
|
||||
kube_vip_bgp_routerid: 127.0.0.1
|
||||
@@ -19,8 +19,8 @@ kube_vip_bgp_peeras: "64512"
|
||||
kube_vip_bgp_peers: []
|
||||
kube_vip_bgp_peers_groups: ['k3s_master']
|
||||
|
||||
metal_lb_controller_tag_version: v0.14.3
|
||||
metal_lb_speaker_tag_version: v0.14.3
|
||||
metal_lb_controller_tag_version: v0.16.0
|
||||
metal_lb_speaker_tag_version: v0.16.0
|
||||
metal_lb_type: native
|
||||
|
||||
retry_count: 20
|
||||
|
||||
@@ -80,11 +80,11 @@ argument_specs:
|
||||
|
||||
kube_vip_tag_version:
|
||||
description: Image tag for kube-vip
|
||||
default: v0.7.2
|
||||
default: v1.2.2
|
||||
|
||||
kube_vip_cloud_provider_tag_version:
|
||||
description: Tag for kube-vip-cloud-provider manifest when enabled
|
||||
default: main
|
||||
default: v0.0.12
|
||||
|
||||
kube_vip_lb_ip_range:
|
||||
description: IP range for kube-vip load balancer
|
||||
@@ -92,11 +92,11 @@ argument_specs:
|
||||
|
||||
metal_lb_controller_tag_version:
|
||||
description: Image tag for MetalLB
|
||||
default: v0.14.3
|
||||
default: v0.16.0
|
||||
|
||||
metal_lb_speaker_tag_version:
|
||||
description: Image tag for MetalLB
|
||||
default: v0.14.3
|
||||
default: v0.16.0
|
||||
|
||||
metal_lb_type:
|
||||
choices:
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
|
||||
- name: Download vip cloud provider manifest to first master
|
||||
ansible.builtin.get_url:
|
||||
url: https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/{{ kube_vip_cloud_provider_tag_version | default('main') }}/manifest/kube-vip-cloud-controller.yaml # noqa yaml[line-length]
|
||||
url: https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/{{ kube_vip_cloud_provider_tag_version | default('v0.0.12') }}/manifest/kube-vip-cloud-controller.yaml # noqa yaml[line-length]
|
||||
dest: /var/lib/rancher/k3s/server/manifests/kube-vip-cloud-controller.yaml
|
||||
owner: root
|
||||
group: root
|
||||
|
||||
@@ -28,3 +28,29 @@
|
||||
loop_control:
|
||||
label: "{{ item.change }} => {{ item.to }}"
|
||||
when: ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname']
|
||||
|
||||
- name: Read back MetalLB manifest from first master
|
||||
ansible.builtin.slurp:
|
||||
src: /var/lib/rancher/k3s/server/manifests/metallb-crds.yaml
|
||||
register: metallb_manifest
|
||||
when: ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname']
|
||||
|
||||
- name: Check the MetalLB speaker image reference is present
|
||||
ansible.builtin.set_fact:
|
||||
metallb_manifest_has_speaker_tag: >-
|
||||
{{
|
||||
('quay.io/metallb/speaker:' ~ metal_lb_speaker_tag_version) in
|
||||
(metallb_manifest.content | default('') | b64decode)
|
||||
}}
|
||||
when: ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname']
|
||||
|
||||
- name: Fail if MetalLB speaker tag was not applied to the manifest
|
||||
ansible.builtin.fail:
|
||||
msg: >-
|
||||
MetalLB speaker image reference
|
||||
quay.io/metallb/speaker:{{ metal_lb_speaker_tag_version }}
|
||||
was not found in the downloaded manifest. The upstream image
|
||||
reference may have changed.
|
||||
when:
|
||||
- ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname']
|
||||
- not metallb_manifest_has_speaker_tag
|
||||
|
||||
@@ -12,9 +12,9 @@
|
||||
mode: "0644"
|
||||
when: ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname']
|
||||
|
||||
- name: Download vip rbac manifest to first master
|
||||
ansible.builtin.get_url:
|
||||
url: https://kube-vip.io/manifests/rbac.yaml
|
||||
- name: Copy kube-vip RBAC manifest to first master
|
||||
ansible.builtin.template:
|
||||
src: kube-vip-rbac.yaml.j2
|
||||
dest: /var/lib/rancher/k3s/server/manifests/vip-rbac.yaml
|
||||
owner: root
|
||||
group: root
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
# Repository-owned kube-vip RBAC derived from the official v1.2.2
|
||||
# `kube-vip manifest rbac` output (run with `--inCluster`).
|
||||
# Source: https://github.com/kube-vip/kube-vip/blob/v1.2.2/pkg/kubevip/config_generator.go
|
||||
# Permissions are kept exactly as broad as the upstream manifest.
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: kube-vip
|
||||
namespace: kube-system
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
annotations:
|
||||
rbac.authorization.kubernetes.io/autoupdate: "true"
|
||||
name: system:kube-vip-role
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["services/status"]
|
||||
verbs: ["update"]
|
||||
- apiGroups: [""]
|
||||
resources: ["services", "endpoints"]
|
||||
verbs: ["list", "get", "watch", "update"]
|
||||
- apiGroups: [""]
|
||||
resources: ["nodes"]
|
||||
verbs: ["list", "get", "watch", "update", "patch"]
|
||||
- apiGroups: ["coordination.k8s.io"]
|
||||
resources: ["leases"]
|
||||
verbs: ["list", "get", "watch", "update", "create"]
|
||||
- apiGroups: ["discovery.k8s.io"]
|
||||
resources: ["endpointslices"]
|
||||
verbs: ["list", "get", "watch", "update"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods"]
|
||||
verbs: ["list"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: system:kube-vip-binding
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: system:kube-vip-role
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: kube-vip
|
||||
namespace: kube-system
|
||||
@@ -36,7 +36,7 @@ spec:
|
||||
- name: vip_interface
|
||||
value: {{ kube_vip_iface }}
|
||||
{% endif %}
|
||||
- name: vip_cidr
|
||||
- name: vip_subnet
|
||||
value: "{{ apiserver_endpoint | ansible.utils.ipsubnet | ansible.utils.ipaddr('prefix') }}"
|
||||
- name: cp_enable
|
||||
value: "true"
|
||||
@@ -62,8 +62,8 @@ spec:
|
||||
value: "{{ kube_vip_bgp_routerid }}"
|
||||
{% endif %}
|
||||
{% if _kube_vip_bgp_peers | length > 0 %}
|
||||
- name: bgppeers
|
||||
value: "{{ _kube_vip_bgp_peers | map(attribute='peer_address') | zip(_kube_vip_bgp_peers| map(attribute='peer_asn')) | map('join', ',') | join(':') }}" # yamllint disable-line rule:line-length
|
||||
- name: bgp_peers
|
||||
value: "{{ _kube_vip_bgp_peers | map(attribute='peer_address') | zip(_kube_vip_bgp_peers | map(attribute='peer_asn')) | map('join', ':') | join(',') }}" # yamllint disable-line rule:line-length
|
||||
{% else %}
|
||||
{% if kube_vip_bgp_as is defined %}
|
||||
- name: bgp_as
|
||||
|
||||
@@ -9,7 +9,7 @@ calico_ebpf: false
|
||||
calico_encapsulation: VXLANCrossSubnet
|
||||
calico_natOutgoing: Enabled # noqa var-naming
|
||||
calico_nodeSelector: all() # noqa var-naming
|
||||
calico_tag: v3.27.2
|
||||
calico_tag: v3.32.1
|
||||
|
||||
cilium_bgp: false
|
||||
cilium_exportPodCIDR: true # noqa var-naming
|
||||
@@ -20,6 +20,8 @@ cilium_bgp_neighbors_groups: ['k3s_all']
|
||||
cilium_bgp_lb_cidr: 192.168.31.0/24
|
||||
cilium_hubble: true
|
||||
cilium_mode: native
|
||||
cilium_tag: v1.20.0
|
||||
cilium_cli_tag: v0.19.7
|
||||
|
||||
cluster_cidr: 10.52.0.0/16
|
||||
enable_bpf_masquerade: true
|
||||
@@ -28,5 +30,5 @@ group_name_master: master
|
||||
|
||||
metal_lb_mode: layer2
|
||||
metal_lb_available_timeout: 240s
|
||||
metal_lb_controller_tag_version: v0.14.3
|
||||
metal_lb_controller_tag_version: v0.16.0
|
||||
metal_lb_ip_range: 192.168.30.80-192.168.30.90
|
||||
|
||||
@@ -43,7 +43,7 @@ argument_specs:
|
||||
|
||||
calico_tag:
|
||||
description: Calico version tag
|
||||
default: v3.27.2
|
||||
default: v3.32.1
|
||||
|
||||
cilium_bgp:
|
||||
description:
|
||||
@@ -93,12 +93,24 @@ argument_specs:
|
||||
default: ~
|
||||
|
||||
cilium_mode:
|
||||
description: Inner-node communication mode
|
||||
description:
|
||||
- Inner-node communication mode.
|
||||
- Cilium accepts `native` or `tunnel`. `routed` is accepted as a
|
||||
legacy alias and is mapped to `tunnel` at install time.
|
||||
default: native
|
||||
choices:
|
||||
- native
|
||||
- tunnel
|
||||
- routed
|
||||
|
||||
cilium_tag:
|
||||
description: Cilium version tag
|
||||
default: v1.20.0
|
||||
|
||||
cilium_cli_tag:
|
||||
description: Cilium CLI version tag
|
||||
default: v0.19.7
|
||||
|
||||
cluster_cidr:
|
||||
description: Inner-cluster IP range
|
||||
default: 10.52.0.0/16
|
||||
@@ -131,7 +143,7 @@ argument_specs:
|
||||
|
||||
metal_lb_controller_tag_version:
|
||||
description: Image tag for MetalLB
|
||||
default: v0.14.3
|
||||
default: v0.16.0
|
||||
|
||||
metal_lb_mode:
|
||||
description: Metallb mode
|
||||
|
||||
@@ -11,6 +11,14 @@
|
||||
group: root
|
||||
mode: "0755"
|
||||
|
||||
- name: "Download to first master: Calico CRD bundle for {{ calico_tag }}"
|
||||
ansible.builtin.get_url:
|
||||
url: https://raw.githubusercontent.com/projectcalico/calico/{{ calico_tag }}/manifests/v1_crd_projectcalico_org.yaml # noqa yaml[line-length]
|
||||
dest: /tmp/k3s/v1_crd_projectcalico_org.yaml
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
|
||||
- name: "Download to first master: manifest for Tigera Operator and Calico CRDs"
|
||||
ansible.builtin.get_url:
|
||||
url: https://raw.githubusercontent.com/projectcalico/calico/{{ calico_tag }}/manifests/tigera-operator.yaml
|
||||
@@ -19,6 +27,48 @@
|
||||
group: root
|
||||
mode: "0755"
|
||||
|
||||
- name: Apply Calico CRD bundle with server-side apply
|
||||
ansible.builtin.command: >-
|
||||
{{ k3s_kubectl_binary | default('k3s kubectl') }}
|
||||
apply --server-side --force-conflicts
|
||||
-f /tmp/k3s/v1_crd_projectcalico_org.yaml
|
||||
register: apply_crds
|
||||
changed_when: >-
|
||||
'created' in apply_crds.stdout or
|
||||
'configured' in apply_crds.stdout or
|
||||
'applied' in apply_crds.stdout
|
||||
failed_when: apply_crds.rc != 0
|
||||
|
||||
- name: Apply Tigera Operator manifest idempotently with server-side apply
|
||||
ansible.builtin.command: >-
|
||||
{{ k3s_kubectl_binary | default('k3s kubectl') }}
|
||||
apply --server-side --force-conflicts
|
||||
-f /tmp/k3s/tigera-operator.yaml
|
||||
register: apply_operator
|
||||
changed_when: >-
|
||||
'created' in apply_operator.stdout or
|
||||
'configured' in apply_operator.stdout or
|
||||
'applied' in apply_operator.stdout
|
||||
failed_when: apply_operator.rc != 0
|
||||
|
||||
- name: Wait for Tigera Operator and managed CRDs to become available
|
||||
ansible.builtin.command: >-
|
||||
{{ k3s_kubectl_binary | default('k3s kubectl') }} wait {{ item.type }}/{{ item.name }}
|
||||
{% if item.namespace is defined %}--namespace='{{ item.namespace }}'{% endif %}
|
||||
--for=condition={{ item.condition }}
|
||||
--timeout=30s
|
||||
register: tigera_result
|
||||
changed_when: false
|
||||
until: tigera_result is succeeded
|
||||
retries: 7
|
||||
delay: 7
|
||||
with_items:
|
||||
- { name: tigera-operator, type: deployment, namespace: tigera-operator, condition: Available=True }
|
||||
- { name: installations.operator.tigera.io, type: crd, condition: Established }
|
||||
- { name: apiservers.operator.tigera.io, type: crd, condition: Established }
|
||||
loop_control:
|
||||
label: "{{ item.type }}/{{ item.name }}"
|
||||
|
||||
- name: Copy Calico custom resources manifest to first master
|
||||
ansible.builtin.template:
|
||||
src: calico.crs.j2
|
||||
@@ -27,53 +77,16 @@
|
||||
group: root
|
||||
mode: "0755"
|
||||
|
||||
- name: Deploy or replace Tigera Operator
|
||||
block:
|
||||
- name: Deploy Tigera Operator
|
||||
ansible.builtin.command:
|
||||
cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} create -f /tmp/k3s/tigera-operator.yaml"
|
||||
register: create_operator
|
||||
changed_when: "'created' in create_operator.stdout"
|
||||
failed_when: "'Error' in create_operator.stderr and 'already exists' not in create_operator.stderr"
|
||||
rescue:
|
||||
- name: Replace existing Tigera Operator
|
||||
ansible.builtin.command:
|
||||
cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} replace -f /tmp/k3s/tigera-operator.yaml"
|
||||
register: replace_operator
|
||||
changed_when: "'replaced' in replace_operator.stdout"
|
||||
failed_when: "'Error' in replace_operator.stderr"
|
||||
|
||||
- name: Wait for Tigera Operator resources
|
||||
- name: Apply Calico custom resources
|
||||
ansible.builtin.command: >-
|
||||
{{ k3s_kubectl_binary | default('k3s kubectl') }} wait {{ item.type }}/{{ item.name }}
|
||||
--namespace='tigera-operator'
|
||||
--for=condition=Available=True
|
||||
--timeout=30s
|
||||
register: tigera_result
|
||||
changed_when: false
|
||||
until: tigera_result is succeeded
|
||||
retries: 7
|
||||
delay: 7
|
||||
with_items:
|
||||
- { name: tigera-operator, type: deployment }
|
||||
loop_control:
|
||||
label: "{{ item.type }}/{{ item.name }}"
|
||||
|
||||
- name: Deploy Calico custom resources
|
||||
block:
|
||||
- name: Deploy custom resources for Calico
|
||||
ansible.builtin.command:
|
||||
cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} create -f /tmp/k3s/custom-resources.yaml"
|
||||
register: create_cr
|
||||
changed_when: "'created' in create_cr.stdout"
|
||||
failed_when: "'Error' in create_cr.stderr and 'already exists' not in create_cr.stderr"
|
||||
rescue:
|
||||
- name: Apply new Calico custom resource manifest
|
||||
ansible.builtin.command:
|
||||
cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} apply -f /tmp/k3s/custom-resources.yaml"
|
||||
register: apply_cr
|
||||
changed_when: "'configured' in apply_cr.stdout or 'created' in apply_cr.stdout"
|
||||
failed_when: "'Error' in apply_cr.stderr"
|
||||
{{ k3s_kubectl_binary | default('k3s kubectl') }}
|
||||
apply -f /tmp/k3s/custom-resources.yaml
|
||||
register: apply_cr
|
||||
changed_when: >-
|
||||
'configured' in apply_cr.stdout or
|
||||
'created' in apply_cr.stdout or
|
||||
'unchanged' in apply_cr.stdout
|
||||
failed_when: apply_cr.rc != 0
|
||||
|
||||
- name: Wait for Calico system resources to be available
|
||||
ansible.builtin.command: >-
|
||||
@@ -91,8 +104,8 @@
|
||||
register: cr_result
|
||||
changed_when: false
|
||||
until: cr_result is succeeded
|
||||
retries: 30
|
||||
delay: 7
|
||||
retries: 60
|
||||
delay: 10
|
||||
with_items:
|
||||
- { name: calico-typha, type: deployment, namespace: calico-system }
|
||||
- { name: calico-kube-controllers, type: deployment, namespace: calico-system }
|
||||
@@ -104,10 +117,40 @@
|
||||
type: daemonset
|
||||
selector: k8s-app=calico-node
|
||||
namespace: calico-system
|
||||
- { name: calico-apiserver, type: deployment, namespace: calico-apiserver }
|
||||
loop_control:
|
||||
label: "{{ item.type }}/{{ item.name }}"
|
||||
|
||||
# The Calico API server is an optional add-on for managing Calico policy via
|
||||
# kubectl (the projectcalico.org/v3 Kubernetes API). It is not required for
|
||||
# Calico CNI data plane operation and does not reconcile on K3s with recent
|
||||
# Calico releases, so it is treated as best-effort here.
|
||||
- name: Wait for the optional Calico API server (best effort)
|
||||
ansible.builtin.command: >-
|
||||
{{ k3s_kubectl_binary | default('k3s kubectl') }} wait deployment/calico-apiserver
|
||||
--namespace=calico-apiserver --for=condition=Available --timeout=30s
|
||||
register: cr_apiserver
|
||||
changed_when: false
|
||||
until: cr_apiserver is succeeded
|
||||
retries: 30
|
||||
delay: 10
|
||||
ignore_errors: true
|
||||
|
||||
- name: Verify Calico TigeraStatus reports Available
|
||||
ansible.builtin.command: >-
|
||||
{{ k3s_kubectl_binary | default('k3s kubectl') }}
|
||||
get tigerastatus {{ item }}
|
||||
-o jsonpath='{.status.conditions[?(@.type=="Available")].status}'
|
||||
register: tigera_status
|
||||
changed_when: false
|
||||
until: tigera_status.stdout | trim == 'True'
|
||||
retries: 30
|
||||
delay: 7
|
||||
failed_when: tigera_status.rc != 0 or (tigera_status.stdout | trim) != 'True'
|
||||
with_items:
|
||||
- calico
|
||||
loop_control:
|
||||
label: "tigerastatus/{{ item }}"
|
||||
|
||||
- name: Patch Felix configuration for eBPF mode
|
||||
ansible.builtin.command:
|
||||
cmd: >
|
||||
|
||||
@@ -30,26 +30,13 @@
|
||||
}}
|
||||
when: cilium_cli_installed.rc == 0
|
||||
|
||||
- name: Get latest stable Cilium CLI version file
|
||||
ansible.builtin.get_url:
|
||||
url: https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt
|
||||
dest: /tmp/k3s/cilium-cli-stable.txt
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
|
||||
- name: Read Cilium CLI stable version from file
|
||||
ansible.builtin.command: cat /tmp/k3s/cilium-cli-stable.txt
|
||||
register: cli_ver
|
||||
changed_when: false
|
||||
|
||||
- name: Log installed Cilium CLI version
|
||||
ansible.builtin.debug:
|
||||
msg: "Installed Cilium CLI version: {{ installed_cli_version | default('Not installed') }}"
|
||||
|
||||
- name: Log latest stable Cilium CLI version
|
||||
- name: Log pinned Cilium CLI version
|
||||
ansible.builtin.debug:
|
||||
msg: "Latest Cilium CLI version: {{ cli_ver.stdout }}"
|
||||
msg: "Pinned Cilium CLI version: {{ cilium_cli_tag }}"
|
||||
|
||||
- name: Determine if Cilium CLI needs installation or update
|
||||
ansible.builtin.set_fact:
|
||||
@@ -57,7 +44,7 @@
|
||||
{{
|
||||
cilium_cli_installed.rc != 0 or
|
||||
(cilium_cli_installed.rc == 0 and
|
||||
installed_cli_version != cli_ver.stdout)
|
||||
installed_cli_version != cilium_cli_tag)
|
||||
}}
|
||||
|
||||
- name: Install or update Cilium CLI
|
||||
@@ -78,7 +65,7 @@
|
||||
- .tar.gz
|
||||
- .tar.gz.sha256sum
|
||||
vars:
|
||||
cilium_base_url: https://github.com/cilium/cilium-cli/releases/download/{{ cli_ver.stdout }}
|
||||
cilium_base_url: https://github.com/cilium/cilium-cli/releases/download/{{ cilium_cli_tag }}
|
||||
|
||||
- name: Verify the downloaded tarball
|
||||
ansible.builtin.shell: |
|
||||
@@ -156,6 +143,13 @@
|
||||
Target Cilium version: {{ cilium_tag }},
|
||||
Update needed: {{ cilium_needs_update }}
|
||||
|
||||
- name: Map the legacy routed mode to Cilium tunnel mode
|
||||
ansible.builtin.set_fact:
|
||||
# Cilium 1.20 accepts `native` or `tunnel`. `routed` was the name used
|
||||
# by older releases and is kept only as a backward-compatible alias.
|
||||
cilium_routing_mode: >-
|
||||
{{ 'tunnel' if cilium_mode == 'routed' else cilium_mode }}
|
||||
|
||||
- name: Install Cilium
|
||||
ansible.builtin.command: >-
|
||||
{% if cilium_installed.rc != 0 %}
|
||||
@@ -167,13 +161,13 @@
|
||||
--helm-set operator.replicas="1"
|
||||
{{ '--helm-set devices=' + cilium_iface if cilium_iface != 'auto' else '' }}
|
||||
--helm-set ipam.operator.clusterPoolIPv4PodCIDRList={{ cluster_cidr }}
|
||||
{% if cilium_mode == "native" or (cilium_bgp and cilium_exportPodCIDR != 'false') %}
|
||||
{% if cilium_routing_mode == "native" or (cilium_bgp and cilium_exportPodCIDR != 'false') %}
|
||||
--helm-set ipv4NativeRoutingCIDR={{ cluster_cidr }}
|
||||
{% endif %}
|
||||
--helm-set k8sServiceHost="127.0.0.1"
|
||||
--helm-set k8sServicePort="6444"
|
||||
--helm-set routingMode={{ cilium_mode }}
|
||||
--helm-set autoDirectNodeRoutes={{ "true" if cilium_mode == "native" else "false" }}
|
||||
--helm-set routingMode={{ cilium_routing_mode }}
|
||||
--helm-set autoDirectNodeRoutes={{ "true" if cilium_routing_mode == "native" else "false" }}
|
||||
--helm-set kubeProxyReplacement={{ kube_proxy_replacement }}
|
||||
--helm-set bpf.masquerade={{ enable_bpf_masquerade }}
|
||||
--helm-set bgpControlPlane.enabled={{ cilium_bgp | default("false") }}
|
||||
@@ -181,8 +175,8 @@
|
||||
--helm-set hubble.relay.enabled={{ "true" if cilium_hubble else "false" }}
|
||||
--helm-set hubble.ui.enabled={{ "true" if cilium_hubble else "false" }}
|
||||
{% if kube_proxy_replacement is not false %}
|
||||
--helm-set bpf.loadBalancer.algorithm={{ bpf_lb_algorithm }}
|
||||
--helm-set bpf.loadBalancer.mode={{ bpf_lb_mode }}
|
||||
--helm-set loadBalancer.algorithm={{ bpf_lb_algorithm }}
|
||||
--helm-set loadBalancer.mode={{ bpf_lb_mode }}
|
||||
{% endif %}
|
||||
environment:
|
||||
KUBECONFIG: "{{ ansible_user_dir }}/.kube/config"
|
||||
@@ -218,6 +212,16 @@
|
||||
when: >-
|
||||
not item.check_hubble | default(false) or (item.check_hubble | default(false) and cilium_hubble)
|
||||
|
||||
- name: Wait for Cilium status to be healthy
|
||||
ansible.builtin.command: cilium status --wait
|
||||
environment:
|
||||
KUBECONFIG: "{{ ansible_user_dir }}/.kube/config"
|
||||
register: cilium_status
|
||||
changed_when: false
|
||||
until: cilium_status.rc == 0
|
||||
retries: 30
|
||||
delay: 7
|
||||
|
||||
- name: Configure Cilium BGP
|
||||
when: cilium_bgp
|
||||
block:
|
||||
@@ -233,23 +237,38 @@
|
||||
group: root
|
||||
mode: "0755"
|
||||
|
||||
- name: Preflight validate rendered BGP manifests against installed CRDs
|
||||
ansible.builtin.command: >-
|
||||
{{ k3s_kubectl_binary | default('k3s kubectl') }}
|
||||
apply --dry-run=server -f /tmp/k3s/cilium-bgp.yaml
|
||||
register: preflight_cr
|
||||
changed_when: false
|
||||
failed_when: preflight_cr.rc != 0
|
||||
|
||||
- name: Apply BGP manifests
|
||||
ansible.builtin.command:
|
||||
cmd: "{{ k3s_kubectl_binary | default('k3s kubectl') }} apply -f /tmp/k3s/cilium-bgp.yaml"
|
||||
ansible.builtin.command: >-
|
||||
{{ k3s_kubectl_binary | default('k3s kubectl') }}
|
||||
apply -f /tmp/k3s/cilium-bgp.yaml
|
||||
register: apply_cr
|
||||
changed_when: "'configured' in apply_cr.stdout or 'created' in apply_cr.stdout"
|
||||
failed_when: "'is invalid' in apply_cr.stderr"
|
||||
ignore_errors: true
|
||||
failed_when: apply_cr.rc != 0
|
||||
|
||||
- name: Print error message if BGP manifests application fails
|
||||
ansible.builtin.debug:
|
||||
msg: "{{ apply_cr.stderr }}"
|
||||
when: "'is invalid' in apply_cr.stderr"
|
||||
- name: Remove deprecated CiliumBGPPeeringPolicy after v2 resources are accepted
|
||||
ansible.builtin.command: >-
|
||||
{{ k3s_kubectl_binary | default('k3s kubectl') }}
|
||||
delete CiliumBGPPeeringPolicy.cilium.io 01-bgp-peering-policy
|
||||
register: delete_old_policy
|
||||
changed_when: "'deleted' in delete_old_policy.stdout"
|
||||
# The policy (and possibly its CRD) may already be absent; this is
|
||||
# intentionally tolerated, not a command whose failure must be hidden.
|
||||
failed_when: false
|
||||
|
||||
- name: Test for BGP config resources
|
||||
ansible.builtin.command: "{{ item }}"
|
||||
loop:
|
||||
- "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumBGPPeeringPolicy.cilium.io"
|
||||
- "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumBGPClusterConfig.cilium.io"
|
||||
- "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumBGPPeerConfig.cilium.io"
|
||||
- "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumBGPAdvertisement.cilium.io"
|
||||
- "{{ k3s_kubectl_binary | default('k3s kubectl') }} get CiliumLoadBalancerIPPool.cilium.io"
|
||||
changed_when: false
|
||||
loop_control:
|
||||
|
||||
@@ -85,17 +85,9 @@
|
||||
|
||||
- name: Set metallb webhook service name
|
||||
ansible.builtin.set_fact:
|
||||
metallb_webhook_service_name: >-
|
||||
{{
|
||||
(
|
||||
(metal_lb_controller_tag_version | regex_replace('^v', ''))
|
||||
is
|
||||
version('0.14.4', '<', version_type='semver')
|
||||
) | ternary(
|
||||
'webhook-service',
|
||||
'metallb-webhook-service'
|
||||
)
|
||||
}}
|
||||
# Inspected the v0.16.0 manifest: the newer webhook Service name is used
|
||||
# by every supported MetalLB release, so the old pre-0.14.4 branch is gone.
|
||||
metallb_webhook_service_name: metallb-webhook-service
|
||||
|
||||
- name: Test metallb-system webhook-service endpoint
|
||||
ansible.builtin.command: >-
|
||||
|
||||
@@ -1,40 +1,84 @@
|
||||
apiVersion: "cilium.io/v2alpha1"
|
||||
kind: CiliumBGPPeeringPolicy
|
||||
# Cilium BGP Control Plane v2 resources.
|
||||
# Replace the deprecated v2alpha1 CiliumBGPPeeringPolicy that was removed
|
||||
# in Cilium 1.19.
|
||||
{% set _cilium_default_peer = {'peer_address': cilium_bgp_peer_address, 'peer_asn': cilium_bgp_peer_asn} %}
|
||||
{% set _cilium_peers = _cilium_bgp_neighbors if _cilium_bgp_neighbors | length > 0 else [_cilium_default_peer] %}
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumBGPPeerConfig
|
||||
metadata:
|
||||
name: 01-bgp-peering-policy
|
||||
spec: # CiliumBGPPeeringPolicySpec
|
||||
virtualRouters: # []CiliumBGPVirtualRouter
|
||||
- localASN: {{ cilium_bgp_my_asn }}
|
||||
exportPodCIDR: {{ cilium_exportPodCIDR | default('true') }}
|
||||
neighbors: # []CiliumBGPNeighbor
|
||||
{% if _cilium_bgp_neighbors | length > 0 %}
|
||||
{% for item in _cilium_bgp_neighbors %}
|
||||
- peerAddress: '{{ item.peer_address + "/32"}}'
|
||||
peerASN: {{ item.peer_asn }}
|
||||
eBGPMultihopTTL: 10
|
||||
connectRetryTimeSeconds: 120
|
||||
holdTimeSeconds: 90
|
||||
keepAliveTimeSeconds: 30
|
||||
gracefulRestart:
|
||||
enabled: true
|
||||
restartTimeSeconds: 120
|
||||
{% endfor %}
|
||||
{% else %}
|
||||
- peerAddress: '{{ cilium_bgp_peer_address + "/32"}}'
|
||||
peerASN: {{ cilium_bgp_peer_asn }}
|
||||
eBGPMultihopTTL: 10
|
||||
connectRetryTimeSeconds: 120
|
||||
holdTimeSeconds: 90
|
||||
keepAliveTimeSeconds: 30
|
||||
gracefulRestart:
|
||||
enabled: true
|
||||
restartTimeSeconds: 120
|
||||
{% endif %}
|
||||
serviceSelector:
|
||||
matchExpressions:
|
||||
- {key: somekey, operator: NotIn, values: ['never-used-value']}
|
||||
name: cilium-peer
|
||||
spec:
|
||||
# Matches the timers and multihop used by the previous v2alpha1 policy.
|
||||
ebgpMultihop: 10
|
||||
timers:
|
||||
connectRetryTimeSeconds: 120
|
||||
holdTimeSeconds: 90
|
||||
keepAliveTimeSeconds: 30
|
||||
gracefulRestart:
|
||||
enabled: true
|
||||
restartTimeSeconds: 120
|
||||
families:
|
||||
- afi: ipv4
|
||||
safi: unicast
|
||||
advertisements:
|
||||
matchLabels:
|
||||
advertise: "bgp"
|
||||
---
|
||||
apiVersion: "cilium.io/v2alpha1"
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumBGPClusterConfig
|
||||
metadata:
|
||||
name: cilium-bgp
|
||||
spec:
|
||||
# Explicitly select every node so the BGP instance runs across the cluster.
|
||||
nodeSelector:
|
||||
matchExpressions:
|
||||
- key: somekey
|
||||
operator: NotIn
|
||||
values: ['never-used-value']
|
||||
bgpInstances:
|
||||
- name: "instance-{{ cilium_bgp_my_asn }}"
|
||||
localASN: {{ cilium_bgp_my_asn }}
|
||||
peers:
|
||||
{% for peer in _cilium_peers %}
|
||||
- name: "peer-{{ peer.peer_asn }}-{{ loop.index }}"
|
||||
peerASN: {{ peer.peer_asn }}
|
||||
peerAddress: {{ peer.peer_address }}
|
||||
peerConfigRef:
|
||||
name: cilium-peer
|
||||
{% endfor %}
|
||||
{% if cilium_exportPodCIDR %}
|
||||
---
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumBGPAdvertisement
|
||||
metadata:
|
||||
name: cilium-pod-cidrs
|
||||
labels:
|
||||
advertise: "bgp"
|
||||
spec:
|
||||
advertisements:
|
||||
- advertisementType: "PodCIDR"
|
||||
{% endif %}
|
||||
---
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumBGPAdvertisement
|
||||
metadata:
|
||||
name: cilium-lb-services
|
||||
labels:
|
||||
advertise: "bgp"
|
||||
spec:
|
||||
advertisements:
|
||||
- advertisementType: "Service"
|
||||
service:
|
||||
addresses:
|
||||
- LoadBalancerIP
|
||||
# Advertise all Services carrying an ingress address from the pool.
|
||||
selector:
|
||||
matchExpressions:
|
||||
- key: somekey
|
||||
operator: NotIn
|
||||
values: ['never-used-value']
|
||||
---
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumLoadBalancerIPPool
|
||||
metadata:
|
||||
name: "01-lb-pool"
|
||||
|
||||
Reference in New Issue
Block a user