- change the Test metallb-system namespace task to run
k3s kubectl get namespace metallb-system instead of the bare
-n metallb-system, which printed kubectl usage and always exited 0
- add a regression test that asserts the task uses an explicit get
and fails if it ever regresses to the usage-only form
- wire the new test into pre-commit
* feat(kube-vip): add endpoint override for the internal listening address
Add a kube_vip_endpoint variable so the address kube-vip binds and listens on
can differ from the announced apiserver_endpoint. This is useful for complex
routing and site-to-site tunnels where the VIP kube-vip advertises over ARP
differs from the address it listens on internally.
- roles/k3s_server/templates/vip.yaml.j2: use kube_vip_endpoint (defaulting to
apiserver_endpoint) for the `address` env and for deriving `vip_subnet`
- roles/k3s_server/defaults/main.yml: add kube_vip_endpoint default (null)
- roles/k3s_server/meta/main.yml: add kube_vip_endpoint argument_spec
- inventory/sample/group_vars/all.yml: document the new sample variable
- README.md: document the kube_vip_endpoint option
- .github/scripts/test-kube-vip-manifest.py: extend regression test to cover the
default (apiserver_endpoint) and the override case
Closes#221
* chore(ci): extend molecule job timeout to 3 hours
The default scenario occasionally takes longer than 150 minutes on the shared
nested-virt runner (k3s agent notify-wait can exceed the limit under load), and
a single timeout aborts the whole run before the other four scenarios execute.
Raise timeout-minutes from 150 to 180 so a slow-but-progressing run completes
instead of aborting.
The default scenario remains first in the matrix so a failure surfaces fastest.
* fix(kube-vip): fall back on null kube_vip_endpoint and cover it in the test
- vip_subnet and address use default(apiserver_endpoint, true) so the null
role default falls back to the apiserver endpoint instead of rendering an
empty/invalid address and subnet
- change the manifest regression test default case to pass kube_vip_endpoint
as None so it pins the real runtime null condition and fails fast on this
regression rather than timing out in CI
k3s recommends swap be disabled on every node. Add a disable_swap toggle
(default true) to the prereq role that turns swap off now and comments out the
swap entries in /etc/fstab so swap stays off across reboots. The change is
applied uniformly across all k3s_cluster hosts (all-or-nothing) since leaving
swap on for some nodes but not others creates uneven scheduling and latency.
- roles/prereq/defaults/main.yml: add disable_swap: true default
- roles/prereq/tasks/main.yml: idempotent swapoff -a and fstab comment-out block
gated on disable_swap
- inventory/sample/group_vars/all.yml: document the new sample variable
- README.md: document the disable_swap option
- .github/scripts/test-disable-swap.sh: regression test
- .pre-commit-config.yaml: wire the test into pre-commit
Closes#670
- roles/k3s_server_post/templates/metallb.crs.j2: render spec.interfaces in the
L2Advertisement when metal_lb_interfaces is a non-empty list, so MetalLB only
announces on the configured interfaces. Empty list (default) keeps announcing
on all interfaces, preserving existing behavior.
- roles/k3s_server_post/defaults/main.yml: add metal_lb_interfaces: [] default
- roles/k3s_server_post/meta/main.yml: add metal_lb_interfaces argument_spec
- inventory/sample/group_vars/all.yml: document the new sample variable
- .github/scripts/test-metallb-interfaces.py: regression test rendering the
template for empty/single/multiple interfaces and confirming the BGP path is
unaffected
- .pre-commit-config.yaml: wire the new test into pre-commit
Co-authored-by: Leo <leo@kuboschek.me>
Add a `cilium_envoy` variable (default true, matching upstream Cilium 1.20
which installs Envoy by default) that controls whether the Envoy proxy is
deployed for Cilium L7 policies. Pass it to Helm as `envoy.enabled` so users
with no L7 policies can skip Envoy to save resources.
- roles/k3s_server_post/defaults/main.yml: add cilium_envoy: true default
- roles/k3s_server_post/tasks/cilium.yml: add --helm-set envoy.enabled to the
install/upgrade command, driven by the cilium_envoy conditional
- inventory/sample/group_vars/all.yml: document cilium_envoy sample var
- .github/scripts/test-cilium-envoy-toggle.py: regression test asserting the
install command carries the envoy.enabled helm-set and renders true/false
- .pre-commit-config.yaml: wire the new test into pre-commit
Co-authored-by: Léo Nonnenmacher <leo@nonnenmacher-logel.fr>
* chore(deps): apply dependency updates in one combined change
- Bump ansible-core to 2.19.11 and jmespath to 1.1.0 in requirements.in
- Regenerate the Python 3.11 pip-compile lock in requirements.txt
- Bump molecule-plugins to 23.6.0 while keeping molecule on the stable 6.x
series (avoids the molecule-plugins 26 major jump that broke vagrant module
resolution in CI)
- Bump ruamel-yaml-clib to 0.2.15
- Bump the zgosalvez/github-actions-ensure-sha-pinned-actions action to 5.0.6
(SHA-pinned) in lint.yml
* fix(server): make log_destination conditional boolean for ansible-core 2.19
- The always block's 'Save logs of k3s-init.service' task used when: log_destination
where log_destination is a path string derived from an env var
- ansible-core 2.19 rejects string-derived conditionals; evaluate the path as a
real boolean (non-empty) check so the conditional is a true boolean
- Required to keep the k3s_server role working with ansible-core 2.19.11 (the
dependency bump in this change)
* fix(verify): coerce regex_search assertions to bool for ansible-core 2.19
- ansible-core 2.19 requires assert conditionals to be boolean; regex_search
returns a string, which is now rejected
- Wrap all regex_search results used in assert.that with | bool so the calico,
cilium, metallb, and kube-vip image-tag checks produce boolean results
* fix(verify): use boolean is regex_search test instead of | bool
- | bool on a regex_search result coerces a tag string like v0.16.0 to False
in ansible-core 2.19, failing the image-tag assertions
- Use the is regex_search test which returns a real boolean without string
coercion for the calico, cilium, metallb, and kube-vip image assertions
* fix(verify): use is not none for regex_search assertions
- ansible-core 2.19 has no "is regex_search" test and rejects bool string
coercion, so use the regex_search filter with an "is not none" comparison,
which yields a real boolean for the image-tag assertions
- Applies to calico, cilium, metallb, and kube-vip image checks
* fix(metallb): retry transient apiserver resets in config tests
- The Layer 2 and BGP final configuration checks ran a kubectl get per
resource with no retry, so a transient connection refused from the kube
API could abort converge
- Mirror the download_retries/download_delay retry pattern used by the
'Wait for MetalLB resources' task so these checks survive api server
resets on slow runners
- Add shared download_retries/download_delay defaults in k3s_server and k3s_server_post roles
- Retry calico CRD and Tigera operator manifest downloads
- Retry Cilium CLI download and cilium install/upgrade command
- Retry kube-vip cloud provider and MetalLB manifest downloads
- The CI runner's resolver intermittently times out on GitHub-hosted domains
- Correct the Deploy metallb manifest/pool when condition so MetalLB is
installed whenever kube-vip does not own the VIP range and Cilium BGP
is disabled
- The previous guard (cilium_bgp is not defined or cilium_iface is not
defined) skipped MetalLB whenever cilium_iface was set, breaking the
cilium + MetalLB scenario
- Use cilium_bgp | default(false) | bool to stay safe when Cilium vars are
not in scope (#644) while still deploying MetalLB for non-BGP Cilium
- Retry the converge-side MetalLB resource wait so a transient kube API
ServiceUnavailable does not abort the converge play
- Add a regression test that evaluates both when conditions across flannel,
calico, non-BGP cilium, BGP cilium, and kube-vip scenarios
ci: skip CI for Dependabot pull requests
- Add an actor guard to the CI workflow jobs so automatic Dependabot PRs
do not consume the shared self-hosted runner
- Dependabot CI runs need maintainer approval instead of auto-running
- On RHEL 10 / Rocky Linux 10 the default cloud image does not ship the
kernel-modules-extra that contains br_netfilter, so modprobe br_netfilter
fails during prereq on RedHat family
- Detect whether the module exists and install kernel-modules-extra when absent
- No reboot is required: the module is installed for the currently running kernel
and becomes available to modprobe immediately
* fix(metallb): guard cilium_bgp variable before evaluating
- The 'Deploy metallb manifest' and 'Deploy metallb pool' conditionals evaluate
'not cilium_bgp' directly, which raises an undefined-variable error when the
k3s_server role runs without cilium_bgp in scope and no Cilium variables are set
- Guard with 'cilium_bgp is not defined' so the condition resolves cleanly when
Cilium BGP is not configured
- Fixes#644
* docs(apiserver): clarify apiserver_endpoint must be a free routable IP
- Note that apiserver_endpoint must be an unassigned, routable IP on the
network and that it is exposed by kube-vip / MetalLB
- Fixes#678
* fix(molecule): wait for MetalLB resources before asserting images
- The MetalLB image-tag assertion crashed with 'list object has no element 0'
when the controller Deployment was not yet observable at verify time
- Retry the MetalLB controller/speaker lookup until the resources appear
- Fail with a clear message if MetalLB is genuinely absent
- systemd treats % as a specifier in ExecStart, so token values containing %
fail with 'failed to resolve unit specifiers invalid slot'
- escape % to %% in the agent unit file
- escape % to %% in server_init_args so the multi-master bootstrap path
(systemd-run k3s-init) handles % tokens too
Co-authored-by: finaldoom <677609+FinalDoom@users.noreply.github.com>
* fix(calico): support split CRDs for current releases
- Download the v1_crd_projectcalico_org.yaml bundle before the operator
- Apply both files with server-side apply and force-conflicts per the
upstream upgrade procedure
- Wait for the operator Deployment and for the managed CRDs to be
Established after the operator starts
- Replace the create/rescue/replace flow with an idempotent apply that
no longer conceals partial failures
- Verify TigeraStatus for calico and apiserver is Available, not just
that Pods exist
* feat(dependencies): upgrade supported cluster components
- Bump K3s to v1.36.2+k3s1, Calico to v3.32.1, Cilium to v1.20.0,
kube-vip to v1.2.2, kube-vip cloud provider to v0.0.12, and MetalLB to
v0.16.0 across sample inventory, role defaults, and argument specs
- Pin the Cilium CLI with a new cilium_cli_tag (v0.19.7) instead of the
floating stable.txt lookup
- Replace the CiliumBGPPeeringPolicy v2alpha1 BGP template with the
v2 CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement,
and CiliumLoadBalancerIPPool resource set
- Move Cilium load balancer Helm keys from bpf.loadBalancer to the valid
top-level loadBalancer path
- Add preflight schema validation and remove the deprecated policy after
the v2 objects are accepted
- Wait for cilium status after installation
- Pin kube-vip RBAC in a repository template instead of fetching a
mutable URL, and include EndpointSlice permissions
- Fix the kube-vip bgppeers format to address:ASN comma-separated peers
- Fail clearly when the MetalLB speaker tag replacement does not apply
- Drop the obsolete MetalLB webhook service name version branch
* test(molecule): verify upgraded cluster components
- Assert every node reports the expected K3s kubelet version
- Verify the active CNI (Flannel / Calico / Cilium) is Ready and runs
the expected image tag, including Calico TigeraStatus Available
- Verify the active load balancer (MetalLB / kube-vip) runs the expected
image tags and that MetalLB is absent when kube-vip is active
- Assert no Flannel DaemonSet remains when Calico or Cilium is enabled
- Assert the example LoadBalancer address falls inside the configured
pool range
- Add a manifest-only Cilium BGP regression test that renders the v2
template with zero, one, and multiple neighbors and rejects any v2alpha1
or CiliumBGPPeeringPolicy output
* fix(dependencies): correct dependency version pins
- Set the sample kube-vip image to v1.2.2 and repair the damaged comment
- Pin the kube-vip cloud provider default to v0.0.12 in the task URL
- Set the MetalLB controller argument-spec default to v0.16.0
- Restore the MetalLB available timeout default to 240s
* docs(dependencies): document current cluster versions
- Update kube-vip, kube-vip cloud provider, and MetalLB defaults
- Add cilium_tag and cilium_cli_tag rows
- Explain that MetalLB v0.16.0 is the application image target even though
a newer chart-only tag (metallb-chart-0.16.1) exists
- Add an existing-cluster upgrade warning covering the K3s etcd 3.5.26
bridge and one-minor-at-a-time rule, consecutive Cilium minor upgrades,
Calico v3 resource UID handling, and MetalLB app vs chart tags
* fix(dependencies): address PR review findings
- Read the MetalLB speaker tag check from the managed host with slurp
instead of a controller-side file lookup, and match the full image
reference
- Restore the tigera-operator namespace on the Calico operator Deployment
wait while keeping the managed CRD waits cluster-scoped
- Make Molecule verify inputs durable and scenario-specific via a
per-scenario verify-vars.yml, driven by explicit verify_cni/verify_lb
values instead of non-persisted converge facts
- Rename the kube-vip multi-peer BGP env var from bgppeers to bgp_peers
and vip_cidr to vip_subnet so v1.2.2 actually reads them
- Map the legacy Cilium routed mode to tunnel and stop passing the alias
directly to the chart
- Use return-code based failed_when on apply and preflight commands so
non-error failures are no longer treated as success
- Clarify the sequential K3s upgrade path and backups in the README
- Add kube-vip and MetalLB regression tests and a Cilium mode mapping unit
* fix(dependencies): resolve re-review findings
- correct the Calico TigeraStatus resource kind\n- document tunnel as the supported Cilium routing mode\n- validate load balancer addresses across range and CIDR pools
* fix(molecule): verify embedded flannel instead of a flannel DaemonSet
- K3s 1.36 runs flannel embedded in the k3s agent rather than as a
kube-flannel-ds DaemonSet, so the flannel verifier queried a workload
that no longer exists and failed the verify step
- For the flannel scenarios, assert every node is Ready and that neither
the Calico nor the Cilium namespace exists
- Drop the now-invalid kube-flannel-ds DaemonSet assertion
* fix(molecule): wait for the LoadBalancer address before asserting reachability
- The nginx LoadBalancer service had no ingress address when the
reachability assertion ran, so status.loadBalancer.ingress[0].ip was
undefined and the ipwrap filter failed during verify
- Poll the service until MetalLB or kube-vip assigns an external IP
- Record the assigned address once and reuse it for the reachability probe
and the pool membership checks
* fix(ci): harden calico apiserver wait and extend molecule job timeout
- Bump calico system resources wait retries 30->60 and delay 7->10 so the
slow-to-reconcile calico-apiserver deployment has enough time under nested-virt
- Raise the molecule step timeout-minutes from 90 to 150 to accommodate
contended 5-node scenarios (cilium, kube-vip) that were hitting the 90-min cap
* fix(calico): treat optional API server as best-effort on converge
- The Calico API server (calico-apiserver) is an optional add-on for managing
Calico policy through the projectcalico.org/v3 Kubernetes API; it is not
required for Calico CNI data plane operation
- With Calico v3.32.1 on K3s 1.36 the tigera-operator never provisions the
calico-apiserver namespace, causing the converge wait to fail deterministically
- Keep the strict wait for core Calico components (typha, kube-controllers,
calico-node, csi-node-driver) and make the API server wait tolerate failure
- Restrict the TigeraStatus Available check to the calico status, matching the
upstream v3.32.1 K3s quickstart which validates without the API server
- delegate cgroups for transient K3s server units\n- verify inventory node registration without legacy role labels\n- wait for bootstrap CRDs before replacing the transient service
- serialize five-node Vagrant creation to avoid VirtualBox host-only races
- refresh and verify the primary neighbor mapping before convergence
- include interface MAC details in failure diagnostics
- pin kube-vip and cluster traffic to the private guest interface\n- disable disposable guest firewalls and verify API reachability before joins\n- keep control-plane orchestration on the primary and preserve failure diagnostics
- Preserve explicit per-host server initialization overrides\n- Build default join arguments from the delegated host variables\n- Keep initialization commands out of normal task output
- Retry transient k3s release downloads with bounded backoff.\n- Bound failure diagnostics and validate guest release connectivity.\n- Discover repository-owned Molecule state under the actual project root.
* docs: first modules' variable docs table
* docs: variables for k3s_server_post
* docs: lxc and prereq vars in README
* style: lint errors
* docs: argument_specs for proxmox_lxc
* docs: last variables found added to the README
With the kube_vip_bgp_peers it is possible to define
multiple BGP peer ASN & address pairs for kube-vip.
Sample:
```
kube_vip_bgp_peers:
- peer_address: 192.168.128.10
peer_asn: 64512
- peer_address: 192.168.128.11
peer_asn: 64512
- peer_address: 192.168.128.12
peer_asn: 64512
```
It is possible to merge further lists with kube_vip_bgp_peers__*
parameters.
Sample:
```
kube_vip_bgp_peers__extra:
- peer_address: 192.168.128.10
peer_asn: 64512
kube_vip_bgp_peers:
- peer_address: 192.168.128.11
peer_asn: 64512
- peer_address: 192.168.128.12
peer_asn: 64512
```
This will result in the following list of BGP peer ASN & address pairs:
```
- peer_address: 192.168.128.10
peer_asn: 64512
- peer_address: 192.168.128.11
peer_asn: 64512
- peer_address: 192.168.128.12
peer_asn: 64512
```
Signed-off-by: Christian Berendt <berendt@osism.tech>
Co-authored-by: Techno Tim <timothystewart6@gmail.com>
With the cilium_bgp_neighbors parameter it is possible to define
multiple BGP peer ASN & address pairs for Cilium.
Sample:
```
cilium_bgp_neighbors:
- peer_address: 192.168.128.10
peer_asn: 64512
- peer_address: 192.168.128.11
peer_asn: 64512
- peer_address: 192.168.128.12
peer_asn: 64512
```
It is possible to merge further lists with cilium_bgp_neighbors__*
parameters.
Sample:
```
cilium_bgp_neighbors__extra:
- peer_address: 192.168.128.10
peer_asn: 64512
cilium_bgp_neighbors:
- peer_address: 192.168.128.11
peer_asn: 64512
- peer_address: 192.168.128.12
peer_asn: 64512
```
This will result in the following list of BGP peer ASN & address pairs:
```
- peer_address: 192.168.128.10
peer_asn: 64512
- peer_address: 192.168.128.11
peer_asn: 64512
- peer_address: 192.168.128.12
peer_asn: 64512
```
Signed-off-by: Christian Berendt <berendt@osism.tech>
* Change to FQCN with ansible-lint fixer
Since ansible-base 2.10 (later ansible-core), FQCN is the new way to go.
Updated .ansible-lint with a production profile and removed fqcn in skip_list.
Updated .yamllint with rules needed.
Ran ansible-lint --fix=all, then manually applied some minor changes.
* Changed octal value in molecule/ipv6/prepare.yml
The commit 3a20500f9c has introduced
argument specs in the role meta information. These two parameters
were still missing there.
Realted to 2d0596209e
Signed-off-by: Christian Berendt <berendt@osism.tech>
Co-authored-by: Techno Tim <timothystewart6@gmail.com>
With the kube_vip_bgp parameter it is possible to enable the kube-vip
BGP support (https://kube-vip.io/docs/modes/bgp/).
The configuration is possible with the following new parameters:
* kube_vip_bgp_routerid
* kube_vip_bgp_as
* kube_vip_bgp_peeraddress
* kube_vip_bgp_peeras
Signed-off-by: Christian Berendt <berendt@osism.tech>
If k3s_create_kubectl_symlink is set to false the kubectl symlink will
not be created.
If k3s_create_crictl_symlink is set to false the crictl symlink will not
be created.
By default the symlinks will be created. The default behavior is not
changed.
Signed-off-by: Christian Berendt <berendt@osism.tech>
Co-authored-by: Techno Tim <timothystewart6@gmail.com>
* ➕ Add default values to roles
* 🚚 Move to use meta files for roles
* 🛠 Fix descriptions
* ➕ Add meta for server
* 🚧 WIP
* 🌟 Complete
* 🧹 Ran and fix lint errors
* 🔨 Fix required and default conflict
---------
Co-authored-by: Techno Tim <timothystewart6@gmail.com>
With the kube_vip_arp parameter it is possible to set or unset the
vip_arp environment variable of the kube-vip-ds daemonset. The value of
the kube_vip_arp is true by default to not change the existing default.
Signed-off-by: Christian Berendt <berendt@osism.tech>
Co-authored-by: Techno Tim <timothystewart6@gmail.com>
* Fix cgroups cmdline path
* Add check and variable so that we write to the correct file
* Add LSB release checks. This is untested atm.
* Break test into multiple lines so that we can pass lint checks
* Flip logic on its head and check for existence only rather than content per contributor suggestion
---------
Co-authored-by: Techno Tim <timothystewart6@gmail.com>
* Add conditional snapshotter for PXE-booted systems
**Added:**
- PXE Boot Check - Introduced tasks to check if the system is PXE-booted by
analyzing `/proc/cmdline` in `roles/k3s_agent/tasks/main.yml`.
- Conditional Snapshotter in Template - Added logic in `k3s.service.j2` template
to conditionally set `--snapshotter native` for PXE-booted systems.
**Changed:**
- `k3s.service.j2` Template Update - Modified the `ExecStart` line to include a
conditional check for `is_pxe_booted` fact, dynamically setting the
`--snapshotter` option for NFS-mounted systems.
- `main.yml` Task Modification - Added tasks to set `is_pxe_booted` fact based
on the presence of `root=/dev/nfs` in the system's boot command line.
This update allows k3s agents on PXE-booted systems to use the native snapshotter
when running on NFS, addressing compatibility issues with OverlayFS.
* Introduce idiomatic practices for affected areas from previous commits
**Added:**
- Structured HTTP Proxy Configuration Block - Added a structured block in
`http_proxy.yml` for managing HTTP proxy settings, aligning with Ansible's
recommended practices. This includes creating directories and deploying
configuration files in a clear, modular fashion.
- Conditional Execution for Proxy Setup - Implemented conditional execution
for the proxy setup in `http_proxy.yml`, utilizing `proxy_env` to adhere
to Ansible's best practices for conditional tasks.
- Improved PXE-Boot System Check Block - Introduced a more structured approach
in `main.yml` for checking PXE-booted systems, enhancing readability and
maintainability.
**Changed:**
- Adopted Ansible Builtin Modules - Transitioned existing tasks to use
`ansible.builtin` modules, ensuring compatibility and future-proofing the
role.
- Refined Task Grouping - Reorganized tasks into logical blocks, improving
the overall structure and readability, and showcasing Ansible's capabilities
for efficient task management.
- Updated K3s Service Configuration - Modified the K3s service configuration
task in `main.yml` for a more streamlined approach using Ansible's template
module, reflecting community-driven best practices.
**Removed:**
- Streamlined Task Definitions - Optimized task definitions to reduce
redundancy, focusing on clarity and adherence to the evolving Ansible
community standards.
* Added missing checks causing failures for agents
* Add Tigera Operator/Calico CNI option
Small tweak to reduce delta from head
Set calico option to be disabled by default
Add rescue blocks in case updating existing
Refactor items and update comments
Refactor and consolidate calico.yml into block
Refactor to use template for Calico CRs
Revert use_calico to false
Template blockSize
Align default cidr in template with all.yml sample
Apply upstream version tags
Revert to current ver tags. Upstream's don't work.
Update template address detection
Add Tigera Operator/Calico CNI option
* Add calico-apiserver check
* Add eBPF dataplane option
* Add kube svc endpoint configmap when ebpf enabled
* Add /etc/cni/net.d to reset task
* Refactor based on comments
* Add molecule scenario
* Fix lint
---------
Co-authored-by: Techno Tim <timothystewart6@gmail.com>
* feat(k3s): added support for latest raspberrypi os (debian 12 bookworm)
* Update test.yml
* Revert test workflow
---------
Co-authored-by: Techno Tim <timothystewart6@gmail.com>
* Fix http_proxy service dir in k3s_agent role
* Fix http_proxy reset: rm conf files before dirs
* Fix http_proxy reset rm order
---------
Co-authored-by: Techno Tim <timothystewart6@gmail.com>
Timothy Stewart
4t1l4
Christian Berendt
Vincent Link
Richard Holmboe
Dov Benyomin Sohacheski
Rina Y
Phil Bolduc
Daniel SP
Jayson Grace
Devin Buhl
sholdee
Gereon Vey
egandro
João Gonçalves
Balázs Hasprai