Compare commits

15 Commits

Author SHA1 Message Date
Timothy Stewart e0ac53dba3 fix(metallb): verify the metallb-system namespace actually exists
- change the Test metallb-system namespace task to run
  k3s kubectl get namespace metallb-system instead of the bare
  -n metallb-system, which printed kubectl usage and always exited 0
- add a regression test that asserts the task uses an explicit get
  and fails if it ever regresses to the usage-only form
- wire the new test into pre-commit
2026-08-06 09:08:11 -05:00
Techno Tim 287d8b7a27 feat(kube-vip): add endpoint override for the internal listening address (#699)
* feat(kube-vip): add endpoint override for the internal listening address

Add a kube_vip_endpoint variable so the address kube-vip binds and listens on
can differ from the announced apiserver_endpoint. This is useful for complex
routing and site-to-site tunnels where the VIP kube-vip advertises over ARP
differs from the address it listens on internally.

- roles/k3s_server/templates/vip.yaml.j2: use kube_vip_endpoint (defaulting to
  apiserver_endpoint) for the `address` env and for deriving `vip_subnet`
- roles/k3s_server/defaults/main.yml: add kube_vip_endpoint default (null)
- roles/k3s_server/meta/main.yml: add kube_vip_endpoint argument_spec
- inventory/sample/group_vars/all.yml: document the new sample variable
- README.md: document the kube_vip_endpoint option
- .github/scripts/test-kube-vip-manifest.py: extend regression test to cover the
  default (apiserver_endpoint) and the override case

Closes #221

* chore(ci): extend molecule job timeout to 3 hours

The default scenario occasionally takes longer than 150 minutes on the shared
nested-virt runner (k3s agent notify-wait can exceed the limit under load), and
a single timeout aborts the whole run before the other four scenarios execute.
Raise timeout-minutes from 150 to 180 so a slow-but-progressing run completes
instead of aborting.

The default scenario remains first in the matrix so a failure surfaces fastest.

* fix(kube-vip): fall back on null kube_vip_endpoint and cover it in the test

- vip_subnet and address use default(apiserver_endpoint, true) so the null
  role default falls back to the apiserver endpoint instead of rendering an
  empty/invalid address and subnet
- change the manifest regression test default case to pass kube_vip_endpoint
  as None so it pins the real runtime null condition and fails fast on this
  regression rather than timing out in CI
2026-08-06 02:48:57 -05:00
Techno Tim 56bb912bd3 feat(prereq): add toggle to disable swap on all cluster nodes (#698)
k3s recommends swap be disabled on every node. Add a disable_swap toggle
(default true) to the prereq role that turns swap off now and comments out the
swap entries in /etc/fstab so swap stays off across reboots. The change is
applied uniformly across all k3s_cluster hosts (all-or-nothing) since leaving
swap on for some nodes but not others creates uneven scheduling and latency.

- roles/prereq/defaults/main.yml: add disable_swap: true default
- roles/prereq/tasks/main.yml: idempotent swapoff -a and fstab comment-out block
  gated on disable_swap
- inventory/sample/group_vars/all.yml: document the new sample variable
- README.md: document the disable_swap option
- .github/scripts/test-disable-swap.sh: regression test
- .pre-commit-config.yaml: wire the test into pre-commit

Closes #670
2026-08-05 07:39:00 -05:00
Techno Tim cf76292169 feat(site): fail fast when cluster nodes share a hostname (#697)
Add a preflight assertion to site.yml that confirms every host in the
k3s_cluster group reports a unique hostname. k3s registers each node keyed by
its hostname, so duplicate hostnames prevent nodes from joining the cluster
and are hard to troubleshoot. The check fails fast in the Pre tasks instead of
surfacing as a cryptic registration failure later.

- site.yml: assert (cluster_hostnames | unique | length) == length for the
  k3s_cluster group, skipping hosts outside that group
- README.md: document the unique-hostname requirement
- .github/scripts/test-unique-hostname-precheck.sh: regression test
- .pre-commit-config.yaml: wire the test into pre-commit

Closes #636
2026-08-05 05:14:29 -05:00
Techno Tim 52c086d638 feat(metallb): add option to limit MetalLB layer2 announcements to specific interfaces (#696)
- roles/k3s_server_post/templates/metallb.crs.j2: render spec.interfaces in the
  L2Advertisement when metal_lb_interfaces is a non-empty list, so MetalLB only
  announces on the configured interfaces. Empty list (default) keeps announcing
  on all interfaces, preserving existing behavior.
- roles/k3s_server_post/defaults/main.yml: add metal_lb_interfaces: [] default
- roles/k3s_server_post/meta/main.yml: add metal_lb_interfaces argument_spec
- inventory/sample/group_vars/all.yml: document the new sample variable
- .github/scripts/test-metallb-interfaces.py: regression test rendering the
  template for empty/single/multiple interfaces and confirming the BGP path is
  unaffected
- .pre-commit-config.yaml: wire the new test into pre-commit

Co-authored-by: Leo <leo@kuboschek.me>
2026-08-05 02:48:24 -05:00
Techno Tim 010551b8d2 feat(cilium): add toggle to enable or disable the Envoy proxy (#695)
Add a `cilium_envoy` variable (default true, matching upstream Cilium 1.20
which installs Envoy by default) that controls whether the Envoy proxy is
deployed for Cilium L7 policies. Pass it to Helm as `envoy.enabled` so users
with no L7 policies can skip Envoy to save resources.

- roles/k3s_server_post/defaults/main.yml: add cilium_envoy: true default
- roles/k3s_server_post/tasks/cilium.yml: add --helm-set envoy.enabled to the
  install/upgrade command, driven by the cilium_envoy conditional
- inventory/sample/group_vars/all.yml: document cilium_envoy sample var
- .github/scripts/test-cilium-envoy-toggle.py: regression test asserting the
  install command carries the envoy.enabled helm-set and renders true/false
- .pre-commit-config.yaml: wire the new test into pre-commit

Co-authored-by: Léo Nonnenmacher <leo@nonnenmacher-logel.fr>
2026-08-05 00:00:39 -05:00
Techno Tim c82f2e0415 chore(deps): apply dependency updates in one combined change (#694)
* chore(deps): apply dependency updates in one combined change

- Bump ansible-core to 2.19.11 and jmespath to 1.1.0 in requirements.in
- Regenerate the Python 3.11 pip-compile lock in requirements.txt
- Bump molecule-plugins to 23.6.0 while keeping molecule on the stable 6.x
  series (avoids the molecule-plugins 26 major jump that broke vagrant module
  resolution in CI)
- Bump ruamel-yaml-clib to 0.2.15
- Bump the zgosalvez/github-actions-ensure-sha-pinned-actions action to 5.0.6
  (SHA-pinned) in lint.yml

* fix(server): make log_destination conditional boolean for ansible-core 2.19

- The always block's 'Save logs of k3s-init.service' task used when: log_destination
  where log_destination is a path string derived from an env var
- ansible-core 2.19 rejects string-derived conditionals; evaluate the path as a
  real boolean (non-empty) check so the conditional is a true boolean
- Required to keep the k3s_server role working with ansible-core 2.19.11 (the
  dependency bump in this change)

* fix(verify): coerce regex_search assertions to bool for ansible-core 2.19

- ansible-core 2.19 requires assert conditionals to be boolean; regex_search
  returns a string, which is now rejected
- Wrap all regex_search results used in assert.that with | bool so the calico,
  cilium, metallb, and kube-vip image-tag checks produce boolean results

* fix(verify): use boolean is regex_search test instead of | bool

- | bool on a regex_search result coerces a tag string like v0.16.0 to False
  in ansible-core 2.19, failing the image-tag assertions
- Use the is regex_search test which returns a real boolean without string
  coercion for the calico, cilium, metallb, and kube-vip image assertions

* fix(verify): use is not none for regex_search assertions

- ansible-core 2.19 has no "is regex_search" test and rejects bool string
  coercion, so use the regex_search filter with an "is not none" comparison,
  which yields a real boolean for the image-tag assertions
- Applies to calico, cilium, metallb, and kube-vip image checks

* fix(metallb): retry transient apiserver resets in config tests

- The Layer 2 and BGP final configuration checks ran a kubectl get per
  resource with no retry, so a transient connection refused from the kube
  API could abort converge
- Mirror the download_retries/download_delay retry pattern used by the
  'Wait for MetalLB resources' task so these checks survive api server
  resets on slow runners
2026-08-04 17:54:58 -05:00
Techno Tim ac1e3288c0 feat(reboot): support staggered reboots of cluster nodes (#693)
- Add serial to the reboot play so concurrent_reboots controls how many nodes
  reboot at a time, defaulting to 100% (reboot all at once) for backward
  compatibility
- Add a per-node test_command that verifies the node is healthy after reboot
- Add an optional wait_seconds_after_reboot pause between staggered batches so
  pods can settle before the next batch reboots
- Document the new concurrent_reboots and wait_seconds_after_reboot variables

Co-authored-by: Felix Seifert <mail@felix-seifert.com>
2026-08-04 06:11:33 -05:00
Techno Tim db30128468 fix(ci): retry transient DNS failures on remote downloads and cilium install (#690)
- Add shared download_retries/download_delay defaults in k3s_server and k3s_server_post roles
- Retry calico CRD and Tigera operator manifest downloads
- Retry Cilium CLI download and cilium install/upgrade command
- Retry kube-vip cloud provider and MetalLB manifest downloads
- The CI runner's resolver intermittently times out on GitHub-hosted domains
2026-08-04 08:52:16 +00:00
Techno Tim 249238c7a4 fix(metallb): deploy MetalLB with a non-BGP Cilium CNI (#692)
- Correct the Deploy metallb manifest/pool when condition so MetalLB is
  installed whenever kube-vip does not own the VIP range and Cilium BGP
  is disabled
- The previous guard (cilium_bgp is not defined or cilium_iface is not
  defined) skipped MetalLB whenever cilium_iface was set, breaking the
  cilium + MetalLB scenario
- Use cilium_bgp | default(false) | bool to stay safe when Cilium vars are
  not in scope (#644) while still deploying MetalLB for non-BGP Cilium
- Retry the converge-side MetalLB resource wait so a transient kube API
  ServiceUnavailable does not abort the converge play
- Add a regression test that evaluates both when conditions across flannel,
  calico, non-BGP cilium, BGP cilium, and kube-vip scenarios

ci: skip CI for Dependabot pull requests

- Add an actor guard to the CI workflow jobs so automatic Dependabot PRs
  do not consume the shared self-hosted runner
- Dependabot CI runs need maintainer approval instead of auto-running
2026-08-04 01:45:05 -05:00
Techno Tim f5483cdabe fix(flannel): default the interface to each host's default IPv4 interface (#689)
* fix(flannel): default the interface to each host's default IPv4 interface

- Replace the hardcoded flannel_iface: eth0 with a per-host default derived
  from ansible_facts.default_ipv4.interface
- KVM/cloud hosts that are not named eth0 (e.g. enp1s0, ens3) now resolve the
  interface automatically instead of failing the k3s_node_ip lookup
- Update the commented calico_iface / cilium_iface examples to match
- Co-authored-by: Fritz Dunkel <677609+FinalDoom@users.noreply.github.com>
- Fixes #621

* test(flannel): add regression test for per-host interface default

- Add a focused test that renders the sample inventory's flannel_iface
  expression against fake ansible facts
- Assert a non-eth0 host (enp1s0, ens3) resolves its own interface and that
  the expression defaults from ansible_facts.default_ipv4.interface
- Wire it as a local pre-commit hook (default-interface-test)

* test(molecule): assert nodes register a non-loopback InternalIP

- Add a flannel-scenario verify assertion that every node reports an
  InternalIP derived from its configured flannel_iface
- Guards against k3s binding to 127.0.0.1 instead of the cluster interface
- Complements the pre-commit default-interface test with an end-to-end check
2026-08-03 17:01:50 +00:00
Techno Tim 88159b3875 fix(prereq): install kernel-modules-extra when br_netfilter is missing (#684)
- On RHEL 10 / Rocky Linux 10 the default cloud image does not ship the
  kernel-modules-extra that contains br_netfilter, so modprobe br_netfilter
  fails during prereq on RedHat family
- Detect whether the module exists and install kernel-modules-extra when absent
- No reboot is required: the module is installed for the currently running kernel
  and becomes available to modprobe immediately
2026-08-03 14:55:07 +00:00
Techno Tim db85fa960c fix(metallb): guard cilium_bgp when undefined; clarify apiserver_endpoint docs (#683)
* fix(metallb): guard cilium_bgp variable before evaluating

- The 'Deploy metallb manifest' and 'Deploy metallb pool' conditionals evaluate
  'not cilium_bgp' directly, which raises an undefined-variable error when the
  k3s_server role runs without cilium_bgp in scope and no Cilium variables are set
- Guard with 'cilium_bgp is not defined' so the condition resolves cleanly when
  Cilium BGP is not configured
- Fixes #644

* docs(apiserver): clarify apiserver_endpoint must be a free routable IP

- Note that apiserver_endpoint must be an unassigned, routable IP on the
  network and that it is exposed by kube-vip / MetalLB
- Fixes #678

* fix(molecule): wait for MetalLB resources before asserting images

- The MetalLB image-tag assertion crashed with 'list object has no element 0'
  when the controller Deployment was not yet observable at verify time
- Retry the MetalLB controller/speaker lookup until the resources appear
- Fail with a clear message if MetalLB is genuinely absent
2026-08-03 04:52:27 +00:00
Techno Tim 6aea6e71b6 docs(agent): improve repository guidance [skip ci] (#685)
- Add canonical agent and contributor documentation.

- Modernize issue and pull request templates.

- Skip CI for documentation and issue-template-only changes.
2026-08-02 16:31:40 -05:00
Techno Tim 890d43b339 fix(token): escape percent signs used in k3s token (#682)
- systemd treats % as a specifier in ExecStart, so token values containing %
  fail with 'failed to resolve unit specifiers invalid slot'
- escape % to %% in the agent unit file
- escape % to %% in server_init_args so the multi-master bootstrap path
  (systemd-run k3s-init) handles % tokens too

Co-authored-by: finaldoom <677609+FinalDoom@users.noreply.github.com>
2026-08-02 19:27:17 +00:00
44 changed files with 1478 additions and 148 deletions
-85
View File
@@ -1,85 +0,0 @@
<!-- It's a good idea to check this post first for general troubleshooting https://github.com/timothystewart6/k3s-ansible/discussions/19 -->
<!--- Provide a general summary of the issue in the Title above -->
## Expected Behavior
<!--- Tell us what should happen -->
## Current Behavior
<!--- Tell us what happens instead of the expected behavior -->
## Steps to Reproduce
<!--- reproduce this bug. Include code to reproduce, if relevant -->
1.
2.
3.
4.
## Context (variables)
<!--- please include which OS, along with the variables used when running the playbook -->
Operating system:
Hardware:
### Variables Used
`all.yml`
```yml
k3s_version: ""
ansible_user: NA
systemd_dir: ""
flannel_iface: ""
#calico_iface: ""
calico_ebpf: ""
calico_cidr: ""
calico_tag: ""
apiserver_endpoint: ""
k3s_token: "NA"
extra_server_args: ""
extra_agent_args: ""
kube_vip_tag_version: ""
kube_vip_cloud_provider_tag_version: ""
kube_vip_lb_ip_range: ""
metal_lb_speaker_tag_version: ""
metal_lb_controller_tag_version: ""
metal_lb_ip_range: ""
```
### Hosts
`host.ini`
```ini
[master]
IP.ADDRESS.ONE
IP.ADDRESS.TWO
IP.ADDRESS.THREE
[node]
IP.ADDRESS.FOUR
IP.ADDRESS.FIVE
[k3s_cluster:children]
master
node
```
## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
- [ ] I've checked the [General Troubleshooting Guide](https://github.com/timothystewart6/k3s-ansible/discussions/20)
+102
View File
@@ -0,0 +1,102 @@
---
name: Bug report
description: Report a reproducible problem with the playbooks, roles, or generated resources.
title: "[Bug]: "
labels:
- bug
body:
- type: markdown
attributes:
value: |
Thanks for reporting a problem. Search existing issues and review the troubleshooting link first.
Remove credentials, tokens, public IP addresses, and private hostnames from all fields and logs.
- type: checkboxes
id: prerequisites
attributes:
label: Prerequisites
options:
- label: I searched existing issues and discussions for this problem.
required: true
- label: I reviewed the troubleshooting guidance linked from the issue chooser.
required: true
- label: I removed secrets and identifying infrastructure details from this report.
required: true
- type: textarea
id: summary
attributes:
label: Problem summary
description: Describe what failed and its impact.
placeholder: A concise description of the problem and affected nodes or components.
validations:
required: true
- type: textarea
id: expected
attributes:
label: Expected behavior
description: What should have happened?
validations:
required: true
- type: textarea
id: reproduction
attributes:
label: Steps to reproduce
description: Provide the smallest reliable sequence that reproduces the issue.
placeholder: |
1. Configure ...
2. Run ...
3. Observe ...
validations:
required: true
- type: input
id: revision
attributes:
label: Repository revision
description: Release, tag, branch, or commit SHA used.
placeholder: v1.36.2+k3s1+tt1 or a commit SHA
validations:
required: true
- type: input
id: ansible-version
attributes:
label: Ansible version
description: Output of `ansible --version`, shortened to version and Python details.
placeholder: ansible-core 2.18.0, Python 3.12
validations:
required: true
- type: textarea
id: environment
attributes:
label: Environment
description: Include target OS and version, architecture, node counts, platform, and network provider.
placeholder: Debian 13, amd64, 3 control nodes and 2 agents, bare metal, Cilium
validations:
required: true
- type: textarea
id: configuration
attributes:
label: Relevant sanitized configuration
description: Include only variables and inventory groups needed to reproduce the problem.
render: yaml
validations:
required: true
- type: textarea
id: logs
attributes:
label: Relevant logs or task output
description: Include the failing task and surrounding output. Redact sensitive or identifying values.
render: shell
- type: textarea
id: context
attributes:
label: Additional context
description: Add attempted fixes, suspected causes, regressions, or other useful context.
+9
View File
@@ -0,0 +1,9 @@
---
blank_issues_enabled: false
contact_links:
- name: Troubleshooting and support
url: https://github.com/timothystewart6/k3s-ansible/discussions/20
about: Review common troubleshooting guidance and ask configuration or usage questions.
- name: General discussions
url: https://github.com/timothystewart6/k3s-ansible/discussions
about: Discuss ideas and questions that are not confirmed bugs or concrete feature requests.
@@ -0,0 +1,55 @@
---
name: Feature request
description: Propose a focused improvement to supported repository behavior.
title: "[Feature]: "
labels:
- enhancement
body:
- type: markdown
attributes:
value: |
Describe the use case before proposing an implementation. Search existing issues and discussions first.
- type: checkboxes
id: prerequisites
attributes:
label: Prerequisites
options:
- label: I searched existing issues and discussions for this request.
required: true
- label: This request is about reusable project behavior, not support for one private environment.
required: true
- type: textarea
id: problem
attributes:
label: Problem or use case
description: What limitation exists, who encounters it, and why does it matter?
validations:
required: true
- type: textarea
id: proposal
attributes:
label: Proposed behavior
description: Describe the desired user-visible result. Include example variables or commands when useful.
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives considered
description: Describe workarounds or other designs and their tradeoffs.
- type: textarea
id: compatibility
attributes:
label: Compatibility and operational impact
description: Note affected operating systems, architectures, CNIs, existing clusters, or reset behavior.
- type: textarea
id: context
attributes:
label: Additional context
description: Add relevant upstream documentation, examples, or prior discussion.
+42 -11
View File
@@ -1,15 +1,46 @@
# Proposed Changes
<!--- Provide a general summary of your changes -->
## Summary
<!-- Explain the problem and the resulting behavior. Keep implementation details in the sections below. -->
## Changes
-
-
-
## Checklist
## Related issues
- [ ] Tested locally
- [ ] Ran `site.yml` playbook
- [ ] Ran `reset.yml` playbook
- [ ] Did not add any unnecessary changes
- [ ] Ran pre-commit install at least once before committing
- [ ] 🚀
<!-- Use "Fixes #123" when this pull request should close an issue. Write "None" when not applicable. -->
## Testing
<!-- List exact commands, scenarios, and relevant manual checks. Do not check a box for a test that was not run. -->
- [ ] `pre-commit run --all-files`
- [ ] Relevant Ansible syntax checks
- [ ] Relevant focused regression tests
- [ ] Relevant Molecule scenario
- [ ] Provisioning tested against a non-production cluster
- [ ] Reset behavior tested against a non-production cluster
Not run, with reason:
## Risk and compatibility
<!-- Cover existing clusters, upgrades, networking, supported platforms, security, and rollback. Write "None" when a
category is not affected. -->
- Existing cluster or upgrade impact:
- Networking or CNI impact:
- Security impact:
- Rollback plan:
## Documentation
<!-- Identify updated docs and sample configuration, or explain why no documentation change is needed. -->
## Final checklist
- [ ] The change is focused and contains no unrelated edits.
- [ ] Tests cover new behavior or a regression, where applicable.
- [ ] User-facing variables are documented in role defaults, sample inventory, and the README.
- [ ] Logs, examples, and configuration contain no secrets or identifying infrastructure details.
- [ ] Generated files and local environment files are not included.
+6
View File
@@ -0,0 +1,6 @@
# GitHub Copilot instructions
Read and follow the repository's root-level `AGENTS.md` before proposing or making changes. It is the canonical guide
for architecture, safety, implementation, validation, and documentation expectations.
Do not duplicate repository guidance here. If instructions need to change, update `AGENTS.md`.
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env python3
"""Regression test for the Cilium Envoy toggle.
The `cilium_envoy` variable lets users enable or disable the Cilium Envoy
proxy. The Install/upgrade Cilium task in
roles/k3s_server_post/tasks/cilium.yml passes the value through to Helm as
`envoy.enabled`. This test:
- loads the real "Install Cilium" task and confirms the install/upgrade
command actually contains the `envoy.enabled` Helm value,
- renders the conditional that computes the Helm value and confirms it
produces `true` when cilium_envoy is enabled and `false` when disabled,
- confirms the task stays forward/backward compatible (no raw `true` /
`false` hardcoded in place of the conditional).
"""
from __future__ import print_function
import os
import re
import subprocess
import yaml
from jinja2 import Environment
ENVOY_EXPRESSION = '{{ "true" if cilium_envoy else "false" }}'
def repo_root():
return subprocess.check_output(
["git", "rev-parse", "--show-toplevel"], text=True
).strip()
def fail(message):
raise SystemExit("Cilium Envoy toggle test failed: " + message)
def extract_install_command(path):
"""Return the command string for the 'Install Cilium' task.
Walks both top-level tasks and tasks nested inside a `block`/`always`/
`rescue` list, since the Cilium deploy steps are grouped under the
'Prepare Cilium CLI on first master and deploy CNI' block.
"""
with open(path, encoding="utf-8") as handle:
doc = yaml.safe_load(handle)
def find_command(tasks):
for task in tasks:
if not isinstance(task, dict):
continue
if task.get("name") == "Install Cilium":
command = task.get("ansible.builtin.command")
if command is None:
raise SystemExit(
"Cilium Envoy toggle test failed: "
"'Install Cilium' task has no ansible.builtin.command"
)
return command
# Recurse into block/always/rescue sub-lists.
for key in ("block", "always", "rescue"):
nested = task.get(key)
if isinstance(nested, list):
found = find_command(nested)
if found is not None:
return found
return None
command = find_command(doc)
if command is None:
raise SystemExit(
"Cilium Envoy toggle test failed: could not find 'Install Cilium' task"
)
return command
def assert_envoy_in_command(command):
if "envoy.enabled" not in command:
fail("install command is missing --helm-set envoy.enabled")
if ENVOY_EXPRESSION not in command:
fail(
"install command does not use the cilium_envoy conditional: "
"expected {0!r}".format(ENVOY_EXPRESSION)
)
# The conditional must be a WYSIWYG helm-set value, not a pre-rendered
# true/false literal (which would ignore the cilium_envoy variable).
if re.search(r"--helm-set envoy\.enabled=true(?:$|\s)", command):
fail("install command hardcodes envoy.enabled=true")
if re.search(r"--helm-set envoy\.enabled=false(?:$|\s)", command):
fail("install command hardcodes envoy.enabled=false")
def assert_render():
env = Environment()
def render_for(value):
template = env.from_string(ENVOY_EXPRESSION)
return template.render(cilium_envoy=value)
if render_for(True) != "true":
fail("envoy conditional did not render 'true' when enabled")
if render_for(False) != "false":
fail("envoy conditional did not render 'false' when disabled")
def main():
root = repo_root()
cilium_tasks = os.path.join(
root, "roles", "k3s_server_post", "tasks", "cilium.yml"
)
command = extract_install_command(cilium_tasks)
assert_envoy_in_command(command)
assert_render()
print("Cilium Envoy toggle regression test passed")
if __name__ == "__main__":
main()
+90
View File
@@ -0,0 +1,90 @@
#!/usr/bin/env python3
"""Assert the sample inventory resolves the flannel interface per host.
flannel_iface defaults to the host's default IPv4 interface rather than a
hardcoded eth0. This test extracts the flannel_iface expression from the
sample inventory and proves that a host whose primary interface is not named
eth0 (e.g. enp1s0, ens3) resolves the interface from ansible facts.
"""
from __future__ import print_function
import os
import re
import subprocess
from jinja2 import Environment, StrictUndefined
def repo_root():
return subprocess.check_output(
["git", "rev-parse", "--show-toplevel"], text=True
).strip()
def fail(message):
raise SystemExit("default-interface test failed: " + message)
class FakeAnsibleFacts(object):
"""Stand-in for the per-host ``ansible_facts`` dict."""
def __init__(self, default_iface, iface_ip):
self._default = {"interface": default_iface, "address": iface_ip}
self._ifaces = {
default_iface: {"ipv4": {"address": iface_ip}},
}
@property
def default_ipv4(self):
return self._default
def __getitem__(self, key):
return self._ifaces[key]
def read_all_yml(root):
path = os.path.join(root, "inventory", "sample", "group_vars", "all.yml")
with open(path, "r") as handle:
return handle.read()
def extract_value(content, key):
# Match a quoted value assigned to the key, e.g. flannel_iface: "...".
match = re.search(r"^%s:\s*\"(.+)\"\s*$" % re.escape(key), content, re.M)
if not match:
fail("could not find %s in the sample inventory" % key)
return match.group(1)
def resolve(env, expression, facts):
template = env.from_string(expression)
return template.render(ansible_facts=facts)
def main():
root = repo_root()
content = read_all_yml(root)
env = Environment(undefined=StrictUndefined)
flannel_expr = extract_value(content, "flannel_iface")
if "default_ipv4.interface" not in flannel_expr:
fail("flannel_iface no longer defaults from ansible facts")
# A host whose primary interface is enp1s0 (the core #621 scenario).
facts = FakeAnsibleFacts("enp1s0", "192.168.30.11")
resolved = resolve(env, flannel_expr, facts)
if resolved != "enp1s0":
fail("flannel_iface resolved to %r, expected enp1s0" % resolved)
# A different host with a different interface must resolve independently.
facts2 = FakeAnsibleFacts("ens3", "192.168.30.12")
resolved2 = resolve(env, flannel_expr, facts2)
if resolved2 != "ens3":
fail("flannel_iface resolved to %r, expected ens3" % resolved2)
print("default-interface regression test passed")
if __name__ == "__main__":
main()
+37
View File
@@ -0,0 +1,37 @@
#!/usr/bin/env bash
set -Eeuo pipefail
repo_root="$(git rev-parse --show-toplevel)"
prereq_defaults="$repo_root/roles/prereq/defaults/main.yml"
prereq_tasks="$repo_root/roles/prereq/tasks/main.yml"
# #670: k3s recommends swap be disabled on all nodes. The prereq role must expose
# a disable_swap toggle (defaulting to true) that turns swap off now and comments
# out the /etc/fstab swap entries so swap stays off across reboots.
grep -Eq -- '^disable_swap: true' "$prereq_defaults" || {
printf 'prereq defaults are missing disable_swap: true\n' >&2
exit 1
}
grep -Fq -- 'Disable swap on all cluster nodes' "$prereq_tasks" || {
printf 'prereq tasks are missing the swap-disable block\n' >&2
exit 1
}
grep -Fq -- 'swapoff -a' "$prereq_tasks" || {
printf 'swap-disable block does not run swapoff -a\n' >&2
exit 1
}
grep -Fq -- '/etc/fstab' "$prereq_tasks" || {
printf 'swap-disable block does not comment out /etc/fstab swap entries\n' >&2
exit 1
}
if ! grep -Eq -- 'when: disable_swap' "$prereq_tasks"; then
printf 'swap-disable block is not gated on the disable_swap toggle\n' >&2
exit 1
fi
printf 'Swap disable regression test passed\n'
+32
View File
@@ -123,6 +123,38 @@ def main():
if "name: bgp_peers" in output:
fail("bgp_peers present even though the peer list is empty")
# kube_vip_endpoint defaults to null (defined in role defaults): the
# address and subnet must fall back to the apiserver endpoint. default()
# without a truthy flag does NOT fall back on null, only on undefined, so
# this case pins the null runtime condition to prevent that regression.
output = render(
env,
{
"_kube_vip_bgp_peers": [],
"kube_vip_endpoint": None,
"kube_vip_arp": True,
"kube_vip_bgp": False,
},
)
if "value: 192.168.30.222" not in output:
fail("null kube_vip_endpoint does not fall back to apiserver_endpoint")
# kube_vip_endpoint set: overrides the internal listening address AND the
# subnet derivation while the advertised apiserver_endpoint stays separate.
output = render(
env,
{
"_kube_vip_bgp_peers": [],
"kube_vip_endpoint": "10.66.1.5",
"kube_vip_arp": True,
"kube_vip_bgp": False,
},
)
if "value: 10.66.1.5" not in output:
fail("kube_vip_endpoint did not override the address")
if "value: 192.168.30.222" in output:
fail("apiserver_endpoint leaked into address when kube_vip_endpoint set")
print("kube-vip manifest regression test passed")
@@ -0,0 +1,170 @@
#!/usr/bin/env python3
"""Regression test for the MetalLB deploy conditions.
The MetalLB manifest (roles/k3s_server/tasks/main.yml) and the MetalLB pool
(roles/k3s_server_post/tasks/main.yml) are included under a `when` condition
that decides whether MetalLB provides load balancing. A previous change (#683)
guarded `cilium_bgp` but accidentally skipped MetalLB whenever a non-BGP
Cilium CNI was in use (`cilium_iface` defined), breaking the cilium + MetalLB
scenario.
This test loads the real `when` expressions from both task files and evaluates
them against representative variable sets, asserting MetalLB is deployed in
every topology except when kube-vip owns the VIP range or Cilium BGP is enabled.
"""
from __future__ import print_function
import os
import re
import subprocess
import yaml
from jinja2 import Environment
METALLB_WHEN = (
"kube_vip_lb_ip_range is not defined and "
"not (cilium_bgp | default(false) | bool)"
)
def repo_root():
return subprocess.check_output(
["git", "rev-parse", "--show-toplevel"], text=True
).strip()
def fail(message):
raise SystemExit("MetalLB deploy condition test failed: " + message)
def extract_when(path, task_name):
"""Return the `when:` expression string for the named task."""
with open(path, encoding="utf-8") as handle:
doc = yaml.safe_load(handle)
for task in doc:
if task.get("name") == task_name:
when = task.get("when")
return (when or "").strip()
return None
def evaluate(when, variables):
"""Evaluate a `when` expression against variables using Jinja2."""
env = Environment()
def fake_bool(value):
# Minimal stand-in for Ansible's truthiness filter used by `| bool`.
if isinstance(value, bool):
return value
if value is None:
return False
return str(value).lower() in ("1", "true", "yes", "on")
env.filters["bool"] = fake_bool
template = env.from_string("{{ " + when + " }}")
rendered = template.render(**variables)
# The expression renders to the literal strings "True"/"False".
if rendered == "True":
return True
if rendered == "False":
return False
fail("condition did not render to a boolean: {0!r}".format(rendered))
def assert_deployment(when, variables, expected, label):
result = evaluate(when, variables)
verdict = "deploy" if result else "skip"
expected_verdict = "deploy" if expected else "skip"
if result != expected:
fail(
"{0}: expected to {1} MetalLB but the condition chose to {2} "
"(vars: {3})".format(label, expected_verdict, verdict, variables)
)
def scenarios():
"""Yield (variables, expected_deploy, label) pairs."""
yield (
# Default Flannel inventory (all.yml sets cilium_bgp: false).
{
"cilium_bgp": False,
"cilium_iface": None,
},
True,
"flannel default (cilium_bgp: false)",
)
yield (
# Calico CNI with no Cilium variable in scope (issue #644): cilium_bgp
# is genuinely undefined, so `default(false)` must keep MetalLB on.
{
"calico_iface": "eth1",
},
True,
"calico, cilium_bgp undefined (#644)",
)
yield (
# Cilium CNI with BGP disabled: MetalLB must still be deployed.
{
"cilium_bgp": False,
"cilium_iface": "eth1",
},
True,
"cilium non-BGP (regression catch)",
)
yield (
# Cilium CNI with BGP enabled: Cilium provides the LB, skip MetalLB.
{
"cilium_bgp": True,
"cilium_iface": "eth1",
},
False,
"cilium BGP enabled",
)
yield (
# kube-vip is the load balancer provider: skip MetalLB.
{
"kube_vip_lb_ip_range": "192.168.30.80-192.168.30.90",
"cilium_bgp": False,
},
False,
"kube-vip owns the VIP range",
)
def main():
root = repo_root()
server_tasks = os.path.join(root, "roles", "k3s_server", "tasks", "main.yml")
server_post_tasks = os.path.join(
root, "roles", "k3s_server_post", "tasks", "main.yml"
)
server_when = extract_when(server_tasks, "Deploy metallb manifest")
server_post_when = extract_when(server_post_tasks, "Deploy metallb pool")
if server_when is None:
fail("could not find 'Deploy metallb manifest' when condition")
if server_post_when is None:
fail("could not find 'Deploy metallb pool' when condition")
for when, source in (
(server_when, "k3s_server/tasks/main.yml"),
(server_post_when, "k3s_server_post/tasks/main.yml"),
):
if when != METALLB_WHEN:
fail(
"{0} when condition changed unexpectedly:\n"
" expected: {1}\n got: {2}".format(source, METALLB_WHEN, when)
)
for variables, expected, label in scenarios():
assert_deployment(server_when, variables, expected, "server " + label)
assert_deployment(
server_post_when, variables, expected, "server_post " + label
)
print("MetalLB deploy condition regression test passed for all scenarios")
if __name__ == "__main__":
main()
@@ -0,0 +1,88 @@
#!/usr/bin/env python3
"""Regression test for the MetalLB L2Advertisement interfaces.
`metal_lb_interfaces` restricts which network interfaces MetalLB announces
load balancer IPs on in layer2 mode. When the list is non-empty, the
L2Advertisement in roles/k3s_server_post/templates/metallb.crs.j2 must render
a `spec.interfaces` block; when it is empty (the default), no spec is rendered
so MetalLB announces on all interfaces.
This renders the template and asserts both cases plus the BGP path (which must
not be affected by the L2 interfaces variable).
"""
from __future__ import print_function
import os
import subprocess
from jinja2 import Environment, FileSystemLoader, StrictUndefined
def repo_root():
return subprocess.check_output(
["git", "rev-parse", "--show-toplevel"], text=True
).strip()
def fail(message):
raise SystemExit("MetalLB interfaces test failed: " + message)
def render(env, extra_vars):
base_vars = {
"metal_lb_mode": "layer2",
"metal_lb_ip_range": "192.168.30.80-192.168.30.90",
}
base_vars.update(extra_vars)
template = env.get_template("metallb.crs.j2")
return template.render(**base_vars)
def main():
root = repo_root()
template_dir = os.path.join(
root, "roles", "k3s_server_post", "templates"
)
env = Environment(
loader=FileSystemLoader(template_dir), undefined=StrictUndefined
)
# Empty list (default): no spec.interfaces in the L2Advertisement.
output = render(env, {"metal_lb_interfaces": []})
if "spec:\n interfaces:" in output:
fail("spec.interfaces rendered with an empty metal_lb_interfaces")
if "kind: L2Advertisement" not in output:
fail("L2Advertisement missing in layer2 mode")
# Single interface.
output = render(env, {"metal_lb_interfaces": ["eth1"]})
if "spec:\n interfaces:\n - eth1" not in output:
fail("single interface was not rendered in spec.interfaces")
# Multiple interfaces.
output = render(env, {"metal_lb_interfaces": ["eth1", "eth2"]})
if "spec:\n interfaces:\n - eth1\n - eth2" not in output:
fail("multiple interfaces were not rendered in spec.interfaces")
# BGP mode must not emit an L2Advertisement spec at all.
output = render(
env,
{
"metal_lb_mode": "bgp",
"metal_lb_interfaces": ["eth1"],
"metal_lb_bgp_my_asn": "64513",
"metal_lb_bgp_peer_asn": "64512",
"metal_lb_bgp_peer_address": "192.168.30.1",
},
)
if "kind: L2Advertisement" in output:
fail("L2Advertisement rendered in bgp mode")
if "interfaces:" in output:
fail("interfaces rendered in bgp mode")
print("MetalLB interfaces regression test passed")
if __name__ == "__main__":
main()
+72
View File
@@ -0,0 +1,72 @@
#!/usr/bin/env python3
"""Regression test for the MetalLB namespace existence check.
The "Test metallb-system namespace" task in
roles/k3s_server_post/tasks/metallb.yml must actually verify the namespace
exists. A previous version ran `k3s kubectl -n metallb-system` with no
subcommand, which only printed a usage page and always exited 0, so the task
always succeeded even when the namespace did not exist (issue #350).
This test loads the real task and asserts the command performs an explicit
`get namespace metallb-system`, which returns non-zero when the namespace is
absent.
"""
from __future__ import print_function
import os
import subprocess
import yaml
def repo_root():
return subprocess.check_output(
["git", "rev-parse", "--show-toplevel"], text=True
).strip()
def fail(message):
raise SystemExit(
"MetalLB namespace test failed: " + message
)
def main():
task_file = os.path.join(
repo_root(), "roles", "k3s_server_post", "tasks", "metallb.yml"
)
with open(task_file, encoding="utf-8") as handle:
tasks = yaml.safe_load(handle)
task = None
for entry in tasks:
if entry.get("name") == "Test metallb-system namespace":
task = entry
break
if task is None:
fail("could not find the 'Test metallb-system namespace' task")
cmd = task.get("ansible.builtin.command")
if not cmd:
cmd = task.get("command")
if not cmd:
fail("task does not use ansible.builtin.command")
command_text = cmd if isinstance(cmd, str) else " ".join(cmd)
# A bare `-n metallb-system` with no subcommand prints kubectl usage and
# always exits 0, so it never proves the namespace exists. The fix must
# use an explicit get.
if "get namespace metallb-system" not in command_text:
fail(
"command does not run 'get namespace metallb-system'; "
"the task would only print usage and never verify the namespace "
"(got: {0!r})".format(command_text)
)
print("MetalLB namespace check regression test passed")
if __name__ == "__main__":
main()
+28
View File
@@ -0,0 +1,28 @@
#!/usr/bin/env bash
set -Eeuo pipefail
repo_root="$(git rev-parse --show-toplevel)"
site_play="$repo_root/site.yml"
# #636: verify the "Pre tasks" play asserts that all k3s_cluster hosts have
# unique hostnames, so a duplicate-hostname inventory fails fast instead of
# silently breaking node registration/joining.
grep -Fq -- 'Verify all cluster nodes have unique hostnames' "$site_play" || {
printf 'site.yml is missing the unique-hostname preflight check\n' >&2
exit 1
}
# The check must deduplicate the cluster hostname list via the `unique` filter
# and compare lengths, i.e. groups['k3s_cluster'] must be referenced.
grep -Fq -- "groups['k3s_cluster']" "$site_play" || {
printf 'unique-hostname check does not iterate the k3s_cluster group\n' >&2
exit 1
}
if ! grep -Eq -- 'cluster_hostnames.*\|.*unique|\| unique' "$site_play"; then
printf 'unique-hostname check does not deduplicate the hostname list\n' >&2
exit 1
fi
printf 'Unique hostname preflight regression test passed\n'
+4
View File
@@ -14,6 +14,7 @@ on:
- '**/FUNDING.yml'
- '**/host.ini'
- '**/*.md'
- '.github/ISSUE_TEMPLATE/**'
- '**/.editorconfig'
- '**/ansible.example.cfg'
- '**/deploy.sh'
@@ -31,10 +32,13 @@ concurrency:
jobs:
pre:
if: github.actor != 'dependabot[bot]'
uses: ./.github/workflows/cache.yml
needs: [lint]
lint:
if: github.actor != 'dependabot[bot]'
uses: ./.github/workflows/lint.yml
test:
if: github.actor != 'dependabot[bot]'
uses: ./.github/workflows/test.yml
needs: [pre, lint]
+1 -1
View File
@@ -68,7 +68,7 @@ jobs:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # 7.0.1
- name: Ensure SHA pinned actions
uses: zgosalvez/github-actions-ensure-sha-pinned-actions@3db98c0363e2fa5df3e1c4c471777a7c10b24cc9 # 5.0.5
uses: zgosalvez/github-actions-ensure-sha-pinned-actions@46cfe808a5f1588656ef299eedd0ce2fd7ec0dcc # 5.0.6
with:
allowlist: |
aws-actions/
+1 -1
View File
@@ -88,7 +88,7 @@ jobs:
trap stop_monitor EXIT
/usr/bin/time -v -o "$timing_file" \
molecule test --scenario-name ${{ matrix.scenario }}
timeout-minutes: 150
timeout-minutes: 180
env:
ANSIBLE_K3S_LOG_DIR: ${{ runner.temp }}/logs/k3s-ansible/${{ matrix.scenario }}
ANSIBLE_SSH_RETRIES: 4
+54
View File
@@ -62,6 +62,18 @@ repos:
language: system
pass_filenames: false
files: ^roles/k3s_server/tasks/(main|join_master)\.yml$|^\.github/scripts/test-k3s-server-bootstrap\.sh$
- id: unique-hostname-precheck-test
name: Unique hostname precheck test
entry: .github/scripts/test-unique-hostname-precheck.sh
language: system
pass_filenames: false
files: ^site\.yml$|^\.github/scripts/test-unique-hostname-precheck\.sh$
- id: disable-swap-test
name: Disable swap test
entry: .github/scripts/test-disable-swap.sh
language: system
pass_filenames: false
files: ^roles/prereq/(tasks/main|defaults/main)\.yml$|^\.github/scripts/test-disable-swap\.sh$
- id: cilium-bgp-manifest-test
name: Cilium BGP manifest test
entry: python3 .github/scripts/test-cilium-bgp-manifest.py
@@ -70,6 +82,15 @@ repos:
- Jinja2>=3.1
pass_filenames: false
files: ^roles/k3s_server_post/templates/cilium\.crs\.j2$|^\.github/scripts/test-cilium-bgp-manifest\.py$
- id: cilium-envoy-toggle-test
name: Cilium Envoy toggle test
entry: python3 .github/scripts/test-cilium-envoy-toggle.py
language: python
additional_dependencies:
- Jinja2>=3.1
- PyYAML
pass_filenames: false
files: ^roles/k3s_server_post/tasks/cilium\.yml$|^\.github/scripts/test-cilium-envoy-toggle\.py$
- id: kube-vip-manifest-test
name: kube-vip manifest test
entry: python3 .github/scripts/test-kube-vip-manifest.py
@@ -84,3 +105,36 @@ repos:
language: system
pass_filenames: false
files: ^roles/k3s_server/tasks/metallb\.yml$|^\.github/scripts/test-metallb-remote-read\.sh$
- id: metallb-interfaces-test
name: MetalLB interfaces test
entry: python3 .github/scripts/test-metallb-interfaces.py
language: python
additional_dependencies:
- Jinja2>=3.1
pass_filenames: false
files: ^roles/k3s_server_post/templates/metallb\.crs\.j2$|^\.github/scripts/test-metallb-interfaces\.py$
- id: metallb-namespace-test
name: MetalLB namespace test
entry: python3 .github/scripts/test-metallb-namespace.py
language: python
additional_dependencies:
- PyYAML
pass_filenames: false
files: ^roles/k3s_server_post/tasks/metallb\.yml$|^\.github/scripts/test-metallb-namespace\.py$
- id: metallb-deploy-condition-test
name: MetalLB deploy condition test
entry: python3 .github/scripts/test-metallb-deploy-condition.py
language: python
additional_dependencies:
- Jinja2>=3.1
- PyYAML
pass_filenames: false
files: ^roles/k3s_server/tasks/main\.yml$|^roles/k3s_server_post/tasks/main\.yml$|^\.github/scripts/test-metallb-deploy-condition\.py$ # noqa yaml[line-length]
- id: default-interface-test
name: default interface test
entry: python3 .github/scripts/test-default-interface.py
language: python
additional_dependencies:
- Jinja2>=3.1
pass_filenames: false
files: ^inventory/sample/group_vars/all\.yml$|^\.github/scripts/test-default-interface\.py$
+130
View File
@@ -0,0 +1,130 @@
# Agent Guide
This file is the canonical repository guide for coding agents and automated contributors. Read it before making
changes. Human contributors should also review [CONTRIBUTING.md](CONTRIBUTING.md).
## Project overview
This repository is an Ansible collection that provisions and resets highly available k3s clusters. It supports
multiple networking choices, including Flannel, Calico, Cilium, kube-vip, and MetalLB.
The main entry points are:
- `site.yml`: provision or update a cluster.
- `reset.yml`: remove k3s from a cluster.
- `reboot.yml`: reboot cluster nodes.
- `inventory/sample/`: example inventory and variables.
- `roles/`: reusable Ansible roles used by the playbooks.
- `molecule/`: integration scenarios run by CI.
- `.github/scripts/`: CI support scripts and focused regression tests.
## Source of truth
- Role defaults belong in `roles/<role>/defaults/main.yml`.
- Tasks belong in `roles/<role>/tasks/` and handlers in `roles/<role>/handlers/`.
- Example user configuration belongs in `inventory/sample/`.
- User-facing setup and variable documentation belongs in `README.md`.
- Contributor workflows and review expectations belong in `CONTRIBUTING.md`.
- Agent-specific repository instructions belong in this file.
Keep `CLAUDE.md` and `.github/copilot-instructions.md` as small pointers to this file. Do not duplicate these
instructions in tool-specific files.
## Development setup
Use a Python virtual environment. Do not commit the environment, generated logs, inventories, kubeconfigs, or
credentials.
```bash
python3 -m venv .env
source .env/bin/activate
python3 -m pip install -r requirements.txt
ansible-galaxy collection install -r collections/requirements.yml
pre-commit install
```
`ansible.cfg` is intentionally ignored. Copy `ansible.example.cfg` when local configuration is needed.
## Working rules
1. Inspect the current branch and worktree before editing. Preserve unrelated user changes.
2. Keep changes focused. Avoid drive-by formatting or dependency updates.
3. Never add real IP addresses, hostnames, tokens, private keys, kubeconfigs, or inventory secrets.
4. Use placeholders in examples and redact sensitive values from logs and issue reports.
5. Preserve idempotence. An already-converged host should not report changes without a real state transition.
6. Prefer Ansible modules over `ansible.builtin.command` or `ansible.builtin.shell`. When a command is required,
define accurate `changed_when` and `failed_when` behavior.
7. Use fully qualified collection names, such as `ansible.builtin.copy`.
8. Put configurable values in role defaults or inventory variables. Avoid embedding environment-specific values in
tasks and templates.
9. Maintain compatibility with the operating systems and architectures listed in `README.md`.
10. Do not weaken lint rules, tests, or CI checks to make a change pass.
## Change guidance
### Ansible tasks and roles
- Use descriptive task names in sentence case.
- Use YAML booleans (`true` and `false`) rather than aliases.
- Quote file modes, for example `mode: "0644"`.
- Notify handlers only when the managed resource changes.
- Use `become: true` only where privilege escalation is needed.
- Update role defaults, sample inventory, and the README together when adding or renaming user-facing variables.
- Check reset behavior when provisioning introduces persistent services, files, mounts, or network state.
### Templates and manifests
- Keep Jinja logic small and readable. Move complicated decisions into task variables where practical.
- Render valid YAML after Jinja evaluation.
- Preserve explicit handling for optional and undefined variables.
- Add or update a focused test under `.github/scripts/` when changing generated Kubernetes manifests or bootstrap
behavior.
### Molecule scenarios
- Reuse `molecule/resources/` for shared behavior.
- Put scenario-specific inputs in `molecule/<scenario>/overrides.yml` and `verify-vars.yml`.
- Update `molecule/README.md` when adding, removing, or materially changing a scenario.
- Clean up resources created by tests, including failure paths.
## Validation
Run the smallest relevant checks while iterating, then run the complete local validation before considering a change
ready:
```bash
pre-commit run --all-files
```
For playbook or role changes, also run syntax checks with a non-sensitive inventory:
```bash
ansible-playbook site.yml --syntax-check -i inventory/sample/hosts.ini
ansible-playbook reset.yml --syntax-check -i inventory/sample/hosts.ini
```
Run focused regression scripts when their related files change. The mapping is defined in
`.pre-commit-config.yaml`.
Molecule tests require Vagrant, VirtualBox, host-only networking, and substantial local resources. Run the most
relevant scenario when that environment is available:
```bash
molecule test --scenario-name <scenario>
```
If a required test can't be run locally, state exactly which check was skipped and why. Never claim a check passed
unless it was executed.
## Documentation and review
- Keep commands copyable and examples free of secrets.
- Update documentation in the same change as user-visible behavior.
- Explain behavior changes, compatibility concerns, operational risks, and rollback steps in the pull request.
- Use conventional commit messages with a scope, for example `fix(k3s-server): handle an existing token safely`.
- Do not commit, push, open a pull request, or modify remote resources unless the user explicitly requests it.
## Definition of done
A change is ready for review when it is focused, documented, linted, tested in proportion to its risk, and shown in a
clean diff with no secrets or generated artifacts.
+3
View File
@@ -0,0 +1,3 @@
@AGENTS.md
`AGENTS.md` is the canonical repository guide. Follow it for all work in this repository.
+94
View File
@@ -0,0 +1,94 @@
# Contributing to k3s-ansible
Thank you for improving k3s-ansible. Contributions should be focused, safe to apply to existing clusters, and tested
in proportion to their operational impact.
## Before opening an issue
- Search existing issues and discussions for the same behavior.
- Review the [troubleshooting discussion](https://github.com/timothystewart6/k3s-ansible/discussions/20).
- Remove tokens, credentials, public IP addresses, private hostnames, and other sensitive values from logs and
configuration.
- For support requests, include the k3s-ansible revision, Ansible version, target operating system, architecture,
network provider, relevant sanitized variables, and a minimal reproduction.
Use the bug report template for reproducible defects and the feature request template for proposed behavior.
## Development environment
Fork and clone the repository, then create a branch from the latest `master`:
```bash
git switch master
git pull --ff-only
git switch -c <type>/<short-description>
```
Create a Python environment and install the pinned development dependencies:
```bash
python3 -m venv .env
source .env/bin/activate
python3 -m pip install -r requirements.txt
ansible-galaxy collection install -r collections/requirements.yml
pre-commit install
```
Copy `ansible.example.cfg` to the ignored `ansible.cfg` file if local Ansible configuration is needed. Start custom
inventories from `inventory/sample/`, keep them out of Git, and never use production credentials in tests.
## Making changes
- Keep each pull request focused on one problem or feature.
- Follow [AGENTS.md](AGENTS.md) for repository structure, implementation conventions, and safety requirements.
- Preserve idempotence and existing-cluster compatibility.
- Add or update tests for behavior changes and regressions.
- Update role defaults, sample inventory, and documentation when user-facing variables change.
- Consider both provisioning and reset behavior for persistent resources.
- Avoid unrelated reformatting and generated files.
## Validation
Run all pre-commit checks before submitting a pull request:
```bash
pre-commit run --all-files
```
For playbook or role changes, run syntax checks:
```bash
ansible-playbook site.yml --syntax-check -i inventory/sample/hosts.ini
ansible-playbook reset.yml --syntax-check -i inventory/sample/hosts.ini
```
Run the most relevant Molecule scenario when Vagrant, VirtualBox, and the required host networking are available:
```bash
molecule test --scenario-name <scenario>
```
See [molecule/README.md](molecule/README.md) for scenario details and local requirements. Pull requests should list
every check that was run and clearly identify checks that could not be run locally.
## Commits and pull requests
Use a conventional commit subject with a scope:
```text
type(scope): short description
```
Common types are `feat`, `fix`, `refactor`, `test`, `docs`, and `chore`. Write subjects in the imperative mood and
keep commits logically focused.
Pull requests should:
- Describe the problem and the resulting behavior.
- Identify compatibility, security, networking, and upgrade risks.
- Include testing evidence without sensitive data.
- Call out documentation and sample configuration changes.
- Link related issues with `Fixes #<issue>` when applicable.
- Avoid checking boxes for tests that were not run.
Maintainers may ask for a change to be split when unrelated work makes it difficult to review or roll back.
+76 -23
View File
@@ -2,51 +2,70 @@
![Fully Automated K3S etcd High Availability Install](https://img.youtube.com/vi/CbkEWcUZ7zM/0.jpg)
This playbook will build an HA Kubernetes cluster with `k3s`, `kube-vip` and MetalLB via `ansible`.
This Ansible collection builds a highly available Kubernetes cluster with k3s. It supports kube-vip for the control
plane virtual IP, multiple CNI options, and either MetalLB or kube-vip for service load balancing.
This is based on the work from [this fork](https://github.com/212850a/k3s-ansible) which is based on the work from [k3s-io/k3s-ansible](https://github.com/k3s-io/k3s-ansible). It uses [kube-vip](https://kube-vip.io/) to create a load balancer for control plane, and [metal-lb](https://metallb.universe.tf/installation/) for its service `LoadBalancer`.
If you want more context on how this works, see:
For more context on how it works, see:
📄 [Documentation](https://technotim.com/posts/k3s-etcd-ansible/) (including example commands)
📺 [Watch the Video](https://www.youtube.com/watch?v=CbkEWcUZ7zM)
## Project guides
- [Getting started](#-getting-started)
- [Configuration variables](#variables)
- [Upgrading an existing cluster](#-upgrading-an-existing-cluster)
- [Local Molecule testing](molecule/README.md)
- [Contributing guidelines](CONTRIBUTING.md)
- [Repository guide for coding agents](AGENTS.md)
## 📖 k3s Ansible Playbook
Build a Kubernetes cluster using Ansible with k3s. The goal is easily install a HA Kubernetes cluster on machines running:
Build a Kubernetes cluster using Ansible and k3s. The goal is to make a highly available cluster straightforward to
install on machines running:
- [x] Debian (tested on version 13)
- [x] Ubuntu (tested on version 26.04 LTS)
- [x] Rocky (tested on version 10)
on processor architecture:
Supported processor architectures are:
- [X] x64
- [X] arm64
- [X] armhf
- [x] x64
- [x] arm64
- [x] armhf
## ✅ System requirements
- Control Node (the machine you are running `ansible` commands) must have Ansible 2.11+ If you need a quick primer on Ansible [you can check out my docs and setting up Ansible](https://technotim.com/posts/ansible-automation/).
- The control node, which runs the Ansible commands, must have Ansible 2.11 or newer. For a quick primer, see
[setting up Ansible](https://technotim.com/posts/ansible-automation/).
- You will also need to install collections that this playbook uses by running `ansible-galaxy collection install -r ./collections/requirements.yml` (important❗)
- Install the required collections with
`ansible-galaxy collection install -r ./collections/requirements.yml`.
- [`netaddr` package](https://pypi.org/project/netaddr/) must be available to Ansible. If you have installed Ansible via apt, this is already taken care of. If you have installed Ansible via `pip`, make sure to install `netaddr` into the respective virtual environment.
- `server` and `agent` nodes should have passwordless SSH access, if not you can supply arguments to provide credentials `--ask-pass --ask-become-pass` to each command.
- Server and agent nodes should support passwordless SSH access. Otherwise, pass `--ask-pass --ask-become-pass` to
each playbook command.
- Every node in the cluster must have a **unique hostname**. k3s registers each node keyed by its hostname, so
two nodes with the same hostname cannot join the cluster. `site.yml` asserts this up front and fails fast if
any duplicate is found.
## 🚀 Getting Started
### 🍴 Preparation
First create a new directory based on the `sample` directory within the `inventory` directory:
Create a cluster-specific inventory from the sample. The `inventory/` directory ignores custom inventory content so
credentials and environment details aren't committed accidentally.
```bash
cp -R inventory/sample inventory/my-cluster
```
Second, edit `inventory/my-cluster/hosts.ini` to match the system information gathered above
Edit `inventory/my-cluster/hosts.ini` to match the target hosts.
For example:
@@ -67,9 +86,10 @@ node
If multiple hosts are in the master group, the playbook will automatically set up k3s in [HA mode with etcd](https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/).
Finally, copy `ansible.example.cfg` to `ansible.cfg` and adapt the inventory path to match the files that you just created.
Copy `ansible.example.cfg` to `ansible.cfg`, then update its inventory path. The local `ansible.cfg` file is ignored by
Git.
This requires at least k3s version `1.19.1` however the version is configurable by using the `k3s_version` variable.
The minimum k3s version is `1.19.1`. Select the desired version with the `k3s_version` variable.
If needed, you can also edit `inventory/my-cluster/group_vars/all.yml` to match your environment.
@@ -81,7 +101,8 @@ Start provisioning of the cluster using the following command:
ansible-playbook site.yml -i inventory/my-cluster/hosts.ini
```
After deployment control plane will be accessible via virtual ip-address which is defined in inventory/group_vars/all.yml as `apiserver_endpoint`
After deployment, the control plane is accessible through the virtual IP defined by `apiserver_endpoint` in the
inventory variables.
### 🔥 Remove k3s cluster
@@ -89,7 +110,25 @@ After deployment control plane will be accessible via virtual ip-address which i
ansible-playbook reset.yml -i inventory/my-cluster/hosts.ini
```
>You should also reboot these nodes due to the VIP not being destroyed
> Reboot the nodes after reset because the virtual IP may remain configured.
### ⏻️ Reboot Cluster Nodes
Reboot all cluster nodes at once or stage the reboot across the cluster.
```bash
ansible-playbook reboot.yml -i inventory/my-cluster/hosts.ini
```
To reboot the nodes in batches, set `concurrent_reboots` to the number of nodes
to reboot at a time (or a percentage). Optionally set `wait_seconds_after_reboot`
to pause after each batch so pods in the freshly rebooted batch can settle
before the next batch reboots.
```bash
ansible-playbook reboot.yml -i inventory/my-cluster/hosts.ini \
--extra-vars 'concurrent_reboots=2 wait_seconds_after_reboot=30'
```
## 🔁 Upgrading an existing cluster
@@ -124,13 +163,21 @@ To copy your `kube config` locally so that you can access your **Kubernetes** cl
```bash
scp debian@master_ip:/etc/rancher/k3s/k3s.yaml ~/.kube/config
```
If you get file Permission denied, go into the node and temporarly run:
If the copy fails with a permission error, grant the SSH user temporary read access using the least permissive method
available for the target system. Restore the original ownership and permissions immediately after copying. Avoid
world-writable permissions on the kubeconfig because it contains cluster credentials.
For example, copy the file to a temporary user-readable path from the control node:
```bash
sudo chmod 777 /etc/rancher/k3s/k3s.yaml
ssh debian@master_ip 'sudo install -o "$(id -un)" -m 0600 /etc/rancher/k3s/k3s.yaml /tmp/k3s.yaml'
```
Then copy with the scp command and reset the permissions back to:
Copy `/tmp/k3s.yaml`, then remove the temporary remote copy:
```bash
sudo chmod 600 /etc/rancher/k3s/k3s.yaml
scp debian@master_ip:/tmp/k3s.yaml ~/.kube/config
ssh debian@master_ip rm -f /tmp/k3s.yaml
```
You'll then want to modify the config to point to master IP by running:
@@ -150,7 +197,7 @@ See the commands [here](https://technotim.com/posts/k3s-etcd-ansible/#testing-yo
| `download` | `k3s_version` | string | ❌ | Required | K3s binaries version |
| `k3s_agent`, `k3s_server`, `k3s_server_post` | `apiserver_endpoint` | string | ❌ | Required | Virtual ip-address configured on each master |
| `k3s_agent` | `extra_agent_args` | string | `null` | Not required | Extra arguments for agents nodes |
| `k3s_agent`, `k3s_server` | `group_name_master` | string | `null` | Not required | Name othe master group |
| `k3s_agent`, `k3s_server` | `group_name_master` | string | `null` | Not required | Name of the master group |
| `k3s_agent` | `k3s_token` | string | `null` | Not required | Token used to communicate between masters |
| `k3s_agent`, `k3s_server` | `proxy_env` | dict | `null` | Not required | Internet proxy configurations |
| `k3s_agent`, `k3s_server` | `proxy_env.HTTP_PROXY` | string | ❌ | Required | HTTP internet proxy |
@@ -172,6 +219,7 @@ See the commands [here](https://technotim.com/posts/k3s-etcd-ansible/#testing-yo
| `k3s_server` | `kube_vip_bgp_peers` | list | `[]` | Not required | List of BGP peer ASN & address pairs |
| `k3s_server` | `kube_vip_bgp_peers_groups` | list | `['k3s_master']` | Not required | Inventory group in which to search for additional `kube_vip_bgp_peers` parameters to merge. |
| `k3s_server` | `kube_vip_iface` | string | `~` | Not required | Explicitly define an interface that ALL control nodes should use to propagate the VIP, define it here. Otherwise, kube-vip will determine the right interface automatically at runtime. |
| `k3s_server` | `kube_vip_endpoint` | string | `~` | Not required | Overrides the internal address kube-vip binds/listens on, which can differ from the announced apiserver_endpoint for complex routing/tunnels. Defaults to apiserver_endpoint. |
| `k3s_server` | `kube_vip_tag_version` | string | `v1.2.2` | Not required | Image tag for kube-vip |
| `k3s_server` | `kube_vip_cloud_provider_tag_version` | string | `v0.0.12` | Not required | Tag for kube-vip-cloud-provider manifest when enable |
| `k3s_server`, `k3_server_post` | `kube_vip_lb_ip_range` | string | `~` | Not required | IP range for kube-vip load balancer |
@@ -211,7 +259,10 @@ See the commands [here](https://technotim.com/posts/k3s-etcd-ansible/#testing-yo
| `k3s_server_post` | `metal_lb_bgp_peer_asn` | string | `~` | Not required | BGP peer ASN configurations |
| `k3s_server_post` | `metal_lb_bgp_peer_address` | string | `~` | Not required | BGP peer address |
| `lxc` | `custom_reboot_command` | string | `~` | Not required | Command to run on reboot |
| `reboot` (playbook) | `concurrent_reboots` | int/string | `100%` | Not required | Number (or percentage) of nodes to reboot at a time for a staggered reboot |
| `reboot` (playbook) | `wait_seconds_after_reboot` | int | `0` | Not required | Pause in seconds between staggered reboot batches |
| `prereq` | `system_timezone` | string | `null` | Not required | Timezone to be set on all nodes |
| `prereq` | `disable_swap` | bool | `true` | Not required | Disable swap on all cluster nodes (swapoff + comment out /etc/fstab swap entries), all-or-nothing |
| `proxmox_lxc`, `reset_proxmox_lxc` | `proxmox_lxc_ct_ids` | list | ❌ | Required | Proxmox container ID list |
| `raspberrypi` | `state` | string | `present` | Not required | Indicates whether the k3s prerequisites for Raspberry Pi should be set up (possible values are `present` and `absent`) |
@@ -227,9 +278,11 @@ It is run automatically in CI, but you can also run the tests locally.
This might be helpful for quick feedback in a few cases.
You can find more information about it [here](molecule/README.md).
### Pre-commit Hooks
### Pre-commit hooks
This repo uses `pre-commit` and `pre-commit-hooks` to lint and fix common style and syntax errors. Be sure to install python packages and then run `pre-commit install`. For more information, see [pre-commit](https://pre-commit.com/)
This repository uses `pre-commit` to check style, syntax, Ansible content, and shell scripts. Install the Python
dependencies, run `pre-commit install` once, and run `pre-commit run --all-files` before submitting a change. See
[CONTRIBUTING.md](CONTRIBUTING.md) for the complete development workflow.
## 🌌 Ansible Galaxy
+27 -4
View File
@@ -7,21 +7,31 @@ systemd_dir: /etc/systemd/system
# Set your timezone
system_timezone: Your/Timezone
# k3s recommends swap be disabled on every cluster node. Applied uniformly to all
# nodes (all-or-nothing) in the prereq role. Set to false to leave swap enabled.
disable_swap: true
# interface which will be used for flannel
flannel_iface: eth0
# Defaults to each host's default IPv4 interface (e.g. eth0, enp1s0, ens3)
# so KVM/cloud hosts without eth0 work out of the box. Override per-host if needed.
flannel_iface: "{{ ansible_facts.default_ipv4.interface }}"
# uncomment calico_iface to use tigera operator/calico cni instead of flannel https://docs.tigera.io/calico/latest/about
# calico_iface: "eth0"
# calico_iface: "{{ ansible_facts.default_ipv4.interface }}"
calico_ebpf: false # use eBPF dataplane instead of iptables
calico_tag: v3.32.1 # calico version tag
# uncomment cilium_iface to use cilium cni instead of flannel or calico
# ensure v4.19.57, v5.1.16, v5.2.0 or more recent kernel
# cilium_iface: "eth0"
# cilium_iface: "{{ ansible_facts.default_ipv4.interface }}"
cilium_mode: native # native when nodes are on the same subnet or use BGP, otherwise set tunnel
cilium_tag: v1.20.0 # cilium version tag
cilium_cli_tag: v0.19.7 # cilium cli version tag
cilium_hubble: true # enable hubble observability relay and ui
cilium_envoy: true # enable the Envoy proxy for Cilium L7 policies
# disable cilium_envoy to skip the Envoy proxy entirely (e.g. no L7 policies)
# cilium_envoy: false
# if using calico or cilium, you may specify the cluster pod cidr pool
cluster_cidr: 10.52.0.0/16
@@ -38,6 +48,11 @@ cilium_bgp_lb_cidr: 192.168.31.0/24 # cidr for cilium loadbalancer ipam
# enable kube-vip ARP broadcasts
kube_vip_arp: true
# (optional) overrides the address kube-vip binds/listens on internally, which
# can differ from the announced apiserver_endpoint for complex routing/tunnels.
# Defaults to apiserver_endpoint. Also used to derive the kube-vip subnet.
# kube_vip_endpoint: 10.66.1.5
# enable kube-vip BGP peering
kube_vip_bgp: false
@@ -47,7 +62,9 @@ kube_vip_bgp_as: "64513" # Defines the AS for the BGP server
kube_vip_bgp_peeraddress: "192.168.30.1" # Defines the address for the BGP peer
kube_vip_bgp_peeras: "64512" # Defines the AS for the BGP peer
# apiserver_endpoint is virtual ip-address which will be configured on each master
# apiserver_endpoint is virtual ip-address which will be configured on each master.
# This must be a free, routable IP on your network (not already assigned to a host
# or service), and is used by kube-vip / MetalLB to expose the Kubernetes API.
apiserver_endpoint: 192.168.30.222
# k3s_token is required masters can talk together securely
@@ -112,6 +129,12 @@ metal_lb_controller_tag_version: v0.16.0
# metallb ip range for load balancer
metal_lb_ip_range: 192.168.30.80-192.168.30.90
# (optional) limit MetalLB layer2 announcements to specific network interfaces.
# Leave empty (default) to announce on all interfaces.
# metal_lb_interfaces:
# - eth1
# - eth2
# Only enable if your nodes are proxmox LXC nodes, make sure to configure your proxmox nodes
# in your hosts.ini file.
# Please read https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185 before using this.
@@ -38,6 +38,30 @@
loop_control:
label: "{{ item.metadata.name }} ready"
- name: Assert every node registered a node IP from its interface
# Each k3s node is launched with --node-ip derived from flannel_iface.
# Confirm every node carries a real InternalIP (not a loopback), which
# proves k3s bound to the cluster interface rather than defaulting to 127.0.0.1.
ansible.builtin.assert:
that: >-
(node_internal_ips | length) >= 1 and
(node_internal_ips | reject('eq', '127.0.0.1') | list | length) == node_internal_ips | length
success_msg: "{{ item.metadata.name }} is bound to {{ node_internal_ips | join(', ') }}"
fail_msg: >-
{{ item.metadata.name }} has no non-loopback InternalIP
(got: {{ node_internal_ips | join(', ') }})
vars:
node_internal_ips: >-
{{
(item.status.addresses | default([]))
| selectattr('type', 'equalto', 'InternalIP')
| map(attribute='address')
| list
}}
loop: "{{ verify_nodes.resources }}"
loop_control:
label: "{{ item.metadata.name }} InternalIP"
- name: Get any Calico namespaces with Flannel enabled
kubernetes.core.k8s_info:
kind: Namespace
@@ -79,7 +103,7 @@
ansible.builtin.assert:
that:
- calico_node_ds.resources | length == 1
- calico_node_image | regex_search(':' ~ calico_tag)
- calico_node_image | regex_search(':' ~ calico_tag) is not none
success_msg: "Calico node image uses tag {{ calico_tag }}"
fail_msg: >-
Calico node image {{ calico_node_image }},
@@ -145,8 +169,8 @@
- name: Assert Cilium agent and operator use the expected image tag
ansible.builtin.assert:
that:
- cilium_agent_image | regex_search(':' ~ cilium_tag)
- cilium_operator_image | regex_search(':' ~ cilium_tag)
- cilium_agent_image | regex_search(':' ~ cilium_tag) is not none
- cilium_operator_image | regex_search(':' ~ cilium_tag) is not none
success_msg: "Cilium agent and operator use {{ cilium_tag }}"
fail_msg: >-
Cilium agent {{ cilium_agent_image }},
@@ -219,17 +243,34 @@
namespace: metallb-system
kubeconfig: "{{ kubecfg_path }}"
register: metallb_info
until: metallb_info.resources | length > 0
retries: 15
delay: 10
loop:
- { kind: Deployment, name: controller }
- { kind: DaemonSet, name: speaker }
loop_control:
label: "{{ item.kind }}/{{ item.name }}"
- name: Fail with a clear message if MetalLB resources are missing
ansible.builtin.fail:
msg: >-
Did not find {{ item.kind | lower }} {{ item.name }} in
metallb-system. Expected MetalLB to be deployed in this
scenario (verify_lb: {{ verify_lb }}).
when: item.resources | length == 0
loop: "{{ metallb_info.results }}"
loop_control:
label: "{{ item.item.kind }}/{{ item.item.name }}"
- name: Assert MetalLB controller and speaker use the expected image tags
ansible.builtin.assert:
# regex_search returns a string or none; check for a match with `is not
# none` so the assertion is a real boolean (ansible-core 2.19 rejects
# string conditionals and `| bool` deprecates string coercion).
that:
- controller_image | regex_search(metal_lb_controller_tag_version)
- speaker_image | regex_search(metal_lb_speaker_tag_version)
- controller_image | regex_search(metal_lb_controller_tag_version) is not none
- speaker_image | regex_search(metal_lb_speaker_tag_version) is not none
success_msg: >-
MetalLB controller {{ metal_lb_controller_tag_version }},
speaker {{ metal_lb_speaker_tag_version }}
@@ -273,8 +314,8 @@
- name: Assert the kube-vip and cloud provider image tags
ansible.builtin.assert:
that:
- kubevip_image | regex_search(':' ~ kube_vip_tag_version)
- cloud_provider_image | regex_search(verify_kube_vip_cloud_provider_tag)
- kubevip_image | regex_search(':' ~ kube_vip_tag_version) is not none
- cloud_provider_image | regex_search(verify_kube_vip_cloud_provider_tag) is not none
success_msg: >-
kube-vip {{ kube_vip_tag_version }},
cloud provider {{ verify_kube_vip_cloud_provider_tag }}
+21 -1
View File
@@ -2,9 +2,29 @@
- name: Reboot k3s_cluster
hosts: k3s_cluster
gather_facts: true
# Stagger the reboot across the cluster when concurrent_reboots is set.
# Defaults to '100%' so the whole cluster reboots at once (backward compatible).
serial: "{{ concurrent_reboots | default('100%') }}"
tasks:
- name: Reboot the nodes (and Wait upto 5 mins max)
- name: >-
{{
'Reboot all nodes at once'
if (concurrent_reboots is not defined)
else 'Reboot nodes with concurrency of ' ~ concurrent_reboots
}}
become: true
ansible.builtin.reboot:
reboot_command: "{{ custom_reboot_command | default(omit) }}"
reboot_timeout: 300
test_command: >-
{{ 'kubectl get nodes' if 'master' in group_names else 'whoami' }}
- name: Optional wait before rebooting the next batch of nodes
ansible.builtin.pause:
seconds: "{{ wait_seconds_after_reboot | int }}"
when: >-
concurrent_reboots is defined and
wait_seconds_after_reboot is defined and
wait_seconds_after_reboot | int > 0
+2 -2
View File
@@ -1,5 +1,5 @@
ansible-core>=2.16.2
jmespath>=1.0.1
ansible-core>=2.19.11
jmespath>=1.1.0
jsonpatch>=1.33
kubernetes>=29.0.0
molecule-plugins[vagrant]
+5 -5
View File
@@ -2,11 +2,11 @@
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# pip-compile requirements.in
# pip-compile --output-file=requirements.txt requirements.in
#
ansible-compat==4.1.11
# via molecule
ansible-core==2.18.0
ansible-core==2.19.11
# via
# -r requirements.in
# ansible-compat
@@ -53,7 +53,7 @@ jinja2==3.1.3
# via
# ansible-core
# molecule
jmespath==1.0.1
jmespath==1.1.0
# via -r requirements.in
jsonpatch==1.33
# via -r requirements.in
@@ -77,7 +77,7 @@ molecule==6.0.3
# via
# -r requirements.in
# molecule-plugins
molecule-plugins[vagrant]==23.5.3
molecule-plugins[vagrant]==23.6.0
# via -r requirements.in
netaddr==0.10.1
# via -r requirements.in
@@ -146,7 +146,7 @@ rsa==4.9
# via google-auth
ruamel-yaml==0.18.5
# via pre-commit-hooks
ruamel-yaml-clib==0.2.8
ruamel-yaml-clib==0.2.15
# via ruamel-yaml
six==1.16.0
# via
+1 -1
View File
@@ -11,7 +11,7 @@ ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s agent \
--server https://{{ apiserver_endpoint | ansible.utils.ipwrap }}:6443 \
{% if is_pxe_booted | default(false) %}--snapshotter native \
{% endif %}--token {{ hostvars[groups[group_name_master | default('master')][0]]['token'] | default(k3s_token) }} \
{% endif %}--token {{ hostvars[groups[group_name_master | default('master')][0]]['token'] | default(k3s_token) | replace( '%', '%%' ) }} \
{{ extra_agent_args }}
KillMode=process
Delegate=yes
+8 -1
View File
@@ -7,6 +7,7 @@ group_name_master: master
kube_vip_arp: true
kube_vip_iface:
kube_vip_endpoint:
kube_vip_cloud_provider_tag_version: v0.0.12
kube_vip_tag_version: v1.2.2
@@ -23,6 +24,12 @@ metal_lb_controller_tag_version: v0.16.0
metal_lb_speaker_tag_version: v0.16.0
metal_lb_type: native
# Shared retry/delay for remote manifest and asset downloads. The CI runner's
# resolver intermittently times out on GitHub-hosted domains (helm.cilium.io,
# raw.githubusercontent.com, github.com), so retry transient DNS/network failures.
download_retries: 5
download_delay: 10
retry_count: 20
# yamllint disable rule:line-length
@@ -33,7 +40,7 @@ server_init_args: >-
{% else %}
--server https://{{ hostvars[groups[group_name_master | default('master')][0]].k3s_node_ip | split(",") | first | ansible.utils.ipwrap }}:6443
{% endif %}
--token {{ k3s_token }}
--token {{ k3s_token | replace('%', '%%') }}
{% endif %}
{{ extra_server_args }}
+9
View File
@@ -78,6 +78,15 @@ argument_specs:
- automatically at runtime.
default: ~
kube_vip_endpoint:
description:
- Overrides the address kube-vip binds/listens on internally, which
- can differ from the announced apiserver_endpoint for complex
- routing and site-to-site tunnels.
- Defaults to apiserver_endpoint and is used to derive the kube-vip
- subnet.
default: ~
kube_vip_tag_version:
description: Image tag for kube-vip
default: v1.2.2
+4
View File
@@ -15,6 +15,10 @@
owner: root
group: root
mode: "0644"
register: kube_vip_manifest_download
retries: "{{ download_retries }}"
delay: "{{ download_delay }}"
until: kube_vip_manifest_download is succeeded
when: ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname']
- name: Copy kubevip configMap manifest to first master
+9 -2
View File
@@ -27,7 +27,11 @@
- name: Deploy metallb manifest
ansible.builtin.include_tasks: metallb.yml
tags: metallb
when: kube_vip_lb_ip_range is not defined and (not cilium_bgp or cilium_iface is not defined)
# Deploy MetalLB unless kube-vip owns the load balancer IP range, or Cilium
# BGP is enabled (Cilium then provides its own load balancing). The cilium_bgp
# default keeps this safe when Cilium variables are not in scope at all (#644)
# while still deploying MetalLB when a non-BGP Cilium CNI is in use.
when: kube_vip_lb_ip_range is not defined and not (cilium_bgp | default(false) | bool)
- name: Deploy kube-vip manifest
ansible.builtin.include_tasks: kube-vip.yml
@@ -118,7 +122,10 @@
always:
- name: Save logs of k3s-init.service
ansible.builtin.include_tasks: fetch_k3s_init_logs.yml
when: log_destination
# ANSIBLE_K3S_LOG_DIR is a path string when set; evaluate it as a boolean
# so the conditional is a real boolean (ansible-core 2.19 rejects string
# conditionals derived from env vars).
when: log_destination | default('') != ''
vars:
log_destination: >-
{{ lookup('ansible.builtin.env', 'ANSIBLE_K3S_LOG_DIR', default=False) }}
+4
View File
@@ -15,6 +15,10 @@
owner: root
group: root
mode: "0644"
register: metallb_manifest_download
retries: "{{ download_retries }}"
delay: "{{ download_delay }}"
until: metallb_manifest_download is succeeded
when: ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname']
- name: Set image versions in manifest for metallb-{{ metal_lb_type }}
+2 -2
View File
@@ -37,7 +37,7 @@ spec:
value: {{ kube_vip_iface }}
{% endif %}
- name: vip_subnet
value: "{{ apiserver_endpoint | ansible.utils.ipsubnet | ansible.utils.ipaddr('prefix') }}"
value: "{{ (kube_vip_endpoint | default(apiserver_endpoint, true)) | ansible.utils.ipsubnet | ansible.utils.ipaddr('prefix') }}"
- name: cp_enable
value: "true"
- name: cp_namespace
@@ -55,7 +55,7 @@ spec:
- name: vip_retryperiod
value: "2"
- name: address
value: {{ apiserver_endpoint }}
value: {{ kube_vip_endpoint | default(apiserver_endpoint, true) }}
{% if kube_vip_bgp | default(false) | bool %}
{% if kube_vip_bgp_routerid is defined %}
- name: bgp_routerid
+9
View File
@@ -18,11 +18,19 @@ cilium_bgp_peer_asn: 64512
cilium_bgp_neighbors: []
cilium_bgp_neighbors_groups: ['k3s_all']
cilium_bgp_lb_cidr: 192.168.31.0/24
cilium_envoy: true
cilium_hubble: true
cilium_mode: native
cilium_tag: v1.20.0
cilium_cli_tag: v0.19.7
# Shared retry/delay for remote manifest, asset downloads, and waiting on
# Kubernetes resources. The CI runner's resolver intermittently times out on
# GitHub-hosted domains and the kube API can transiently return
# ServiceUnavailable, so retry transient DNS/network/API failures.
download_retries: 5
download_delay: 10
cluster_cidr: 10.52.0.0/16
enable_bpf_masquerade: true
kube_proxy_replacement: true
@@ -31,4 +39,5 @@ group_name_master: master
metal_lb_mode: layer2
metal_lb_available_timeout: 240s
metal_lb_controller_tag_version: v0.16.0
metal_lb_interfaces: []
metal_lb_ip_range: 192.168.30.80-192.168.30.90
+8
View File
@@ -141,6 +141,14 @@ argument_specs:
description: MetalLB ip range for load balancer
default: 192.168.30.80-192.168.30.90
metal_lb_interfaces:
description: >-
List of network interfaces on which MetalLB should announce the
load balancer IPs in layer2 mode. When empty (default), MetalLB
announces on all interfaces.
type: list
default: []
metal_lb_controller_tag_version:
description: Image tag for MetalLB
default: v0.16.0
+8
View File
@@ -18,6 +18,10 @@
owner: root
group: root
mode: "0755"
register: calico_crd_download
retries: "{{ download_retries }}"
delay: "{{ download_delay }}"
until: calico_crd_download is succeeded
- name: "Download to first master: manifest for Tigera Operator and Calico CRDs"
ansible.builtin.get_url:
@@ -26,6 +30,10 @@
owner: root
group: root
mode: "0755"
register: tigera_operator_download
retries: "{{ download_retries }}"
delay: "{{ download_delay }}"
until: tigera_operator_download is succeeded
- name: Apply Calico CRD bundle with server-side apply
ansible.builtin.command: >-
+12
View File
@@ -66,6 +66,10 @@
- .tar.gz.sha256sum
vars:
cilium_base_url: https://github.com/cilium/cilium-cli/releases/download/{{ cilium_cli_tag }}
register: cilium_cli_download
retries: "{{ download_retries }}"
delay: "{{ download_delay }}"
until: cilium_cli_download is succeeded
- name: Verify the downloaded tarball
ansible.builtin.shell: |
@@ -174,6 +178,7 @@
--helm-set hubble.enabled={{ "true" if cilium_hubble else "false" }}
--helm-set hubble.relay.enabled={{ "true" if cilium_hubble else "false" }}
--helm-set hubble.ui.enabled={{ "true" if cilium_hubble else "false" }}
--helm-set envoy.enabled={{ "true" if cilium_envoy else "false" }}
{% if kube_proxy_replacement is not false %}
--helm-set loadBalancer.algorithm={{ bpf_lb_algorithm }}
--helm-set loadBalancer.mode={{ bpf_lb_mode }}
@@ -182,6 +187,13 @@
KUBECONFIG: "{{ ansible_user_dir }}/.kube/config"
register: cilium_install_result
changed_when: cilium_install_result.rc == 0
# cilium install/upgrade fetches the Helm chart from helm.cilium.io, which is
# fronted by GitHub Pages and can transiently fail DNS resolution through the
# host resolver (intermittent "lookup helm.cilium.io ... i/o timeout"). Retry
# so a transient name/network failure does not abort the whole converge play.
until: cilium_install_result.rc == 0
retries: "{{ download_retries }}"
delay: "{{ download_delay }}"
when: cilium_installed.rc != 0 or cilium_needs_update
- name: Wait for Cilium resources
+5 -1
View File
@@ -12,7 +12,11 @@
- name: Deploy metallb pool
ansible.builtin.include_tasks: metallb.yml
tags: metallb
when: kube_vip_lb_ip_range is not defined and (not cilium_bgp or cilium_iface is not defined)
# Deploy MetalLB unless kube-vip owns the load balancer IP range, or Cilium
# BGP is enabled (Cilium then provides its own load balancing). The cilium_bgp
# default keeps this safe when Cilium variables are not in scope at all (#644)
# while still deploying MetalLB when a non-BGP Cilium CNI is in use.
when: kube_vip_lb_ip_range is not defined and not (cilium_bgp | default(false) | bool)
- name: Remove tmp directory used for manifests
ansible.builtin.file:
+15 -1
View File
@@ -40,7 +40,7 @@
- name: Test metallb-system namespace
ansible.builtin.command: >-
{{ k3s_kubectl_binary | default('k3s kubectl') }} -n metallb-system
{{ k3s_kubectl_binary | default('k3s kubectl') }} get namespace metallb-system
changed_when: false
with_items: "{{ groups[group_name_master | default('master')] }}"
run_once: true
@@ -54,6 +54,12 @@
{% if item.condition | default(False) -%}{{ item.condition }}{%- endif %}
--timeout='{{ metal_lb_available_timeout }}'
changed_when: false
# The kube API can briefly return ServiceUnavailable while MetalLB converges,
# which would otherwise abort the whole converge play on a transient error.
register: metallb_wait_result
until: metallb_wait_result.rc == 0
retries: "{{ download_retries }}"
delay: "{{ download_delay }}"
run_once: true
with_items:
- description: controller
@@ -112,6 +118,10 @@
changed_when: false
run_once: true
when: metal_lb_mode == "layer2"
register: metallb_l2_test_result
until: metallb_l2_test_result.rc == 0
retries: "{{ download_retries }}"
delay: "{{ download_delay }}"
with_items:
- IPAddressPool
- L2Advertisement
@@ -122,6 +132,10 @@
changed_when: false
run_once: true
when: metal_lb_mode == "bgp"
register: metallb_bgp_test_result
until: metallb_bgp_test_result.rc == 0
retries: "{{ download_retries }}"
delay: "{{ download_delay }}"
with_items:
- IPAddressPool
- BGPPeer
@@ -21,6 +21,11 @@ kind: L2Advertisement
metadata:
name: default
namespace: metallb-system
{% if metal_lb_interfaces | default([]) | length > 0 %}
spec:
interfaces:{% for iface in metal_lb_interfaces %}
- {{ iface }}{% endfor %}
{% endif %}
{% endif %}
{% if metal_lb_mode == "bgp" %}
---
+2
View File
@@ -1,4 +1,6 @@
---
disable_swap: true
secure_path:
RedHat: /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin
Suse: /usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin
+43
View File
@@ -4,6 +4,28 @@
name: "{{ system_timezone }}"
when: (system_timezone is defined) and (system_timezone != "Your/Timezone")
# k3s recommends swap be disabled on all nodes. Disabling swap is all-or-nothing
# across the cluster: leaving it enabled on some nodes but not others creates
# uneven scheduling/latency behavior. This block turns swap off and comments out
# the swap entries in /etc/fstab so it stays off across reboots. It is idempotent
# and a no-op when swap is already disabled or swapoff is unavailable.
- name: Disable swap on all cluster nodes
when: disable_swap
block:
- name: Turn off swap now
ansible.builtin.command: swapoff -a
register: swapoff_result
changed_when: false
failed_when: false
- name: Comment out swap entries in fstab
ansible.builtin.replace:
path: /etc/fstab
regexp: '^([^#][^\n]*\s+swap\s+)'
replace: '# \\1'
register: fstab_swap
- name: Set SELinux to disabled state
ansible.posix.selinux:
state: disabled
@@ -33,6 +55,27 @@
reload: true
tags: sysctl
- name: Check if br_netfilter module exists
ansible.builtin.shell: |
set -o pipefail
find /lib/modules/$(uname -r) -name "br_netfilter.ko*" | wc -l
args:
executable: /bin/bash
register: br_netfilter_exists
changed_when: false
when: ansible_os_family == "RedHat"
tags: sysctl
- name: Install kernel-modules-extra if br_netfilter missing
ansible.builtin.yum:
name: kernel-modules-extra
state: present
update_cache: true
register: kernel_modules_installed
when:
- ansible_os_family == "RedHat"
- br_netfilter_exists.stdout | int == 0
- name: Add br_netfilter to /etc/modules-load.d/
ansible.builtin.copy:
content: br_netfilter
+17
View File
@@ -8,6 +8,23 @@
msg: >
"Ansible is out of date. See here for more info: https://docs.technotim.com/posts/ansible-automation/"
- name: Verify all cluster nodes have unique hostnames
ansible.builtin.assert:
that: (cluster_hostnames | unique | length) == (cluster_hostnames | length)
msg: >-
k3s nodes must have unique hostnames. Found a duplicate in:
{{ cluster_hostnames | unique }}. Each node registers in the cluster
keyed by its hostname, so matching hostnames prevent nodes from joining.
vars:
cluster_hostnames: >-
{{
groups['k3s_cluster']
| map('extract', hostvars, 'ansible_hostname')
| list
}}
run_once: true
when: "'k3s_cluster' in groups"
- name: Prepare Proxmox cluster
hosts: proxmox
gather_facts: true