mirror of
https://github.com/k3s-io/k3s-ansible.git
synced 2026-10-23 03:22:03 +02:00
Kris
b1f5388471
* feat: allow the server and agent plays to roll one node at a time The server and agent roles restart their k3s service on every run, and the plays run against the whole group at once. On a cluster with three or more servers that restarts every etcd member together, which costs the cluster its quorum and its API server. k3s_server_serial and k3s_agent_serial set the serial of the two plays. Both default to 100%, the whole group in one batch, so a run that does not set them behaves as before. The HA integration test now rolls the servers with the variable instead of --forks=1, which serialises the role rather than the individual tasks. Signed-off-by: Kristofer <kristofer@atlascloud.is> * feat: hold each node until it serves again during a rolling run Setting k3s_server_serial to 1 rolls the servers one at a time, but the role returns as soon as the service manager reports k3s started, which is before the node serves again. On an embedded etcd cluster the next restart can then land while the previous member is still a learner, and a learner does not count towards the quorum. k3s_wait_ready holds each node at the end of its role until the API server reports it Ready. k3s_server_wait_etcd_voters holds each server until every server the run has already started carries EtcdIsVoter=True. Both default to false, are independent of each other, are skipped in check mode, and the upgrade role honors them as well. The new voter gate and the existing join barrier both wait only for the servers the run has reached, taken from the play's host list up to the end of the current batch: a whole-group count never completes on a fresh HA install rolled one node at a time. They match nodes by name rather than by count, so a re-run against a cluster that already holds more nodes also completes, and the comparison ignores case. k3s_node_name carries the name K3s registers the node under. It defaults to the lower-cased hostname and has to be set when a node is installed with --node-name. The HA integration job installs through tests/serial.yml, which imports site.yml with the serial variables in its vars, and the external job gives its server a node name that differs from its host name. Signed-off-by: Kristofer <kristofer@atlascloud.is> * docs: say why the node name is recorded as a fact A role default is not reachable through hostvars, so the barriers read a fact that each host sets for itself. Signed-off-by: Kristofer <kristofer@atlascloud.is> * docs: describe what upgrade.yml does rather than that it is unchanged Signed-off-by: Kristofer <kristofer@atlascloud.is> * Tolerate a host that left the play when listing started servers Extracting a key from hostvars raises when the key is missing. A host that went unreachable earlier in the play never recorded its node name, so the list comprehension that both barriers read failed the whole batch rather than the one host that dropped out. Select the hosts that carry the fact instead. Signed-off-by: Kristofer <kristofer@atlascloud.is> * Count the servers started so far in the server join barrier The barrier that confirms the servers joined matched the node names the run had recorded against the names the API server returned. Counting is enough for what the barrier is for, and it keeps the check independent of the node name: at the default serial it waits for the whole server group, which is what it waited for before, and under a serialised run it waits for the servers the run has started. Name matching stays in the two opt-in gates, which need it. Signed-off-by: Kristofer <kristofer@atlascloud.is> * Take the node name default from the OS nodename The kubelet registers a node under the name the operating system reports, which on a host with a search domain is the fully qualified name, not the short host name the default used. The server and agent plays gather no facts of their own, so the default also falls back to the inventory host name rather than requiring a gathered fact. Signed-off-by: Kristofer <kristofer@atlascloud.is> * Show the node name as a per-host setting in the sample inventory k3s_node_name names one node, so the sample carries it under a host entry rather than in the group vars every node reads. The sample also says where the serial variables are set, since a reader who meets them in the group vars block would reasonably try to set them there. Signed-off-by: Kristofer <kristofer@atlascloud.is> * State what the etcd voter gate checks and how to read a gate timeout The voter gate was described as the thing that keeps a rolling run from taking the quorum down, which overstates it: it checks one condition on the servers the run has started, and a quorum lost some other way is outside what it sees. Say that, say which check counts nodes and which match by name, and give the reader the two failure modes worth knowing before a gate times out on them. Signed-off-by: Kristofer <kristofer@atlascloud.is> * Install the HA test cluster at the default serial and give it an agent The HA job installed through tests/serial.yml, which left no coverage of a fresh multi-server install at the default serial, the setting a cluster that passes no serial variable actually runs. Put that back as the install step and move tests/serial.yml to the version-bump re-run, where rolling one node at a time is what the step is testing. The inventory had no agent, so the agent play never batched and the agent readiness gate never ran. Adding one covers both, and the job's container loops and assertions follow it. Signed-off-by: Kristofer <kristofer@atlascloud.is> * feat: roll the servers one at a time with an etcd voter wait k3s_server_serial sets the serial of the server play. It defaults to 100%, the whole group in one batch, so a run that does not set it behaves as before. Set it to 1 to restart the servers one at a time, and pass it through extra vars or import_playbook vars, since serial is templated before any inventory variable is bound. The join check waits for the servers started so far rather than the whole group, so a fresh HA install rolled one server at a time completes. A multi-server cluster on embedded etcd waits until every server reports EtcdIsVoter=True before the next batch starts. The agent play is unchanged. The HA integration job re-runs site.yml with k3s_server_serial=1 in place of --forks=1. Signed-off-by: Kristofer <kristofer@atlascloud.is> * docs: clarify the serial server rollout in the README Signed-off-by: Kristofer <kristofer@atlascloud.is> * docs: describe the etcd voter wait precisely Signed-off-by: Kristofer <kristofer@atlascloud.is> --------- Signed-off-by: Kristofer <kristofer@atlascloud.is>
253 lines
9.4 KiB
Markdown
253 lines
9.4 KiB
Markdown
# Build a Kubernetes cluster using K3s via Ansible
|
|
|
|
Author: <https://github.com/itwars>
|
|
Current Maintainer: <https://github.com/dereknola>
|
|
|
|
Easily bring up a cluster on machines running:
|
|
|
|
- [X] Debian
|
|
- [X] Ubuntu
|
|
- [X] Raspberry Pi OS
|
|
- [X] RHEL Family (CentOS, Redhat, Rocky Linux, Oracle Linux...)
|
|
- [X] SUSE Family (SLES, OpenSUSE Leap, Tumbleweed...)
|
|
- [X] ArchLinux
|
|
|
|
on processor architectures:
|
|
|
|
- [X] x64
|
|
- [X] arm64
|
|
- [X] armhf
|
|
|
|
## System requirements
|
|
|
|
The control node **must** have Ansible 8.0+ (ansible-core 2.15+) and the Python `netaddr` library (`pip install netaddr`), required by the `ansible.utils.ipwrap` filter for IPv6 `api_endpoint` support.
|
|
|
|
All managed nodes in inventory must have:
|
|
- Passwordless SSH access
|
|
- Root access (or a user with equivalent permissions)
|
|
|
|
It is also recommended that all managed nodes disable firewalls and swap. See [K3s Requirements](https://docs.k3s.io/installation/requirements) for more information.
|
|
|
|
## Installation
|
|
|
|
### With ansible-galaxy
|
|
|
|
`k3s-ansible` is an Ansible collection and can be installed with the `ansible-galaxy` command:
|
|
|
|
```console
|
|
$ ansible-galaxy collection install git+https://github.com/k3s-io/k3s-ansible.git
|
|
```
|
|
|
|
Alternatively, add it to the [`requirements.yaml` file of your Ansible project](https://docs.ansible.com/ansible/latest/collections_guide/collections_installing.html#install-multiple-collections-with-a-requirements-file) as follows:
|
|
|
|
```yaml
|
|
collections:
|
|
- name: https://github.com/k3s-io/k3s-ansible.git
|
|
type: git
|
|
version: <comit-ish>
|
|
```
|
|
|
|
As a comit-ish, you can use a branch, a version tag, or a specific commit.
|
|
|
|
### From source
|
|
|
|
Alternatively to an installation with `ansible-galaxy`, the `k3s-ansible` repository can simply be cloned from github:
|
|
|
|
```console
|
|
$ git clone https://github.com/k3s-io/k3s-ansible.git
|
|
$ cd k3s-ansible
|
|
```
|
|
|
|
## Usage
|
|
|
|
First copy the sample inventory to `inventory.yml`.
|
|
|
|
```bash
|
|
cp inventory-sample.yml inventory.yml
|
|
```
|
|
|
|
If you have installed `k3s-ansible` with ansible-galaxy, you can grab the [inventory-sample.yml](./inventory-sample.yml) from github.
|
|
|
|
Second edit the inventory file to match your cluster setup. For example:
|
|
```bash
|
|
k3s_cluster:
|
|
children:
|
|
server:
|
|
hosts:
|
|
192.16.35.11:
|
|
agent:
|
|
hosts:
|
|
192.16.35.12:
|
|
192.16.35.13:
|
|
```
|
|
|
|
If needed, you can also edit `vars` section at the bottom to match your environment.
|
|
|
|
If multiple hosts are in the server group the playbook will automatically setup k3s in HA mode with embedded etcd.
|
|
An odd number of server nodes is required (3,5,7). Read the [official documentation](https://docs.k3s.io/datastore/ha-embedded) for more information.
|
|
|
|
Setting up a loadbalancer or VIP beforehand to use as the API endpoint is possible but not covered here.
|
|
|
|
|
|
Start provisioning of the cluster using one of the following commands. The command to be used depends on whether you installed `k3s-ansible` with `ansible-galaxy` or if you run the playbook from within the cloned git repository:
|
|
|
|
*Installed with ansible-galaxy*
|
|
|
|
```bash
|
|
ansible-playbook k3s.orchestration.site -i inventory.yml
|
|
```
|
|
|
|
*Running the playbook from inside the repository*
|
|
|
|
```bash
|
|
ansible-playbook playbooks/site.yml -i inventory.yml
|
|
```
|
|
|
|
Alternatively, to run the playbook from your existing project setup, run the playbook from within your own playbook:
|
|
|
|
*Installed with ansible-galaxy*
|
|
|
|
```yaml
|
|
- name: Import kube cluster playbook
|
|
ansible.builtin.import_playbook: k3s.orchestration.site
|
|
```
|
|
|
|
*Running the playbook from inside the repository*
|
|
|
|
```yaml
|
|
- name: Import kube cluster playbook
|
|
ansible.builtin.import_playbook: k3s-ansible/playbooks/site.yml
|
|
```
|
|
|
|
|
|
### Using an external database
|
|
|
|
If an external database is preferred, this can be achieved by passing the `--datastore-endpoint` as an extra server argument as well as setting the `use_external_database` flag to true.
|
|
|
|
```bash
|
|
k3s_cluster:
|
|
children:
|
|
server:
|
|
hosts:
|
|
192.16.35.11:
|
|
192.16.35.12:
|
|
agent:
|
|
hosts:
|
|
192.16.35.13:
|
|
|
|
vars:
|
|
use_external_database: true
|
|
extra_server_args: "--datastore-endpoint=postgres://username:password@hostname:port/database-name"
|
|
```
|
|
|
|
The `use_external_database` flag is required when more than one server is defined, as otherwise an embedded etcd cluster will be created instead.
|
|
|
|
The format of the datastore-endpoint parameter is dependent upon the datastore backend, please visit the [K3s datastore endpoint format](https://docs.k3s.io/datastore#datastore-endpoint-format-and-functionality) for details on the format and supported datastores.
|
|
|
|
### Server config file permissions
|
|
|
|
The server role writes `/etc/rancher/k3s/config.yaml` at mode `0644`. That file can contain the cluster token, so any local account on a server node can read the credential that joins a node to the cluster. Set `k3s_server_config_mode` to narrow it:
|
|
|
|
```yaml
|
|
k3s_server_config_mode: "0640"
|
|
```
|
|
|
|
K3s reads the file as root, so `0600` is the safe floor: no other account needs the file. The default stays `0644`, which is the mode the file already has on a running cluster, so a re-run changes nothing until the variable is set. The upgrade role rewrites the same file and honors the same variable.
|
|
|
|
## Upgrading
|
|
|
|
A playbook is provided to upgrade K3s on all nodes in the cluster. To use it, update `k3s_version` with the desired version in `inventory.yml` and run one of the following commands. Again, the syntax is slightly different depending on whether you installed `k3s-ansible` with `ansible-galaxy` or if you run the playbook from within the cloned git repository:
|
|
|
|
|
|
*Installed with ansible-galaxy*
|
|
|
|
```bash
|
|
ansible-playbook k3s.orchestration.upgrade -i inventory.yml
|
|
```
|
|
|
|
*Running the playbook from inside the repository*
|
|
|
|
```bash
|
|
ansible-playbook playbooks/upgrade.yml -i inventory.yml
|
|
```
|
|
|
|
Re-running the `site.yml` playbook after bumping `k3s_version` performs the same upgrade declaratively: it restarts the k3s services so the cluster picks up the new runtime. On a multi-server (HA) cluster, add `-e k3s_server_serial=1` so the servers restart one at a time and etcd keeps quorum. With embedded etcd, the next server restarts only after the restarted ones report as etcd voters. By default all servers restart together. Set it through extra vars or `import_playbook` vars, not inventory: `serial` is a play keyword, so inventory variables don't apply to it.
|
|
|
|
```bash
|
|
ansible-playbook playbooks/site.yml -i inventory.yml -e k3s_server_serial=1
|
|
```
|
|
|
|
The dedicated `upgrade.yml` playbook remains available and unchanged.
|
|
|
|
## Airgap Install
|
|
|
|
Airgap installation is supported via the `airgap_dir` variable. This variable should be set to the path of a directory containing the K3s binary and images. The release artifacts can be downloaded from the [K3s Releases](https://github.com/k3s-io/k3s/releases). You must download the appropriate images for you architecture (any of the compression formats will work). Additionally, you must run the `airgap` role to set up the airgapped environment.
|
|
|
|
An example folder for an x86_64 cluster:
|
|
```bash
|
|
$ ls ./playbooks/my-airgap/
|
|
total 248M
|
|
-rwxr-xr-x 1 $USER $USER 58M Nov 14 11:28 k3s
|
|
-rw-r--r-- 1 $USER $USER 190M Nov 14 11:30 k3s-airgap-images-amd64.tar.gz
|
|
|
|
$ cat inventory.yml
|
|
...
|
|
airgap_dir: ./my-airgap # Paths are relative to the playbooks directory
|
|
```
|
|
|
|
Additionally, if deploying on an OS with SELinux, you will also need to download the latest [k3s-selinux RPM](https://github.com/k3s-io/k3s-selinux/releases/latest) and its dependencies `selinux-policy` and `container-selinux` RPMs and place them in the airgap folder.
|
|
|
|
|
|
It is assumed that the control node has access to the internet. The playbook will automatically download the k3s install script on the control node, and then distribute all three artifacts to the managed nodes.
|
|
|
|
## Kubeconfig
|
|
|
|
After successful bringup, the kubeconfig of the cluster is copied to the control node and merged with `~/.kube/config` under the `k3s-ansible` context unless overwritten by the `cluster_context` variable.
|
|
Assuming you have [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) installed, you can confirm access to your **Kubernetes** cluster with the following:
|
|
|
|
```bash
|
|
kubectl config use-context k3s-ansible
|
|
kubectl get nodes
|
|
```
|
|
|
|
If you wish for your kubeconfig to be copied elsewhere and not merged, you can set the `kubeconfig` variable in `inventory.yml` to the desired path.
|
|
|
|
If you wish to get a new copy of the kubeconfig after installation you can run the site playbook again using the `kubeconfig` tag.
|
|
|
|
*Installed with ansible-galaxy*
|
|
|
|
```bash
|
|
ansible-playbook k3s.orchestration.site -i inventory.yml --tags kubeconfig
|
|
```
|
|
|
|
*Running the playbook from inside the repository*
|
|
|
|
```bash
|
|
ansible-playbook playbooks/site.yml -i inventory.yml --tags kubeconfig
|
|
```
|
|
## Local Testing
|
|
|
|
A Vagrantfile is provided that provision a 5 nodes cluster using Vagrant (LibVirt or Virtualbox as provider). To use it:
|
|
|
|
```bash
|
|
vagrant up
|
|
```
|
|
|
|
By default, each node is given 2 cores and 2GB of RAM and runs Ubuntu 24.04. You can customize these settings by editing the `Vagrantfile`.
|
|
|
|
## Need More Features?
|
|
|
|
This project is intended to provide a "vanilla" K3s install. If you need more features, such as:
|
|
- Private Registry
|
|
- Advanced Storage (Longhorn, Ceph, etc)
|
|
- External Database
|
|
- External Load Balancer or VIP
|
|
- Alternative CNIs
|
|
|
|
See these other projects:
|
|
- https://github.com/PyratLabs/ansible-role-k3s
|
|
- https://github.com/techno-tim/k3s-ansible
|
|
- https://github.com/jon-stumpf/k3s-ansible
|
|
- https://github.com/alexellis/k3sup
|
|
- https://github.com/axivo/k3s-cluster
|