* feat: allow the server and agent plays to roll one node at a time
The server and agent roles restart their k3s service on every run, and
the plays run against the whole group at once. On a cluster with three
or more servers that restarts every etcd member together, which costs
the cluster its quorum and its API server.
k3s_server_serial and k3s_agent_serial set the serial of the two plays.
Both default to 100%, the whole group in one batch, so a run that does
not set them behaves as before. The HA integration test now rolls the
servers with the variable instead of --forks=1, which serialises the
role rather than the individual tasks.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* feat: hold each node until it serves again during a rolling run
Setting k3s_server_serial to 1 rolls the servers one at a time, but the role
returns as soon as the service manager reports k3s started, which is before the
node serves again. On an embedded etcd cluster the next restart can then land
while the previous member is still a learner, and a learner does not count
towards the quorum.
k3s_wait_ready holds each node at the end of its role until the API server
reports it Ready. k3s_server_wait_etcd_voters holds each server until every
server the run has already started carries EtcdIsVoter=True. Both default to
false, are independent of each other, are skipped in check mode, and the upgrade
role honors them as well.
The new voter gate and the existing join barrier both wait only for the servers
the run has reached, taken from the play's host list up to the end of the current
batch: a whole-group count never completes on a fresh HA install rolled one node
at a time. They match nodes by name rather than by count, so a re-run against a
cluster that already holds more nodes also completes, and the comparison ignores
case.
k3s_node_name carries the name K3s registers the node under. It defaults to the
lower-cased hostname and has to be set when a node is installed with --node-name.
The HA integration job installs through tests/serial.yml, which imports site.yml
with the serial variables in its vars, and the external job gives its server a
node name that differs from its host name.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* docs: say why the node name is recorded as a fact
A role default is not reachable through hostvars, so the barriers read a fact
that each host sets for itself.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* docs: describe what upgrade.yml does rather than that it is unchanged
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* Tolerate a host that left the play when listing started servers
Extracting a key from hostvars raises when the key is missing. A host that
went unreachable earlier in the play never recorded its node name, so the
list comprehension that both barriers read failed the whole batch rather
than the one host that dropped out. Select the hosts that carry the fact
instead.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* Count the servers started so far in the server join barrier
The barrier that confirms the servers joined matched the node names the
run had recorded against the names the API server returned. Counting is
enough for what the barrier is for, and it keeps the check independent of
the node name: at the default serial it waits for the whole server group,
which is what it waited for before, and under a serialised run it waits
for the servers the run has started. Name matching stays in the two
opt-in gates, which need it.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* Take the node name default from the OS nodename
The kubelet registers a node under the name the operating system reports,
which on a host with a search domain is the fully qualified name, not the
short host name the default used. The server and agent plays gather no
facts of their own, so the default also falls back to the inventory host
name rather than requiring a gathered fact.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* Show the node name as a per-host setting in the sample inventory
k3s_node_name names one node, so the sample carries it under a host entry
rather than in the group vars every node reads. The sample also says where
the serial variables are set, since a reader who meets them in the group
vars block would reasonably try to set them there.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* State what the etcd voter gate checks and how to read a gate timeout
The voter gate was described as the thing that keeps a rolling run from
taking the quorum down, which overstates it: it checks one condition on
the servers the run has started, and a quorum lost some other way is
outside what it sees. Say that, say which check counts nodes and which
match by name, and give the reader the two failure modes worth knowing
before a gate times out on them.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* Install the HA test cluster at the default serial and give it an agent
The HA job installed through tests/serial.yml, which left no coverage of a
fresh multi-server install at the default serial, the setting a cluster
that passes no serial variable actually runs. Put that back as the install
step and move tests/serial.yml to the version-bump re-run, where rolling
one node at a time is what the step is testing.
The inventory had no agent, so the agent play never batched and the agent
readiness gate never ran. Adding one covers both, and the job's container
loops and assertions follow it.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* feat: roll the servers one at a time with an etcd voter wait
k3s_server_serial sets the serial of the server play. It defaults to
100%, the whole group in one batch, so a run that does not set it behaves
as before. Set it to 1 to restart the servers one at a time, and pass it
through extra vars or import_playbook vars, since serial is templated
before any inventory variable is bound.
The join check waits for the servers started so far rather than the whole
group, so a fresh HA install rolled one server at a time completes. A
multi-server cluster on embedded etcd waits until every server reports
EtcdIsVoter=True before the next batch starts. The agent play is
unchanged.
The HA integration job re-runs site.yml with k3s_server_serial=1 in place
of --forks=1.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* docs: clarify the serial server rollout in the README
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* docs: describe the etcd voter wait precisely
Signed-off-by: Kristofer <kristofer@atlascloud.is>
---------
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* feat: make the server config file mode configurable
/etc/rancher/k3s/config.yaml holds the cluster token. The server and
upgrade roles write it at 0644, so every local account on a server node
can read the credential that joins a node to the cluster.
k3s_server_config_mode sets the mode of that file in both roles. It
defaults to 0644, so behaviour is unchanged unless it is set.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* docs: describe the server config file contents accurately
/etc/rancher/k3s/config.yaml does not always carry the cluster token. A
first server that lets K3s generate one writes the file without it, so the
file can contain the token rather than always holding it. Correct that
claim in the README, the inventory sample and the role default, and use
the US spelling of "honors".
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* Name the floor the config file mode can be narrowed to
Saying any mode root can read works leaves the reader to work out what
that means in practice. The file is read only by k3s, running as root, so
0600 is the tightest mode that still works.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* Run the basic integration job with a non-default config file mode
The templated mode was only ever rendered with its default in CI, which
is the one value that cannot tell a working template from a broken one.
The basic inventory sets 0640 and the job reads the mode back off the
installed file.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
---------
Signed-off-by: Kristofer <kristofer@atlascloud.is>
Re-running site.yml with a raised k3s_version replaced the on-disk
binary but left agents and joined servers running the old runtime,
because the install script skips the service start and the roles only
restarted the service on a config change. A version-only bump was
therefore not applied until the services were restarted by hand.
Always restart the k3s service in the server and agent roles on a
site.yml run, so the cluster reliably picks up a new config or runtime
without any logic to detect whether the binary changed. On a
multi-server cluster, run the playbook with --forks=1 so the servers
restart one at a time and the etcd quorum is preserved; document this
in the README.
Keep the dedicated upgrade.yml integration test and add a second
upgrade through site.yml, asserting the running version the kubelet
reports rather than the on-disk binary, so the tests fail if the
restart regresses.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Use rancher/systemd-node which doesn't cause k3s to crash
- Expand requirements for testing, ensure deployments become ready at each playbook
- Add logging on failure for debugging
Signed-off-by: Derek Nola <derek.nola@suse.com>
* refactor(ansible): replace deprecated with_items and ansible facts
Replace deprecated `with_items` with `loop` keyword across all roles.
Migrate from legacy `ansible_*` top-level facts to `ansible_facts['*']`
syntax to prepare for ansible-core 2.24 where INJECT_FACTS_AS_VARS
will default to False.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
* chore(ansible): sync minimum version requirements to 2.15
Align version checks in prereq and airgap roles with README requirement
of ansible-core 2.15+.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
---------
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Co-authored-by: Claude <noreply@anthropic.com>
* Auto-add --tls-san={{ api_endpoint }} when it differs from ansible_hostname
* Ensures first server generates certificate with all required SANs
* Add .ansible/ and PR_DESCRIPTION.md to gitignore
Signed-off-by: Guillaume Andre <mail@guillaumea.fr>
* Generate token
If a token is not explicitly provided, let the first server generate a
random one. Such a token is saved on the first server and the playbook
can retrieve it from there and store it a a fact. All other servers and
agents can use that token later to join the cluster. It will be saved
into their environment file as usual.
Signed-off-by: Marko Vukovic <8951449+anon-software@users.noreply.github.com>
* Document that token is (mostly) optional now
The token is still required when using Vagrant.
Signed-off-by: Marko Vukovic <8951449+anon-software@users.noreply.github.com>
* Prevent multiple tokens in k3s.service.env
If site.yml playbook is executed multiple times with different tokens,
they will all accumulate in k3s.service.env. They won't do any harm
because the last one wins, however it is a matter of good housekeeping
to delete the old before inserting a new one.
Signed-off-by: Marko Vukovic <8951449+anon-software@users.noreply.github.com>
* Selectively remove existing token from the environment file
If the existing token in the environment file is the same as the token
used for the playbook run, leave it in the file to avoid false changed
status from the task.
Signed-off-by: Marko Vukovic <8951449+anon-software@users.noreply.github.com>
---------
Signed-off-by: Marko Vukovic <8951449+anon-software@users.noreply.github.com>
* Security exposure related to the token
The installation playbook saves the token into the systemd unit
configuration file /etc/systemd/system/k3s.service. The problem is that
according to K3s' documentation "the server token should be guarded
carefully" (https://docs.k3s.io/cli/token), yet the configuration file
is readable by anybody. A better solution is to save the token into its
corresponding environment file /etc/systemd/system/k3s.service.env which
is readable by the super user only. This is what the standard K3s'
installation script (https://get.k3s.io) does.
Signed-off-by: Marko Vukovic <8951449+anon-software@users.noreply.github.com>
* Restore the server URL into systemd configuration file
There aren't any security implications in keeping it there.
Signed-off-by: Marko Vukovic <8951449+anon-software@users.noreply.github.com>
---------
Signed-off-by: Marko Vukovic <8951449+anon-software@users.noreply.github.com>
* Implement compatible yamllint, make octals explicit
Signed-off-by: Derek Nola <derek.nola@suse.com>
* Replace yum with dnf, yum is deprecated
Signed-off-by: Derek Nola <derek.nola@suse.com>
---------
Signed-off-by: Derek Nola <derek.nola@suse.com>
- [Agent : Download artefact only if needed](roles/k3s_agent/tasks/main.yml#L13)
- [Server : Download artefact only if needed](roles/k3s_server/tasks/main.yml#L13)
- [Upgrade : Upgrade node only if needed](roles/k3s_upgrade/tasks/main.yml#L14)
Linked issue #264 k3s_server and k3s_agent tasks are not idempotent
Signed-off-by: Loïc Dubard <loic97429@gmail.com>
As detailed in https://github.com/k3s-io/k3s-ansible/issues/295, this
commit fixes the issue that if `kubeconfig` is set to anything other
than the defaults value, then:
- `~/.kube/config` is modified.
- No file at `{{ kubeconfig }}` is created.
- Any existing file and `{{ kubeconfig }}` is deleted.
Signed-off-by: Nick To <nick@nickto.net>
* do not blindly overwrite kube config
Signed-off-by: Roman Ivanov <me@roivanov.com>
* don't need to check if an existing config exists
Co-authored-by: Derek Nola <derek.nola@suse.com>
Signed-off-by: Roman Ivanov <me@roivanov.com>
Mustafa
Kris
Stéphane Este-Gracias
Ian Lewis
Archy
Aleksei Sviridkin
Jonathan Cremin
Derek Nola
Jon S. Stumpf
ppascenteandPete Pascente
Phillip Thurston
too-gee
Guillaume A
Zero King
Michal Stanke
alankan-finocompandDerek Nola
WillandWill Brown
anon-software
Lihai Tu
Peter Klijn
Meagan HarrisandDerek Nola
Mykyta Orlov
LawiK974
Nick To
Dmitriy Safronov
Xiangkun LiuandXiangkun Liu
Dani Hodovic
Roman IvanovandDerek Nola