* feat: allow the server and agent plays to roll one node at a time
The server and agent roles restart their k3s service on every run, and
the plays run against the whole group at once. On a cluster with three
or more servers that restarts every etcd member together, which costs
the cluster its quorum and its API server.
k3s_server_serial and k3s_agent_serial set the serial of the two plays.
Both default to 100%, the whole group in one batch, so a run that does
not set them behaves as before. The HA integration test now rolls the
servers with the variable instead of --forks=1, which serialises the
role rather than the individual tasks.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* feat: hold each node until it serves again during a rolling run
Setting k3s_server_serial to 1 rolls the servers one at a time, but the role
returns as soon as the service manager reports k3s started, which is before the
node serves again. On an embedded etcd cluster the next restart can then land
while the previous member is still a learner, and a learner does not count
towards the quorum.
k3s_wait_ready holds each node at the end of its role until the API server
reports it Ready. k3s_server_wait_etcd_voters holds each server until every
server the run has already started carries EtcdIsVoter=True. Both default to
false, are independent of each other, are skipped in check mode, and the upgrade
role honors them as well.
The new voter gate and the existing join barrier both wait only for the servers
the run has reached, taken from the play's host list up to the end of the current
batch: a whole-group count never completes on a fresh HA install rolled one node
at a time. They match nodes by name rather than by count, so a re-run against a
cluster that already holds more nodes also completes, and the comparison ignores
case.
k3s_node_name carries the name K3s registers the node under. It defaults to the
lower-cased hostname and has to be set when a node is installed with --node-name.
The HA integration job installs through tests/serial.yml, which imports site.yml
with the serial variables in its vars, and the external job gives its server a
node name that differs from its host name.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* docs: say why the node name is recorded as a fact
A role default is not reachable through hostvars, so the barriers read a fact
that each host sets for itself.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* docs: describe what upgrade.yml does rather than that it is unchanged
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* Tolerate a host that left the play when listing started servers
Extracting a key from hostvars raises when the key is missing. A host that
went unreachable earlier in the play never recorded its node name, so the
list comprehension that both barriers read failed the whole batch rather
than the one host that dropped out. Select the hosts that carry the fact
instead.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* Count the servers started so far in the server join barrier
The barrier that confirms the servers joined matched the node names the
run had recorded against the names the API server returned. Counting is
enough for what the barrier is for, and it keeps the check independent of
the node name: at the default serial it waits for the whole server group,
which is what it waited for before, and under a serialised run it waits
for the servers the run has started. Name matching stays in the two
opt-in gates, which need it.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* Take the node name default from the OS nodename
The kubelet registers a node under the name the operating system reports,
which on a host with a search domain is the fully qualified name, not the
short host name the default used. The server and agent plays gather no
facts of their own, so the default also falls back to the inventory host
name rather than requiring a gathered fact.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* Show the node name as a per-host setting in the sample inventory
k3s_node_name names one node, so the sample carries it under a host entry
rather than in the group vars every node reads. The sample also says where
the serial variables are set, since a reader who meets them in the group
vars block would reasonably try to set them there.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* State what the etcd voter gate checks and how to read a gate timeout
The voter gate was described as the thing that keeps a rolling run from
taking the quorum down, which overstates it: it checks one condition on
the servers the run has started, and a quorum lost some other way is
outside what it sees. Say that, say which check counts nodes and which
match by name, and give the reader the two failure modes worth knowing
before a gate times out on them.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* Install the HA test cluster at the default serial and give it an agent
The HA job installed through tests/serial.yml, which left no coverage of a
fresh multi-server install at the default serial, the setting a cluster
that passes no serial variable actually runs. Put that back as the install
step and move tests/serial.yml to the version-bump re-run, where rolling
one node at a time is what the step is testing.
The inventory had no agent, so the agent play never batched and the agent
readiness gate never ran. Adding one covers both, and the job's container
loops and assertions follow it.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* feat: roll the servers one at a time with an etcd voter wait
k3s_server_serial sets the serial of the server play. It defaults to
100%, the whole group in one batch, so a run that does not set it behaves
as before. Set it to 1 to restart the servers one at a time, and pass it
through extra vars or import_playbook vars, since serial is templated
before any inventory variable is bound.
The join check waits for the servers started so far rather than the whole
group, so a fresh HA install rolled one server at a time completes. A
multi-server cluster on embedded etcd waits until every server reports
EtcdIsVoter=True before the next batch starts. The agent play is
unchanged.
The HA integration job re-runs site.yml with k3s_server_serial=1 in place
of --forks=1.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* docs: clarify the serial server rollout in the README
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* docs: describe the etcd voter wait precisely
Signed-off-by: Kristofer <kristofer@atlascloud.is>
---------
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* feat: make the server config file mode configurable
/etc/rancher/k3s/config.yaml holds the cluster token. The server and
upgrade roles write it at 0644, so every local account on a server node
can read the credential that joins a node to the cluster.
k3s_server_config_mode sets the mode of that file in both roles. It
defaults to 0644, so behaviour is unchanged unless it is set.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* docs: describe the server config file contents accurately
/etc/rancher/k3s/config.yaml does not always carry the cluster token. A
first server that lets K3s generate one writes the file without it, so the
file can contain the token rather than always holding it. Correct that
claim in the README, the inventory sample and the role default, and use
the US spelling of "honors".
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* Name the floor the config file mode can be narrowed to
Saying any mode root can read works leaves the reader to work out what
that means in practice. The file is read only by k3s, running as root, so
0600 is the tightest mode that still works.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
* Run the basic integration job with a non-default config file mode
The templated mode was only ever rendered with its default in CI, which
is the one value that cannot tell a working template from a broken one.
The basic inventory sets 0640 and the job reads the mode back off the
installed file.
Signed-off-by: Kristofer <kristofer@atlascloud.is>
---------
Signed-off-by: Kristofer <kristofer@atlascloud.is>
Re-running site.yml with a raised k3s_version replaced the on-disk
binary but left agents and joined servers running the old runtime,
because the install script skips the service start and the roles only
restarted the service on a config change. A version-only bump was
therefore not applied until the services were restarted by hand.
Always restart the k3s service in the server and agent roles on a
site.yml run, so the cluster reliably picks up a new config or runtime
without any logic to detect whether the binary changed. On a
multi-server cluster, run the playbook with --forks=1 so the servers
restart one at a time and the etcd quorum is preserved; document this
in the README.
Keep the dedicated upgrade.yml integration test and add a second
upgrade through site.yml, asserting the running version the kubelet
reports rather than the on-disk binary, so the tests fail if the
restart regresses.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Use rancher/systemd-node which doesn't cause k3s to crash
- Expand requirements for testing, ensure deployments become ready at each playbook
- Add logging on failure for debugging
Signed-off-by: Derek Nola <derek.nola@suse.com>
Kris
dependabot[bot]
Derek Nola
Stéphane Este-Gracias
Aleksei Sviridkin
Chris Wayne
Brian Lu
Julien DOCHE
Jeff Geerling