feat: roll the servers one at a time and wait for each to serve again (#562)

* feat: allow the server and agent plays to roll one node at a time

The server and agent roles restart their k3s service on every run, and
the plays run against the whole group at once. On a cluster with three
or more servers that restarts every etcd member together, which costs
the cluster its quorum and its API server.

k3s_server_serial and k3s_agent_serial set the serial of the two plays.
Both default to 100%, the whole group in one batch, so a run that does
not set them behaves as before. The HA integration test now rolls the
servers with the variable instead of --forks=1, which serialises the
role rather than the individual tasks.

Signed-off-by: Kristofer <kristofer@atlascloud.is>

* feat: hold each node until it serves again during a rolling run

Setting k3s_server_serial to 1 rolls the servers one at a time, but the role
returns as soon as the service manager reports k3s started, which is before the
node serves again. On an embedded etcd cluster the next restart can then land
while the previous member is still a learner, and a learner does not count
towards the quorum.

k3s_wait_ready holds each node at the end of its role until the API server
reports it Ready. k3s_server_wait_etcd_voters holds each server until every
server the run has already started carries EtcdIsVoter=True. Both default to
false, are independent of each other, are skipped in check mode, and the upgrade
role honors them as well.

The new voter gate and the existing join barrier both wait only for the servers
the run has reached, taken from the play's host list up to the end of the current
batch: a whole-group count never completes on a fresh HA install rolled one node
at a time. They match nodes by name rather than by count, so a re-run against a
cluster that already holds more nodes also completes, and the comparison ignores
case.

k3s_node_name carries the name K3s registers the node under. It defaults to the
lower-cased hostname and has to be set when a node is installed with --node-name.

The HA integration job installs through tests/serial.yml, which imports site.yml
with the serial variables in its vars, and the external job gives its server a
node name that differs from its host name.

Signed-off-by: Kristofer <kristofer@atlascloud.is>

* docs: say why the node name is recorded as a fact

A role default is not reachable through hostvars, so the barriers read a fact
that each host sets for itself.

Signed-off-by: Kristofer <kristofer@atlascloud.is>

* docs: describe what upgrade.yml does rather than that it is unchanged

Signed-off-by: Kristofer <kristofer@atlascloud.is>

* Tolerate a host that left the play when listing started servers

Extracting a key from hostvars raises when the key is missing. A host that
went unreachable earlier in the play never recorded its node name, so the
list comprehension that both barriers read failed the whole batch rather
than the one host that dropped out. Select the hosts that carry the fact
instead.

Signed-off-by: Kristofer <kristofer@atlascloud.is>

* Count the servers started so far in the server join barrier

The barrier that confirms the servers joined matched the node names the
run had recorded against the names the API server returned. Counting is
enough for what the barrier is for, and it keeps the check independent of
the node name: at the default serial it waits for the whole server group,
which is what it waited for before, and under a serialised run it waits
for the servers the run has started. Name matching stays in the two
opt-in gates, which need it.

Signed-off-by: Kristofer <kristofer@atlascloud.is>

* Take the node name default from the OS nodename

The kubelet registers a node under the name the operating system reports,
which on a host with a search domain is the fully qualified name, not the
short host name the default used. The server and agent plays gather no
facts of their own, so the default also falls back to the inventory host
name rather than requiring a gathered fact.

Signed-off-by: Kristofer <kristofer@atlascloud.is>

* Show the node name as a per-host setting in the sample inventory

k3s_node_name names one node, so the sample carries it under a host entry
rather than in the group vars every node reads. The sample also says where
the serial variables are set, since a reader who meets them in the group
vars block would reasonably try to set them there.

Signed-off-by: Kristofer <kristofer@atlascloud.is>

* State what the etcd voter gate checks and how to read a gate timeout

The voter gate was described as the thing that keeps a rolling run from
taking the quorum down, which overstates it: it checks one condition on
the servers the run has started, and a quorum lost some other way is
outside what it sees. Say that, say which check counts nodes and which
match by name, and give the reader the two failure modes worth knowing
before a gate times out on them.

Signed-off-by: Kristofer <kristofer@atlascloud.is>

* Install the HA test cluster at the default serial and give it an agent

The HA job installed through tests/serial.yml, which left no coverage of a
fresh multi-server install at the default serial, the setting a cluster
that passes no serial variable actually runs. Put that back as the install
step and move tests/serial.yml to the version-bump re-run, where rolling
one node at a time is what the step is testing.

The inventory had no agent, so the agent play never batched and the agent
readiness gate never ran. Adding one covers both, and the job's container
loops and assertions follow it.

Signed-off-by: Kristofer <kristofer@atlascloud.is>

* feat: roll the servers one at a time with an etcd voter wait

k3s_server_serial sets the serial of the server play. It defaults to
100%, the whole group in one batch, so a run that does not set it behaves
as before. Set it to 1 to restart the servers one at a time, and pass it
through extra vars or import_playbook vars, since serial is templated
before any inventory variable is bound.

The join check waits for the servers started so far rather than the whole
group, so a fresh HA install rolled one server at a time completes. A
multi-server cluster on embedded etcd waits until every server reports
EtcdIsVoter=True before the next batch starts. The agent play is
unchanged.

The HA integration job re-runs site.yml with k3s_server_serial=1 in place
of --forks=1.

Signed-off-by: Kristofer <kristofer@atlascloud.is>

* docs: clarify the serial server rollout in the README

Signed-off-by: Kristofer <kristofer@atlascloud.is>

* docs: describe the etcd voter wait precisely

Signed-off-by: Kristofer <kristofer@atlascloud.is>

---------

Signed-off-by: Kristofer <kristofer@atlascloud.is>

.github/workflows/integration.yml

@@ -386,14 +386,14 @@ jobs:
           exit 1
 
       # Re-running site.yml with a bumped version must upgrade every etcd
-      # server. --forks=1 makes Ansible restart them one at a time so the etcd
+      # server. k3s_server_serial=1 restarts them one at a time so the etcd
       # quorum is never lost.
       - name: Modify the k3s_version in inventory for a site.yml upgrade
         run: |
           sed -i 's/k3s_version: v1.34.1+k3s1/k3s_version: v1.34.3+k3s1/' tests/ha.yml
 
       - name: Run site Playbook to upgrade the servers via a version bump
-        run: ansible-playbook playbooks/site.yml -i tests/ha.yml --forks=1
+        run: ansible-playbook playbooks/site.yml -i tests/ha.yml -e k3s_server_serial=1
 
       # Assert the running (kubelet-reported) version, not the on-disk binary:
       # without a rolling restart the joined servers stay on the old runtime.

README.md

@@ -171,10 +171,10 @@ ansible-playbook k3s.orchestration.upgrade -i inventory.yml
 ansible-playbook playbooks/upgrade.yml -i inventory.yml
 ```
 
-Re-running the `site.yml` playbook after bumping `k3s_version` performs the same upgrade declaratively: it restarts the k3s services so the cluster picks up the new runtime. On a multi-server (HA) cluster, add `--forks=1` so Ansible restarts the servers one at a time and the etcd quorum is never lost:
+Re-running the `site.yml` playbook after bumping `k3s_version` performs the same upgrade declaratively: it restarts the k3s services so the cluster picks up the new runtime. On a multi-server (HA) cluster, add `-e k3s_server_serial=1` so the servers restart one at a time and etcd keeps quorum. With embedded etcd, the next server restarts only after the restarted ones report as etcd voters. By default all servers restart together. Set it through extra vars or `import_playbook` vars, not inventory: `serial` is a play keyword, so inventory variables don't apply to it.
 
 ```bash
-ansible-playbook playbooks/site.yml -i inventory.yml --forks=1
+ansible-playbook playbooks/site.yml -i inventory.yml -e k3s_server_serial=1
 ```
 
 The dedicated `upgrade.yml` playbook remains available and unchanged.

playbooks/site.yml

@@ -12,6 +12,7 @@
   hosts: server
   gather_facts: false
   become: true
+  serial: "{{ k3s_server_serial | default('100%') }}"
   roles:
     - role: k3s_server
 

roles/k3s_server/tasks/main.yml

@@ -10,6 +10,11 @@
   ansible.builtin.set_fact:
     k3s_server_installed_version: "{{ k3s_server_version_output.stdout_lines[0].split(' ')[2] }}"
 
+# Limit join checks to completed and current batches during serial installation.
+- name: Count the servers this play has already started
+  ansible.builtin.set_fact:
+    k3s_server_started_count: "{{ ansible_play_hosts_all.index(ansible_play_batch[-1]) + 1 }}"
+
 - name: Construct K3s server config
   block:
     # Start with an empty config
@@ -299,8 +304,7 @@
     # Always restart a joined server so a re-run of site.yml — whether it only
     # bumped k3s_version or changed config — brings it onto the new runtime, the
     # same way the first server above always restarts. On a multi-server cluster
-    # run the playbook with --forks=1 so the servers restart one at a time and
-    # the etcd quorum is preserved (see README).
+    # set k3s_server_serial to 1 so the servers restart one at a time (see README).
     - name: Restart K3s service
       when:
         - ansible_facts.services['k3s.service'] is defined
@@ -316,13 +320,15 @@
         state: started
         enabled: true
 
-    - name: Verify that all server nodes joined
-      when: not ansible_check_mode and (groups[server_group] | length) > 1
+    - name: Verify that the servers started so far joined
+      when: not ansible_check_mode
       ansible.builtin.command:
         cmd: >
           k3s kubectl get nodes -l "node-role.kubernetes.io/control-plane=true" -o=jsonpath="{.items[*].metadata.name}"
       register: k3s_server_nodes
-      until: k3s_server_nodes.rc == 0 and (k3s_server_nodes.stdout.split() | length) == (groups[server_group] | length)
+      until:
+        - k3s_server_nodes.rc == 0
+        - (k3s_server_nodes.stdout.split() | length) >= (k3s_server_started_count | int)
       retries: 20
       delay: 10
       changed_when: false
@@ -335,6 +341,24 @@
   run_once: true
   changed_when: false
 
+- name: Wait for the servers started so far to be etcd voters
+  when:
+    - not ansible_check_mode
+    - (groups[server_group] | length) > 1
+    - not use_external_database
+  ansible.builtin.command:
+    cmd: >-
+      k3s kubectl --request-timeout=10s get nodes -l "node-role.kubernetes.io/etcd=true"
+      -o=jsonpath='{range .items[*]}{.metadata.name}={.status.conditions[?(@.type=="EtcdIsVoter")].status}{"\n"}{end}'
+  register: k3s_server_etcd_voters
+  until:
+    - k3s_server_etcd_voters.rc == 0
+    - (k3s_server_etcd_voters.stdout.split() | length) >= (k3s_server_started_count | int)
+    - (k3s_server_etcd_voters.stdout.split() | reject('search', '=True$') | list | length) == 0
+  retries: 30
+  delay: 10
+  changed_when: false
+
 - name: Setup kubectl for user
   when: user_kubectl
   block: