Adding an E2E Test for a New Apply Phase

The cluster suite in test/e2e/cluster walks the apply phase list one phase at a time against a live bootloose cluster and asserts what each phase left on the hosts. Adding a phase to pkg/action/apply.go without adding a test here leaves a gap that nothing else covers, so this page is the checklist for closing it.

One top-level test, TestClusterPhases, runs five walks against the same cluster in the order they work in: dryrun runs whole actions with --dry-run over nodes with nothing on them and asserts it left them that way, apply installs the distro, join starts one more machine and brings it into the running cluster, upgrade walks the same phases again with a newer package, and reset takes the distro back off. A new phase needs a test in the apply walk, and usually one in the join and upgrade walks too – see Adding the join half and Adding the upgrade half.

For why the suite is built this way, see choice-phase-e2e-tests. For running the e2e suites generally, see e2e-tests.

One number per phase

A phase is identified by its source file’s number, and that number is used twice:

• The test file name mirrors the phase’s source file. A test for pkg/phase/25_modify_hosts_file.go goes in test/e2e/cluster/25_modify_hosts_file_test.go. Same number, same name, one file each.
• The method name carries the same number. That file contains Test_25_ModifyHosts.

Testify runs suite methods in lexicographic order of the method name, so the number is also the order the phases run in here. The numbers are zero-padded to two digits, which makes lexicographic order numeric order.

That order matches apply’s everywhere except the lock. Apply takes the lock third, right after OS detection, and holds it for the whole run; phase/91_lock.go’s number puts it after the install instead, with phase/92_unlock.go right behind it. The lock test still asserts what the phase writes on each host. What it no longer does is hold the lock across the phases in between, so a phase that misbehaves while the cluster is locked is not something this suite would see.

Steps that are not phases – creating the package, prepare, the health check – have no phase number to take, so they use the two ends of the ordering: Test_00 and Test_01 sort before every phase, and Test_ZZ1 onwards sort after every phase, because a letter sorts after a digit. They live in cluster_lifecycle_test.go, alongside ResetSuite, which is the whole of the reset walk. Those steps run whole actions rather than single phases – distro.Create, action.NewPrepare, action.NewApply, action.NewReset, action.NewKubeConfig – each with its own manager, and are given e2e.ClusterConfigPath, the engine-only inventory, rather than the full one the harness is built from; see the section on the upload-only host below.

Adding a phase

Take the new phase’s file number for both the test file and the method. Nothing else is renumbered: inserting phase/23_something.go gives you 23_something_test.go containing Test_23_Something, and it lands between Test_22_PrepareFapolicy and Test_25_ModifyHosts on its own.

If a phase is renumbered in pkg/phase, rename its test file and its method to match, and check that its new number still puts it after everything it depends on – the number is the ordering, so a phase that moves in the source moves here too.

Writing the test

Start from the phase’s source file and ask what an operator would look at to confirm it did its job. That is the assertion. A phase returning nil is not evidence of anything – read the host back.

A minimal test:

// Test_NN_YourPhase covers phase/NN_your_phase.go. <One or two sentences on what the phase
// does and, therefore, what artifact this asserts.>
func (s *ApplyPhaseSuite) Test_NN_YourPhase() {
	s.runPhase(&phase.YourPhase{Distro: s.harness.distro})

	for _, host := range s.harness.hosts() {
		s.Require().Truef(host.FileExist(someArtifactPath),
			"%s: nothing at %s", host, someArtifactPath)
	}
}

s.runPhase runs the phase through the shared manager and fails the test if it errors. Do not call s.harness.run directly unless you need the error itself.

Use s.Require() rather than s.Assert(). These tests are ordered and stateful, so continuing after a failed assertion produces noise about state the failed assertion already told you was wrong.

Each assertion body lives on phaseWalk, the embedded type every suite is built from, with a one-line Test_NN method on each suite calling it. Testify collects methods by their Test prefix, so the shared method is called, not run as a test of its own. Start a shared body with s.T().Helper().

What the harness gives you

phase_harness.go holds everything shared. The pieces a new test is likely to want:

For anything else, the hosts carry a live rig connection: host.ExecOutput, host.ReadFile, host.FileExist, and host.Configurer are all available and are how most assertions are written.

Phases that are expected to skip

Some phases legitimately do nothing on this cluster – SELinux where the runtime exposes none, fapolicyd when no host runs it, config sync when nothing has drifted. Do not write a test that asserts nothing; assert the gate instead. Compute what the hosts report, then require that ran(p) matches it:

var fapolicyd int
for _, host := range s.harness.hosts() {
	if host.Configurer.ServiceIsRunning(host, phase.FAPOLICYD) {
		fapolicyd++
	}
}
carriesRules := s.harness.manager.Distro.Spec.Config.OS.FAPolicyd != ""

p := &phase.PrepareFapolicy{}
s.runPhase(p)
s.Require().Equalf(fapolicyd > 0 && carriesRules, ran(p),
	"phase ran=%v with %d fapolicyd hosts and rules=%v", ran(p), fapolicyd, carriesRules)

if !ran(p) {
	return
}
// ... assert the artifact where it did fire

This tests the routing, which is the part most likely to break, and it keeps working when the cluster gains a host that does trip the gate. 21_prepare_selinux_test.go and 22_prepare_fapolicyd_test.go are the worked examples.

ran(p) is only meaningful after s.runPhase, because the manager calls Prepare first and most gates are computed there.

Phases that route on the OS family

The cluster runs multiple OS families on purpose. In the default full-walk inventory (k3sOS), all five machines join the cluster – one controller and four workers, split between Ubuntu (kc0, kw0, kw1) and Fedora (kwf0, kwf1). The stage-only inventory (stageOS) is not that set plus a machine: it is a different five, one per family per role – an Ubuntu controller (kc0), a Fedora controller (kcf0), an Ubuntu worker (kw0), a Fedora worker (kwf0), and an Alpine machine (kwa0) that receives uploads but never joins the cluster, covered in the section on the upload-only host below. So a Fedora controller only exists in the stage-only walk, and k3sOS has no Alpine host at all. A phase that treats Enterprise Linux differently from Debian must be tested on both sides, not just on the side it claims.

Derive the host set from the same filters the phase uses, so the test cannot disagree with it:

enterprise := s.harness.hosts().Filter(utils.FilterEnterpriseLinux)
debian := s.harness.hosts().Filter(utils.FilterDebianLinux)

Then assert the split: the hosts the phase claims got what it promises, the hosts it does not came back unchanged. 57_rpm_install_test.go and 58_apt_install_test.go are the worked examples.

Require the set you are testing to be non-empty. If someone collapses the cluster to one image, that turns a test which silently stopped covering anything into a failure that says so.

The upload-only host, and the boundary at Test_60

Alpine is neither Enterprise Linux nor Debian, so both of the filters above decline it and it falls through to the BIN upload phase. That is why it is in the inventory: without it, 59_bin_install.go is only ever exercised as the path every host happens to take, never as the fallback it is written to be.

It cannot run the engine, which links against glibc. So Test_60_ConfigureEngine calls s.harness.dropUploadOnlyHosts() before it does anything else, and from that point on the host is gone from the manager entirely, when there is one to drop. Phases up to and including Test_59 see every host in the inventory; everything after sees one fewer, if the inventory carried an upload-only host at all.

The default full-walk inventory (k3sOS) has no upload-only host, so that drop is a no-op there; the Alpine host, and the boundary it exercises, live in the smaller stage-only inventory (stageOS) instead. See clusterConfig in main_test.go.

This matters when adding a phase:

• An upload phase goes before the drop, and should assert against s.harness.uploadOnly() explicitly if it is meant to claim those hosts.
• Anything that installs, starts, configures or queries the engine goes after the drop, and needs nothing special – hosts() and workers() already exclude the dropped hosts by then.
• Do not “fix” a phase that fails on Alpine by skipping the host in the assertion. The phases select their own hosts in Prepare, with no OS gate in most cases, so the phase will still have run against it and spent its retry budget there. If a phase genuinely cannot run on a host, that host must not be on the manager when the phase runs.

s.harness.engineWorkers() exists for the one case where order is not obvious: a test that runs before the drop but is only meaningful for hosts that join.

The dry-run walk

DryRunSuite (dry_run_test.go) is the one walk that does not step through phases, and the one whose assertions are negative. It runs action.NewApply, action.NewPrepare and action.NewReset whole, with Manager.DryRun set, and then asserts that none of the artifacts the other walks look for arrived: no upload manifest, no sysctl file, no lock file, no running engine. It runs first because “the hosts are unchanged” is only checkable before the apply walk changes them.

Two things about it are easy to get wrong when extending it.

The hosts are disconnected by the time a run returns, because every action ends with the disconnect phase and its dry-run path disconnects as well. A read taken after the run reports every file as absent whether it is there or not, which would make each assertion pass over a host a dry run had written to. reconnectHosts runs Connect and DetectOS again – both read-only – and the assertions go after that.

The walk also asserts the positive half, which is what stops the negative half from being vacuous: the read-only phases have to have gathered their facts, so each host carries a configurer, a hostname, an architecture, and phase.UnknownVersion for the engine it is not running. A dry run that never connected to anything would satisfy an untouched-hosts check perfectly.

A new phase needs nothing added here unless it is read-only, in which case the facts it gathers belong in requireFactsGathered, or it writes an artifact worth naming, in which case its absence belongs in requireHostsUntouched next to the path the install test asserts exists.

The two reset cases are split by what they need. reset --dry-run against a cluster that is up is in ResetSuite as Test_0, immediately before the real reset, because that is the only point where a live cluster exists. reset --dry-run with no running controller is in this walk, because nodes nothing was installed on are the natural fixture for the nil-leader path PR #312 found against a live cluster.

Adding the join half

JoinPhaseSuite walks the same phase list a second time, against the cluster the apply walk installed and one machine larger. It carries the same package at the same version, so nothing routes to the upgrade phases and the only host with work left is the one that just appeared. It is the only coverage of apply against a cluster that already exists.

So a new phase usually needs a join test as well. Which kind depends on what it does the second time:

• The phase does the same thing either way – an upload phase, a config render, anything that does not read what is already running. Move the body to an unexported method on phaseWalk and give each suite a one-line Test_NN method calling it. 50_uploadfiles_test.go is the worked example.

• The phase behaves differently – anything gated on what the hosts report. Write two methods with the same name, one per suite, in the same file. 61_initialize_controller_test.go is the worked example: the apply walk requires ran(p), the join walk requires !ran(p) because the control plane is already up, and each says in its comment why.

// Test_NN_YourPhase, on the join walk, <what is different and why>.
func (s *JoinPhaseSuite) Test_NN_YourPhase() {
	p := &phase.YourPhase{Distro: s.harness.distro}
	s.runPhase(p)
	s.Require().False(ran(p), "<the routing this proves>")
}

Phases that assert over the whole matrix of hosts are worth a second look here rather than treating as repetition: Test_25 and Test_26 on the join walk are what prove the established nodes learned about the new one, not only that the new node learned about them.

The join machine is kwf2, a Fedora worker, provisioned on first use by requireJoinMachine. A Fedora worker is deliberate: the new node then comes through the SELinux, fapolicyd, firewalld and dnf branches on its own rather than inheriting anything the apply walk proved on the nodes beside it.

Adding the upgrade half

UpgradePhaseSuite walks the same phase list a third time, against the cluster the apply and join walks left running, with a package one patch release newer. It is what covers the code inside phase/66_upgrade_controller.go and phase/67_upgrade_worker.go: on a fresh install those phases claim no hosts, so the apply walk can only ever assert that they did nothing.

It starts at Test_12_GatherFactsDistro. Connect, DetectOS, GatherFacts and ValidateHosts run as a single Test_01_Reconnect step, because the apply walk already asserts them one at a time and they behave identically the second time round. The lock phases are left out for the same reason they are left out of the join walk.

So a new phase needs an upgrade test if it is numbered 12 or higher, and the two kinds are the same two the join walk has: a shared body on phaseWalk where the phase does the same thing either way, and a method per suite where it does not. 61_initialize_controller_test.go is the worked example again – the apply walk requires ran(p), the upgrade walk requires !ran(p), because an initialize phase that claimed a running node would re-bootstrap a live cluster.

// Test_NN_YourPhase, on the upgrade walk, <what is different and why>.
func (s *UpgradePhaseSuite) Test_NN_YourPhase() {
	p := &phase.YourPhase{Distro: s.harness.distro}
	s.runPhase(p)
	s.Require().False(ran(p), "<the routing this proves>")
}

Four things the upgrade tests use that the others do not:

Assert version routing by calling the phases’ own comparison rather than comparing strings:

compare := &phase.GenericPhase{}
s.Require().True(compare.VersionLess(host, s.harness.manager.Distro.Spec.Version))

VersionLess reads no manager state, so a zero-valued phase is enough, and using it means the test cannot disagree with the routing it is asserting.

Running it

The upgrade walk is off unless CARGOSHIP_E2E_UPGRADE is set. It installs a second complete set of engine images onto every node, which roughly doubles the disk the run needs and adds tens of minutes.

$ CARGOSHIP_E2E_UPGRADE=1 go test -mod=vendor -count=1 -v -timeout=195m ./test/e2e/cluster/...

Without it, TestClusterPhases/upgrade skips with a message naming the variable, and the reset walk runs against the installed cluster as before.

Prerequisites and running it

The suite needs Docker and a real distro package, and it is not fast: it provisions five containers and installs k3s onto all of them, one controller and four workers, then starts a sixth and joins it. It needs no prebuilt binary – every step calls the cargoship packages directly, so a bare checkout is enough.

$ go test -mod=vendor -count=1 -v -timeout=105m ./test/e2e/cluster/...

Or through mage, which also clears leftover containers from a run that was killed before teardown. Unlike the other e2e mage targets, it builds nothing first:

$ mage test:endToEndCluster           # the dry-run, install, join and reset walks
$ mage test:endToEndClusterDryRun     # only the dry-run walk, against the staging cluster
$ mage test:endToEndClusterUpgrade    # the same, with the upgrade walk in between

-short skips the whole suite, and CARGOSHIP_E2E_UPGRADE adds the upgrade walk. TestMain deletes the bootloose cluster on the way out even when tests fail.

This full walk, the one that installs and bootstraps k3s, is not run in CI – see below. It is a local-box check: run mage test:endToEndCluster (or the go test invocation above) on a machine with Docker before trusting a change to the engine-bootstrap phases, phase/61 onward.

For the smaller staging walk on its own, without the engine phases:

$ mage test:endToEndClusterStage

In CI

.github/workflows/e2e-cluster.yaml runs the staging half of this suite – CARGOSHIP_E2E_STAGE_ONLY=1, stopping at the boundary Test_60_ConfigureEngine draws – on its own trigger, separate from e2e.yaml. It runs on every pull request, the same as e2e.yaml, and on demand from the Actions tab. No label is needed.

The engine-bootstrap half, including the join and upgrade walks, does not run in CI at all. It was not reliable enough on a free hosted runner to be a required check, even on the single-controller k3s topology that replaced the original three-controller rke2 one – see choice-e2e-stage-split. Trusting a change to the engine-bootstrap phases, the join walk, or the upgrade walk means running mage test:endToEndCluster or mage test:endToEndClusterUpgrade locally; CI cannot cover that for you.

The upgrade walk has no CI path either. .github/workflows/e2e-cluster.yaml declares an upgrade boolean input for workflow_dispatch, but nothing in the file reads it: the single e2e-cluster-stage job hardcodes CARGOSHIP_E2E_STAGE_ONLY: "1" and runs on a 30-minute budget, so setting the input changes nothing. Run the upgrade walk locally with mage test:endToEndClusterUpgrade when a change touches the upgrade phases, the version comparison, or anything the install walk can only assert has stayed out of the way.

The workflow has no build step and takes no artifact from e2e.yaml. Nothing in the suite runs a binary: Test_00_CreatePackage calls distro.Create, and the prepare, apply, reset and kube-config steps call the matching action.New* entry points. That is what makes the two workflows independent, which is the point of the split.

The job frees disk before it starts, because five containerd image stores do not fit in what a hosted runner leaves free. If it fails with nodes stuck mid-upload, check the diagnostics step for a full disk or an OOM kill before reading the phase failure as a real one, and a larger runner is the fix.

Iterating while writing a new phase test

A new phase test needs the suite up to that point, not the whole run. Four environment variables, combined, cut a write-run-fix loop down to what your phase actually needs.

Run only the walk your phase is in. TestClusterPhases runs apply, join, upgrade and reset as separate subtests of one parent, so -run at that path segment excludes the ones you are not touching:

$ go test -mod=vendor -count=1 -v -timeout=30m -run 'TestClusterPhases/apply' ./test/e2e/cluster/...

This still walks every phase from Test_00 onward – there is no way to start partway through a walk, because each phase depends on state the ones before it left on the host, Test_07_Connect first among them. What this buys you is skipping the walks after the one you are writing for.

Stay in stage-only mode if your phase is before the Test_60 boundary. CARGOSHIP_E2E_STAGE_ONLY=1 uses the five-machine stageOS inventory and never starts the engine, so it finishes in a few minutes instead of tens:

$ CARGOSHIP_E2E_STAGE_ONLY=1 go test -mod=vendor -count=1 -v -timeout=30m -run 'TestClusterPhases/apply' ./test/e2e/cluster/...

A phase numbered 60 or higher needs the full k3sOS walk instead, so drop this variable for those.

Keep the containers after a failure to inspect them by hand. CARGOSHIP_E2E_KEEP_CLUSTER=1 leaves the machines running instead of deleting them when the run fails, so you can docker exec into one and check what your phase actually left, rather than only what the test’s own assertions could see:

$ CARGOSHIP_E2E_STAGE_ONLY=1 CARGOSHIP_E2E_KEEP_CLUSTER=1 \
    go test -mod=vendor -count=1 -v -timeout=30m -run 'TestClusterPhases/apply' ./test/e2e/cluster/...
$ docker ps --filter "label=io.k0sproject.bootloose.owner=bootloose"
$ docker exec -it <container> sh

Run against virtual machines instead of containers. CARGOSHIP_E2E_BACKEND=microvm, or mage test:endToEndClusterVM, provisions the hosts as local Rocky 10 virtual machines rather than bootloose containers. Everything downstream of the inventory is identical; what changes is that the hosts have a real kernel, so the phases that correctly do nothing against a container – SELinux, fapolicyd, the firewalld backend – run as the paths they are:

$ CARGOSHIP_E2E_BACKEND=microvm go test -mod=vendor -count=1 -v -timeout=60m ./test/e2e/cluster/...

It needs KVM, and a bring-up costs a minute or two against a container’s seconds. It is also one distribution, so it covers none of the family routing the container fleet’s Ubuntu, Fedora and Alpine mix exists for – run both, not one instead of the other. See microvm for the fleet itself and choice-microvm-backend for the reasoning.

Run against a fleet the suite did not create. CARGOSHIP_E2E_BACKEND=inventory with CARGOSHIP_E2E_INVENTORY pointing at a ZarfCluster document runs every phase against hosts that already exist, wherever they are. The assertions that are about a fleet this suite chose – the declared OS per host, the OS mix, the per-family upload guards, the join walk – stand down and say so; everything else asserts as usual. See e2e-external-inventory, including the warning that it changes those hosts and does not change them back.

Walk the suite against rke2 instead of k3s. CARGOSHIP_E2E_DISTRO=rke2 switches which distro every step installs, and which package cluster_lifecycle_test.go builds. The default is k3s. A phase that branches on the engine needs a run each way:

$ CARGOSHIP_E2E_DISTRO=rke2 go test -mod=vendor -count=1 -v -timeout=90m -run 'TestClusterPhases/apply' ./test/e2e/cluster/...

See distroEnvVar and distroID in test/e2e/cluster/05_manager_test.go.

Each go test invocation provisions fresh containers – there is no “resume the cluster from last time” path, so a kept cluster from a failed run is for reading, not for the next run to build on. Clean it up before running again:

$ mage test:cleanCluster

or, without mage, the same removal stopBootlooseContainers does – see mage-test-manual.

Things that will cost you time

• You cannot run one phase test on its own. -run 'TestClusterPhases/apply/Test_25_ModifyHosts' fails: the hosts were never connected, because Test_07_Connect and Test_09_DetectOS are earlier methods that -run filtered out. Run the whole suite, or accept that reproducing one phase means reproducing its predecessors.
• The first failure stops the rest. SetupTest skips every remaining step once one has failed, so the output names one phase rather than twenty-five. If the first failure looks like a symptom, it is the cause – everything after it was skipped, not passed.
• Constants the phase package keeps unexported. Where a path is not exported, the test file redeclares it with a comment saying where it came from (uploadManifestPath in 50_uploadfiles_test.go, sysctlConfPath in 20_prepare_host_test.go). Prefer exporting from phase when the constant is genuinely part of the contract; duplicate it, with the comment, when it is not.
• Six containers is still a lot of memory once the join walk has added its machine. Each runs a k3s node, which is the reason this suite is not part of the default go test ./... path.