How the zarf Ansible modules reach zarf

ZEP-0072 proposes native Ansible module support for zarf: the zarf binary dispatches on argv[0], so a zarf_init symlink to /usr/bin/zarf is an Ansible module, with a thin Python action plugin alongside it. That is the design colonel_byte.cargoship already runs, and choice-ansible-module is the record of why.

This repository holds a proof of that proposal for the two commands that converge a cluster, zarf init and zarf package deploy, plus zarf tools kubectl as zarf_kubectl: the collection in ansible/colonel_byte/zarf/, the module logic in internal/zarfmod/, and one module file binary built from cmd/zarfmod/. The collection also ships read-only modules, which are built as plain action plugins rather than as wrapper binaries and are not part of this proof – see choice-zarf-info-modules. It is a proof of concept and is meant to be removed once the proposal is resolved – see choice-zarf-collection-removal.

Both modules link zarf in and execute its own command tree. How that came about, and what it bought, is the point of this note.

Both modules execute zarf’s command tree in this process

zarf_init and zarf_package_deploy import github.com/zarf-dev/zarf/src/cmd, build zarf’s own command tree with cmd.NewZarfCommand(), and execute init and package deploy in their own process – internal/zarfmod/libraryrunner.go. Each therefore enters the same command an operator’s command line enters: the real flag parsing, and for init the real validateInitFlags, the real agent TLS load, the real existing-state guard and the real init package search. None of that is reachable as API – every identifier in zarf’s src/cmd/initialize.go is unexported – which is why going through the command tree is the only honest way in, and why it proves something a reimplementation would not.

They did this one at a time, and init went first. For a while deploy still ran the installed zarf as a child, which was worth keeping as a side-by-side comparison of the two designs. That stopped being worth its cost once the comparison had been made: internal/zarfmod is one package, so zarf was linked into the deploy binary either way and the child process bought nothing but a second code path and a recorded flag list to maintain. The Runner interface survives with one implementation, because it is what lets the unit tests prove an argument vector without reaching zarf at all.

What linking zarf in bought

The flag surface stopped being a copy. This was the proof’s worst divergence. internal/zarfmod/init.go and deploy.go spell their flags out, and the two tests that check them used to hold those flags against recorded lists under internal/zarfmod/testdata/ – recordings that could catch a flag zarf never had, but not a flag zarf renamed. With the command tree linked in, TestInitArgsAreKnownFlags and TestDeployArgsAreKnownFlags resolve the subcommand with Find and parse the rendered argv with ParseFlags, so pflag itself answers. A renamed flag fails them. So does a flag that moved between the root and the subcommand, and so does --verify written in the space form – the mistake recorded below – which shows up as a second positional argument. The recordings and the generate:zarfFlags target that wrote them are gone.

The zarf the modules run gained an owner. It is the github.com/zarf-dev/zarf version in go.mod, which Dependabot’s gomod ecosystem already watches. magefiles/AGENTS.md asks every pin to have one and prefers Dependabot where it fits; a recorded flag list needing a human to run a mage target was the weaker half of that rule.

The version stopped being read back out of a subprocess. It is config.CLIVersion, compiled in.

What it did not buy, and what that tells upstream

Component progress is still read out of a log stream. ZEP-0072 has the deployer emit heartbeats, where the component list and the phase boundaries are already known. root.ExecuteContext returns an error, not a result, so internal/zarfmod/progress.go still reads component boundaries out of zarf’s log output – which is why the modules render --log-format json by default, and why the total component count is reported as 0 (an honest unknown) unless the operator named --components. What changed is only where the lines come from: logger.DestinationDefault is pointed at a writer the module owns instead of a child’s pipe being scanned.

changedSignal can never be complete. Zarf reports which components it deployed, not whether a component found the cluster already in the state it wanted. initOptions.run discards its DeployResult and state.DeployedComponent.Status records Succeeded or Failed rather than already-in-desired-state, so a successful run reports changed: true, changedSignal: partial, and every component named in changedUndeclared.

Both of these are the argument for the proposal rather than defects to engineer around here. A module layer living inside zarf sits beside initOptions.run, where the component list and the DeployResult are both in scope – which no external caller can reach at any price, however it gets in.

What it cost

vendor/ went from 66M to 386M, and the module file binary is about 280M. The proof is air-gapped-first and vendored, so that tree is the dependency list rather than a cache – see CLAUDE.md. Accepted deliberately: a proof that argues for code living inside zarf has to link zarf, and inside zarf the cost is zero because the binary is already there.

Two things the linking made dead were removed rather than left to rot, and both are breaking changes to the converging modules’ parameter surface.

The zarf_binary parameter and the ZARF_ANSIBLE_BINARY environment variable selected which zarf to execute. Nothing executes a zarf by name any more, so a task that sets either was choosing something that could not be honoured – worse than no parameter at all. The command a result reports now begins with the literal zarf, which is still what an operator would type to reproduce the run. The five read-only modules are plain action plugins that genuinely do shell out, so they keep their own zarf_binary; this was never a collection-wide parameter.

The version floor went with it. MinZarfVersion was 0.72.0, the release where zarf init gained the positional package source, and internal/zarfmod/version.go read zarf version before every init so that an older installed zarf failed naming both versions instead of rejecting an argument it had never heard of. With zarf compiled in, that check asked the binary its own version and compared it against a constant it could not fall below: go.mod is the floor, and check-go-mod.yaml enforces it. zarf.version is still reported – it is config.CLIVersion now, read straight off the linked-in zarf.

Calling packager.Deploy with a packager.DeployOptions instead was considered and rejected. That call is the last of eight steps in initOptions.run, and the other seven are unexported; taking that route means reimplementing init’s orchestration outside zarf, which drifts on every zarf release and argues the opposite of what this proof exists to argue.

One binary under one name per module

Every module file is a symlink to the single binary built from cmd/zarfmod, which answers as whichever zarf_ name it was invoked under. That is the proposal’s own design – in it, /usr/bin/zarf_init is a symlink to /usr/bin/zarf – and here the symlinks point at a wrapper of this repository’s own instead, because a proof built here cannot add the dispatch hook to zarf’s main.

It was a main package per module until linking zarf in made that expensive. Each binary carried its own copy of zarf’s dependency tree – 269MB apiece, 538MB for two – to reach dispatch code that was already shared, since internal/zarfmod was one package the whole time. Collapsing them cost the symlinks and saved a copy per module, which is also what a third module would have cost: zarf_kubectl added one dispatch entry and no binary.

zarfmod.ModuleFiles() is the list of names, derived from the dispatch table, and both the build target and .goreleaser.yaml’s packaging read it rather than keeping lists of their own. The binary’s own name is not one of them, so running it by hand prints what it is for instead of answering as a module.

The root command’s shorthands are pruned, and os.Args is left alone

Cobra merges a parent’s persistent flags into the command being run, so zarf’s root flags land in the flag set of whatever zarf tools kubectl resolves to – and -l is zarf’s --log-level and kubectl’s --selector. pflag panics on that: unable to redefine "l" shorthand. Duplicate long names are fine, because AddFlagSet skips a name the child already has; it is only the letters that collide. LibraryRunner therefore clears the shorthand on every root persistent flag before executing, which costs nothing here because the modules render long flags only. Every letter rather than -l alone: the next zarf release that gives a root flag a letter kubectl already uses would panic in a playbook, and nothing in this repository would have seen it coming.

Zarf’s own CLI avoids the collision differently – it installs no root flags at all for a vendored command, which it decides by reading os.Args – and presenting the rendered command line there was tried first. It works, and it is worse. kubectl’s NewDefaultKubectlCommandWithArgs installs its plugin handler when os.Args names tools kubectl, and that handler calls os.Exit(1) for a subcommand it cannot find and syscall.Execs a plugin it can. Either one leaves the module process without the JSON object Ansible is waiting for, which is a worse failure than the one being avoided, and the second one leaves it running something else entirely. Pruning the shorthands reaches the same place without presenting anything.

Two consequences of not having kubectl’s plugin handler are handled in the module instead. kubectl plugins are not reachable at all, which for an air-gapped management node is the answer that was wanted anyway. And the handler is what diagnoses an unknown subcommand on the real CLI, so without it args: [frobnicate] reached kubectl’s root Run, printed help, and reported success – cobra’s own unknown-command check fires only at a root command, and kubectl’s root has a parent here. validateKubectlCommand does that check: resolve as far as the tree matches, let the resolved command parse the rest, and refuse a leftover positional argument on a command that has subcommands.

zarf_kubectl is a passthrough, and deliberately thin

zarf tools kubectl is vendored kubectl, and the module renders nothing into the argument vector a task gives it: args is kubectl’s own argv, and the only other parameter is kubeconfig, which zarf takes from the environment rather than a flag. Projecting kubectl’s flag surface into Ansible parameters would be reimplementing kubectl in YAML, and kubernetes.core.k8s already exists for declarative resource management.

What it adds is that there is no kubectl to install: the one zarf vendors is already inside the module file, and a management node driving an air-gapped cluster would otherwise carry a second binary through the airlock to read one resource. The prerequisites role already reaches the cluster through zarf tools kubectl for the same reason.

Two things about it are honest rather than convenient. It reports changed: true with changedSignal: unknown, because nothing in the module reads the argument vector: a successful run says the command worked and nothing about whether the cluster moved, and reporting changed: false would tell a playbook to skip the handlers after an apply that did change something. A read is what changed_when: false is for. And check mode skips, because nothing here can tell a get from an apply.

Its failures are reported rather than exited on. kubectl reports a fatal error through cmdutil.CheckErr, which prints and calls os.Exit – correct for a CLI, and the end of a module, since Ansible would report MODULE FAILURE in place of what went wrong. That is the ordinary failure path, not an edge: a cluster that cannot be reached goes through it. cmdutil.BehaviorOnFatal is kubectl’s own hook for a caller that is not its main, and its documentation names panic as the thing to do instead, so runCatchingKubectlExit installs a handler that panics and recovers it into an error with kubectl’s message and exit code.

Its output comes back as zarf.output. kubectl writes to os.Stdout, which it takes when the command tree is built – inside the runner, by which point the stdout guard has pointed os.Stdout at stderr – so runKubectl points it at a temporary file for the length of the run. A file rather than a pipe: nothing drains a pipe while the command runs, and get -o yaml over a large cluster would fill one and hang.

What the five modules beyond the converging pair decided

Each of them had one thing worth recording, and all five are the same binary under another name.

zarf_package_pull is the only module here whose changed is an observation. Everything else in the collection reports changed: true with a signal saying how little that is worth, because zarf reports what it did and not whether the cluster already agreed. A pull writes a file, so the module lists the package files in the output directory before and after and reports the arrivals in zarf.packagesPulled, with changedSignal: complete. That list is also the only way a playbook learns the filename, which zarf derives from the package’s own metadata rather than from anything the task said.

zarf_package_remove reports unknown rather than partial. The converging pair report partial because zarf announces each component as it deploys it, and internal/zarfmod/progress.go reads those announcements. A removal announces nothing until it is over – l.Info("package successfully removed", ...) is the whole of it – so there is no component list behind a signal, and nothing distinguishes a package that was there from one that was already gone. The cluster is what the e2e suite asks instead.

zarf_update_creds renders the subcommand and refuses a mismatched credential. Zarf has update-creds registry, git and agent, each accepting only its own service’s credentials, and the bare zarf tools update-creds that accepts all of them warns that it is deprecated. So service is required, and a credential belonging to another service is refused by name rather than rendered as a flag that subcommand has never heard of. force_conflicts is refused for git for the same reason: zarf puts it on the other two only. The artifact server is the fourth service zarf keeps credentials for, has no subcommand, and is deprecated upstream – so this module does not update it.

zarf_package_mirror_resources reports no components, which was found by running it. It was built with the progress reader wired to it, on the assumption that a mirror walks components the way a deploy does and announces them the same way. The cluster suite showed otherwise: zarf logs mirroring images, pushing image and done pushing images – the resources, never the component they came from. So the module reports changed: true with changedSignal: unknown, carries no heartbeat and takes no status_file, and the assertion in the suite is that the component list is empty.

zarf_package_mirror_resources refuses images and repos together. Zarf marks the two mutually exclusive with MarkFlagsMutuallyExclusive, which is checked when the command runs – after the module has already reported the command it was about to run. Refusing the pair up front makes the message name the parameters rather than the flags. TestMirrorExclusivityIsZarfsOwn holds that refusal against zarf’s own declaration, so a release that stopped marking them exclusive shows up as the module inventing a rule.

zarf_kubectl is covered in its own section above.

What the proof does keep

Everything else in the proposal is implemented as described: argv[0] dispatch with a ZARF_ANSIBLE_MODULE override, the WANT_JSON arguments file, the stdout guard that duplicates fd 1 and points os.Stdout at stderr, exit 0 with failure reported inside the JSON, check mode answered as skipped because zarf init has no dry run, and the heartbeat status file with a polling monitor thread in the action plugin.

The stdout guard earns its keep harder on the linked-in path, and in a way worth recording. cmd.OutputWriter is a *os.File that zarf’s cmd package initialises to os.Stdout when it loads – which is before the guard runs – so it holds the true file descriptor 1 that Ansible parses the module’s JSON from. Reassigning os.Stdout does not move it. LibraryRunner points it at stderr explicitly; left alone, anything zarf printed through it would land in the middle of the result object.

Two of the proposal’s open questions are answered here rather than deferred, because a proof has to pick something:

• Argument delivery. Both options are implemented. The action plugin pipes the parameters on stdin when the connection can carry them, which is the proposal’s leaning and keeps registry and git credentials off disk; it falls back to a 0600 file in a 0700 directory when the connection cannot, which is the proposal’s Option A. The module reads both, so the fallback is a different wire and not a different contract.
• Process cancellation. The module cancels its context on the first signal and exits on the second – which is zarf’s own signal wiring, and is why both main functions here and zarf’s own read alike. The cancelled context is the one zarf executes with. That is the proposal’s leaning – a partial initialisation a later run reconciles beats one still mutating a cluster nobody is watching – without the in-flight rollback the proposal leaves open.

Why there are two modules

zarf init alone cannot be proved against a cluster whose distribution ships no storage provider, and that is not a quirk of the test environment – it is the case the init-local-path uds-bundle exists to handle. Zarf’s registry claims a 20Gi PVC during the seed phase. With no StorageClass the claim stays Pending until the Helm install times out, and disabling the claim does not help: the registry chart then renders no storage configuration at all and the container exits with no storage configuration provided. Both were measured rather than reasoned about, on a k3d cluster from that repository’s own config.

What does work is deploying the storage provider package first with --connected, which pushes no images and so needs no registry, and so needs no initialised cluster. That makes zarf package deploy a prerequisite of zarf init rather than a follow-on, which is why the proof carries both actions and why test/e2e/cluster walks them in both orders – the bundle’s own order on a cluster that has k3s’s StorageClass, and provider-first on one that has none.

Both walks deploy the bundle’s own two packages, pinned in test/e2e/cluster/main_test.go: init-slim:v0.88.0 and csi-local-path-provider:0.0.37-upstream. The init package matters more than it looks. It is the bundle’s slim init, whose five components are all optional, and that is what makes the bundle a four-step alternation rather than two deploys: the first init takes zarf-injector and zarf-seed-registry, and the second takes zarf-registry, zarf-agent and git-server once there is a StorageClass for the registry’s volume. The standard zarf-dev init package has no optional components, so a walk using it deploys everything on the first step and the split means nothing.

Two traps came with that, both recorded in the code. The bundle’s two refs as written – init-slim:v0.88.0-upstream and csi-local-path-provider:v0.0.37-upstream – do not exist; the init publishes no -upstream suffix, and the provider dropped its leading v at 0.0.37. And zarf package pull names a ZarfInitConfig after its kind and version, so the slim init and the standard one both land as zarf-init-<arch>-v0.88.0.tar.zst, which is why the suite caches the slim one in a subdirectory of its own.

The suite adds a fifth step the bundle does not have: the second init, repeated with the same parameters. The bundle’s four steps are all disjoint, so none of them asks whether converging an already-converged cluster succeeds – which for an Ansible module is the question. It needs the git credentials pinned on every init step, and which step matters in a way that took two cluster runs to see. Zarf’s init writes a gitServer block into cluster state whether or not the git-server component was asked for, and validateExistingStateMatchesInput compares field by field with IsNotZeroAndNotEqual – which ignores a field the input leaves empty. So an init that passes no git information is always allowed, and the first init that passes some after one that did not is always refused. Pinning them only on the later steps fails; pinning them from the first init means state carries them and every later init agrees. That is zarf’s guard working as designed, not a fault in the module, but it is a sharp edge for any playbook that converges a cluster more than once.

Two findings from that suite are worth keeping even if everything else here goes away:

• --verify must be rendered as --verify=always. It is a flag with a no-argument default, so pflag reads it the way it reads a boolean: the space form leaves always as a positional argument and zarf rejects the command for having one too many. command.valueFlag exists for exactly this, and --registry-mode does not need it. This cost a cluster run to find. TestInitArgsAreKnownFlags now catches it in milliseconds, because it checks what the real command tree does with the positional arguments – which is the clearest single illustration of what linking zarf in is worth.
• The slim init package requires an adrp.xyz/is-infra=true node. Its registry carries a required nodeAffinity on that label, which is a convention of the infrastructure the uds-bundle deploys onto and is set by neither the bundle nor the uds-bundles k3d config. On an unlabelled k3d cluster the registry pod is unschedulable – 0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector – and the seed phase waits until the Helm install times out, which reads exactly like the StorageClass failure above and is not it. Both k3d-config*.yaml fixtures add the label, and that is the only place they diverge from the upstream config they are otherwise copies of.
• Registry proxy mode is what makes a local cluster usable. Zarf’s default nodeport mode pushes every image through a Kubernetes NodePort, and on k3d that push times out and retries for minutes. registry_mode: proxy puts a DaemonSet on a host port instead and the same initialisation finishes in about a minute.

What the proof does not wire up

The reference pages are generated the same way the cargoship collection’s are: ansibleCollections in magefiles/pkg/gen/docs/ansible.go lists both collections, and this one’s pages land under docs/ansible/zarf/ because the hand-written overviews that introduce them cannot share a filename with the cargoship collection’s. The drift between a Go parameter struct and its Python DOCUMENTATION block is held by internal/zarfmod/plugindocs_test.go, which is the zarf side of internal/ansiblemod/plugindocs_test.go.

The end-to-end suite runs against two k3d clusters. One is k3d-config.yaml in colonel-byte/uds-bundles, copied verbatim, so that what the modules initialise is the cluster shape that repository already deploys onto; the other is the same file with k3s local-storage left enabled, which is what lets a walk begin with init. Both fixtures carry headers saying which walk they serve and why.