colonel_byte.zarf

An Ansible collection that initialises an air-gapped Kubernetes cluster with zarf and deploys zarf packages onto it, from a playbook.

It is a working proof of ZEP-0072, which proposes native Ansible module support for zarf: the zarf binary dispatches on argv[0], so a zarf_init symlink to /usr/bin/zarf is an Ansible module, with a thin Python action plugin alongside it. Everything the proposal describes is implemented here – one binary dispatching on argv[0] with a symlink per module file, the WANT_JSON arguments file, the stdout guard, exit 0 with failure reported inside the JSON, the heartbeat status file and its polling monitor. The modules go further and link zarf in: the binary builds zarf’s own command tree and executes its command in its own process, so it enters the same code path an operator’s command line enters rather than shelling out to it. It needs no zarf on the node. choice-zarf-ansible-module records how that works and what it buys.

The proof began in colonel-byte/cargoship, beside the colonel_byte.cargoship collection whose shape it copies.

Modules

The first two converge a cluster and are the proof. zarf_package_remove is the other direction, which ZEP-0072 names and a playbook that only ever adds does not have; zarf_package_pull stages the package a deploy reads, which on an air-gapped node arrives as a file rather than from a registry the cluster can reach; zarf_package_mirror_resources pushes a package’s images and repositories into a registry and git server without deploying, which is what a connected: true deploy needs to have happened first; zarf_update_creds rotates the credentials an initialised cluster holds, and is the write half of zarf_state_info; zarf_kubectl is a passthrough to the kubectl zarf already vendors, which saves a management node a kubectl of its own. All seven are the same binary from cmd/zarfmod, invoked under seven names – argv[0] is what selects the module, which is the proposal’s own design, so each module file is a symlink.

zarf package create is deliberately not among them: it builds an artifact from a definition in a repository, which belongs in a pipeline rather than in a convergence run.

The other five read a cluster, are not part of the proof, and are plain action plugins with no binary behind them – choice-zarf-info-modules records why, and why zarf_state_info redacts.

There are five roles: packages, package_inspect and state wrap a read-only module apiece, deploy_packages manages multi-package stack deployments with drift detection, and prerequisites refuses to deploy a package onto a cluster that cannot run it.

Package prerequisites

A package can state what it needs of a cluster in a machine-readable document carried inside the package itself, and a deployment can refuse to proceed when the cluster does not satisfy it. That is the second thing in this repository: a colonel-byte.dev/v1alpha1 PackagePrerequisites format, a generated JSON Schema, and the modules and role that read one.

It exists for a failure that reports itself as a success. A workload package delivered as a HelmRelease needs a running helm-controller to do anything at all; deploy it onto a cluster without one and zarf applies the manifests, the API server accepts them, zarf package deploy reports a successful deployment, and nothing is ever reconciled.

- name: Refuse to deploy onto a cluster that cannot run this package
  ansible.builtin.include_role:
    name: colonel_byte.zarf.prerequisites
  vars:
    zarf_package: /srv/staging/zarf-package-operator-external-secrets-amd64-2.12.0.tar.zst
    zarf_public_key: /etc/zarf/colonel-byte-zarf-packages.pub
    zarf_kubeconfig: /etc/rancher/rke2/rke2.yaml

- name: Report what could not be checked
  ansible.builtin.debug:
    msg: "{{ zarf_prerequisites_skipped }}"
  when: zarf_prerequisites_skipped | length > 0

The cluster is read through zarf tools kubectl, so this needs no kubernetes.core – the management node is typically air-gapped, and a second Galaxy collection to stage onto it is a cost every consumer would pay. The guide is the format and what reads it; examples/packages holds three reference packages that declare one: a GitOps engine from each of Flux and Argo CD, and a workload that requires one.

Installing

Build the module wrapper, put it on the path of the node the play runs on, and link each module file name to it. Zarf is linked in, so it is a large binary and its build pulls zarf’s dependency tree from vendor/:

go run ./magefiles/core build:modules          # build/zarfmod, with the three symlinks beside it
sudo install -m 0755 build/zarfmod /usr/local/bin/
for module in zarf_init zarf_package_deploy zarf_package_pull zarf_package_remove \
              zarf_package_mirror_resources zarf_update_creds zarf_kubectl; do
  sudo ln -sf zarfmod "/usr/local/bin/$module"
done

Then make the collection visible to Ansible:

ln -s "$PWD/ansible/colonel_byte/zarf" ~/.ansible/collections/ansible_collections/colonel_byte/zarf

Zarf itself is a prerequisite of the read-only modules, which shell out to the installed zarf; the three module files need none, and nothing is installed on the cluster’s nodes. ghcr.io/colonel-byte/zarf-ansible-modules comes with all of it – ansible-core, the collection, the module wrapper with its names linked to it, and a zarf – for a management node that would rather not install any of them:

docker run --rm -v "$PWD:/workspace" ghcr.io/colonel-byte/zarf-ansible-modules:v0.88.0 \
  ansible-playbook -i management.ini playbooks/zarf-bundle-order.yml

Using it

- name: Initialise the cluster
  hosts: localhost
  connection: local
  tasks:
    - name: Run zarf init
      colonel_byte.zarf.zarf_init:
        init_package: /srv/staging/zarf-init-amd64-v0.85.0.tar.zst
        kubeconfig: /etc/rancher/rke2/rke2.yaml
        registry_mode: proxy
        timeout: 15m
      delegate_to: localhost
      run_once: true
      no_log: true

A play running with -v shows each component as zarf reaches it:

[zarf] component 1: zarf-injector [running]
[zarf] component 1: zarf-injector [done]
[zarf] component 2: zarf-seed-registry [running]

Reading a cluster back, with the credentials withheld:

    - name: Read the zarf state
      colonel_byte.zarf.zarf_state_info:
        kubeconfig: /etc/rancher/rke2/rke2.yaml
      register: zarf_state

More playbooks are in ansible/examples/playbooks/, and the guide at docs/guides/zarf-ansible-module.md walks the whole thing.

Why there are two converging modules

A cluster whose distribution ships no storage provider cannot be initialised on its own: zarf’s registry claims a PVC during the seed phase, and with no StorageClass the claim stays Pending until the Helm install times out. A storage provider package deployed with connected: true pushes no images, so it needs no registry, and so needs no initialised cluster – which makes it the one thing that can come first.

That alternation is the shape of the init-local-path uds-bundle, and the cluster suite walks it in both orders against two real k3d clusters.

Building and testing

Targets run through mage. go run ./magefiles/core <target> needs only the Go toolchain:

go run ./magefiles/core build:modules            # both module files into build/
go run ./magefiles/core test:unit                # everything but the e2e suites
go run ./magefiles/core test:endToEndNonCluster  # needs ansible-playbook; no cluster
go run ./magefiles/core test:endToEndCluster     # needs k3d, zarf, Docker and the network
go run ./magefiles/core generate:document        # regenerate docs/ansible, docs/golang, docs/index.md, docs/SUMMARY.md
go run ./magefiles/core generate:schema          # regenerate schema/ from the Go types in api/
go run ./magefiles/core examples:lint            # zarf dev lint over the reference packages

The unit tests are where the collection and the Go code are held against each other: TestActionPluginDocsMatchModuleParams holds each action plugin’s DOCUMENTATION block against the Go parameter struct, TestInfoPluginDocsMatchRenderedFlags does the same for the read-only modules against the flags their command_parts functions name, and TestEveryActionPluginIsCovered fails on a plugin that neither test looks at. TestInitArgsAreKnownFlags and TestDeployArgsAreKnownFlags parse every flag each module can render against zarf’s own init and package deploy commands, so pflag decides what exists – a renamed flag fails them, and nothing has to be refreshed by hand. On the prerequisites side, TestCommittedSchemasAreCurrent fails when a committed schema stops matching the Go type it is reflected from, and TestExampleDocumentsMatchTheTypes decodes each reference package’s document into those types with unknown fields rejected.

Known limits of the proof

• Component progress is read back out of zarf’s log stream, so the modules render --log-format json by default and report a total component count of 0 unless components was named.
• changedSignal is partial on every successful run. Zarf reports which components it deployed, not whether a component found the cluster already in the state it wanted, and the command tree hands back an error rather than a deployment result, so neither module can honestly report complete. This one needs the module to live inside zarf, beside the deployer.
• Check mode reports the converging tasks as skipped: neither zarf init nor zarf package deploy has a dry run. The read-only modules answer in check mode, because reading a cluster changes nothing.

The rest collapse if dispatch moves inside zarf, which is the point of the proposal.

License

Apache 2.0. See LICENSE.

Security Policy

Supported versions

This collection is pre-1.0 and releases from main on every conventional commit that warrants one, so fixes land in a new minor release rather than being backported. Only the most recent release line receives security fixes; upgrade before reporting an issue against an older tag.

Reporting a vulnerability

Report privately through GitHub: open a draft advisory. Do not open a public issue, and do not include a working exploit against a third party’s cluster.

You should expect an acknowledgement within 3 business days and an assessment – whether we consider it a vulnerability, and the severity we assign – within 10 business days. Confirmed issues are fixed in the next release, and we aim to publish the advisory within 90 days of the report, sooner when a fix is already out. You will be credited in the advisory unless you ask not to be.

Include the collection version from galaxy.yml, the output of zarf version on the node the play runs on, the ansible --version, the task that reproduces the behaviour, and what an attacker gains.

Scope

This collection runs zarf on the node a task is delegated to. It holds cluster credentials in flight, and that is where its security surface is.

In scope:

• Credential delivery to a module. The action plugin pipes parameters on stdin where the connection can carry them, and falls back to a 0600 file in a 0700 directory where it cannot. A path by which registry or git credentials reach disk unprotected, survive the task, or become readable by another user is in scope.
• Credential exposure in output. The wrappers mask credential values in the command line they report. zarf_state_info withholds the eight credentials in the zarf-state Secret by default and sets _ansible_no_log when asked for them, and the state role sets zarf_no_log on its set_fact as well as on the module call – see choice-zarf-info-modules. A credential that reaches a log, a fact cache, or a registered variable’s display against those rules is in scope.
• Credentials in the process table. Flags are rendered for the real run, so a secret passed as a parameter is visible to any local user via /proc for the duration. This is documented rather than fixed – name a zarf config file with zarf_config to avoid it – but a case not covered by that advice is in scope.
• The argv[0] dispatch. A module file invoked under an unexpected name, or with ZARF_ANSIBLE_MODULE set by an untrusted party, must not run something other than the module it names.
• Package verification. The verify and key parameters, and the --verify=always rendering they depend on.
• The release workflow and the artifacts it publishes.

Out of scope:

• Vulnerabilities in zarf itself – report those to zarf-dev/zarf. This collection runs the zarf that is installed; it does not vendor or patch it.
• Vulnerabilities in Ansible, or in a connection plugin.
• Anything that requires an attacker to already have root, or the invoking user’s account, on the node the play runs on – that node holds the kubeconfig by design.
• Misconfiguration of a user’s own cluster, and a playbook author’s own failure to set no_log on a task of their own.

Verifying a release

Release archives and checksums.txt are signed with cosign using the key whose public half is cosign.pub in this repository, archives ship with an SBOM, and every tag also publishes ai-bom.json – the AI-usage disclosure manifest described below – attested and signed the same way.

cosign verify-blob --key cosign.pub --bundle checksums.txt.sigstore.json checksums.txt

AI-assisted development

This collection’s code, docs, and design records are written with AI coding agents in the loop. See AI usage for what’s used and how it’s reviewed. Each release publishes ai-bom.json, a per-release snapshot of that disclosure – verify it the same way as the other release artifacts, above.

Security in CI

Pull requests and main are scanned by CodeQL, dependencies are updated by Dependabot, and supply-chain practices are scored weekly by OpenSSF Scorecard. Results appear in the repository’s Security tab.

ZEP findings

A staging ground, not a submission. What follows is what building this proof of ZEP-0072 turned up about zarf itself – things a native Ansible module cannot work around from the outside, with the evidence for each and a proposed change. It is drafted here so the wording can be argued over in this repository before any of it is put to the ZEP team.

Every line reference is into the github.com/zarf-dev/zarf version this repository pins in go.mod and vendors, so each one can be opened and read without fetching anything.

The through-line is one sentence: Ansible’s contract is idempotency, and zarf does not currently give a consumer enough information to honour it. Each finding below is a different face of that.

1. The deployed-package record is replaced, not merged

DeployedPackage.DeployedComponents is assigned from the components of the deploy that is happening, and the secret is written fresh:

// vendor/github.com/zarf-dev/zarf/src/pkg/state/state.go, NewDeployedPackage
deployedPackage := &DeployedPackage{
	Name:               definition.Metadata.Name,
	...
	DeployedComponents: components,

Generation beside it is a single int that increments, with no retained history. So after

zarf init --components zarf-injector,zarf-seed-registry,git-server   # phase one
zarf init --components zarf-registry,zarf-agent                      # phase two

zarf package list reports the init package’s components as zarf-registry, zarf-agent and nothing else. The cluster has all five; its record remembers two.

Why that is blocking rather than untidy

This is the shape every uds-bundle init takes, because the phases exist for a reason – the seed registry has to exist before a storage provider can be deployed through it, and the full registry needs the StorageClass that provider installs. A consumer that wants to skip work already done has to ask “are these components present”, and the only cluster-side answer is the last phase’s.

The practical result, which we measured: a role that deploys such a bundle oscillates forever. Phase one’s item sees phase two’s components recorded, deploys, and overwrites the record with phase one’s; phase two’s item then sees phase one’s, deploys, and overwrites it back. changed: true on every run, for every run, with no stable state to reach. The suite that found this is TestDeployPackagesRoleWalk in test/e2e/cluster/.

Proposed change

Merge DeployedComponents by component name rather than replacing the list, keeping the newest entry per name. The shape for this already exists: DeployedComponent carries an ObservedGeneration, which is only meaningful if several generations can be represented at once.

Removal already prunes correctly – vendor/github.com/zarf-dev/zarf/src/pkg/packager/remove.go deletes entries with slices.DeleteFunc and removes the secret when the list empties – so a merge would not strand a component that was taken off the cluster, and would not strand one dropped from a later revision of the package.

What we did instead

Nothing that recovers it. The role’s drift detection is documented as converging only for a package deployed in a single phase; a package named by several items in one stack is re-deployed on every run. That is safe – re-deploying reconciles rather than fails, which test/e2e/cluster/testdata/bundle-order.yml asserts deliberately in its fifth step – but it is not idempotent, and it is the limitation this finding would remove. See choice-deploy-packages-drift-scope.

2. A deploy reports what it ran, never whether anything changed

changed is the single most load-bearing field in an Ansible module result. Handlers fire on it, --check reports against it, drift gates read it, and a module that always says true is a module an operator learns to ignore.

Zarf reports which components it deployed. It does not report whether deploying them altered the cluster, so a wrapper has no honest way to answer. This repository’s own module says so in docs/agent/choice-zarf-ansible-module.md and signals changed as partial for exactly this reason – the test holding it is assertStepRan in test/e2e/cluster/zarf_init_test.go, which asserts the signal is partial on every successful step:

Partial is the honest answer and the only one a wrapper can give: zarf reports which components it deployed and not whether any of them found the cluster already as it wanted.

Combined with finding 1, the consequence is that the only path to changed: false anywhere in a zarf Ansible collection is the consumer deciding to skip the deploy entirely – which is the decision finding 1 makes impossible for a phased package.

Proposed change

Either of these would be enough, and the first is the smaller:

• A per-component outcome on the deploy result distinguishing installed, upgraded and unchanged. Helm already knows this per release; the information exists inside a deploy and is discarded.
• A deploy mode that reports the delta it would apply without applying it, which would also serve --check.

A native module would map the first onto changed directly, and the second onto check mode, which it currently has to implement as “refuse and report skipped”.

3. The component selection a deploy would make is not exposed

With no --components, the set zarf deploys is decided in vendor/github.com/zarf-dev/zarf/src/pkg/packager/filters/deploy.go. Non-interactively it is every non-optional component, plus every optional component marked default, plus for each component group of more than one the member marked default – and a group with no default is an error rather than an empty selection.

None of that is answerable from outside. zarf package inspect definition returns the authored definition, so a consumer wanting to know “would this item deploy anything new” has to reimplement the filter, across both package schemas, including the group rules. Getting it wrong in the permissive direction means silently declining a deploy that was needed.

Proposed change

Expose the resolved selection – a flag on zarf package inspect definition, or part of the dry-run in finding 2. The authoritative answer is the one the filter itself computes; anything else is a copy that agrees until it does not.

What we did instead

Declined to guess. An item that names no components is deployed on every run, and the reasoning is recorded in todo-deploy-packages-component-selection so that nobody reimplements the filter by accident.

4. required and optional carry inverted defaults between the two package schemas

v1alpha1.ZarfComponent.IsRequired() returns false when Required is nil; v1beta1.Component.Optional is false when absent, which means required. zarf package inspect definition emits whichever version the package was authored in, so a consumer reading a definition must branch on apiVersion to get the sense of the field right – and a consumer that does not will be exactly wrong, not approximately wrong, on half the packages it meets.

This is working as designed per schema; the finding is that the two are a silent-wrong-answer trap for anything reading definitions generically. A note in the migration guidance would cost nothing and would be read by the next person to write a consumer.

5. Leaked viper default for git-push-username breaks subsequent init when git was omitted

In src/cmd/package.go (newPackageMirrorResourcesCommand), viper registers a default for init.git.push_username:

v.SetDefault(VInitGitPushUser, state.ZarfGitPushUser) // "zarf-git-user"

Because viper is a process-level singleton, newInitCommand() in src/cmd/initialize.go picks up this default when registering the flag:

cmd.Flags().StringVar(&o.gitServer.PushUsername, "git-push-username", v.GetString(VInitGitPushUser), lang.CmdInitFlagGitPushUser)

When an initial zarf init runs without git-server (e.g. in a seed phase before dynamic persistent storage exists), s.GitServer in zarf-state remains unpopulated (s.GitServer.PushUsername == ""). On any subsequent zarf init, even if the operator provides no git arguments or excludes git-server, Cobra’s flag parsing defaults o.gitServer.PushUsername to "zarf-git-user".

validateExistingStateMatchesInput then evaluates:

if helpers.IsNotZeroAndNotEqual(gitServer, s.GitServer) {
    return fmt.Errorf("cannot change git server information after initial init, to update run `zarf tools update-creds git`")
}

Because "zarf-git-user" is non-zero while s.GitServer.PushUsername is empty, helpers.IsNotZeroAndNotEqual evaluates to true and the re-init fails immediately.

Proposed change

Do not register command-specific defaults into the shared viper instance, or do not treat the default username as an operator-provided change in validateExistingStateMatchesInput when s.GitServer was not configured by the initial init.

What we did instead

In internal/zarfmod/libraryrunner.go, after cmd.NewZarfCommand() constructs the command tree, pruneInitGitPushUserDefault clears DefValue and resets the flag value to empty on the init command if it holds the leaked default. A subsequent in-process init without an explicit --git-push-username then leaves gitServer.PushUsername empty, allowing validateExistingStateMatchesInput to pass.

6. A package action’s ./zarf resolves to the system PATH or to the consumer’s own binary, and nothing else

Zarf rewrites ./zarf inside a package action to whatever utils.GetFinalExecutableCommand returns (vendor/github.com/zarf-dev/zarf/src/pkg/packager/actions/actions.go, actionCmdMutation), and cluster.InjectData builds its tools kubectl invocation from the same call (vendor/github.com/zarf-dev/zarf/src/pkg/cluster/data.go). That function offers three answers and no fourth (vendor/github.com/zarf-dev/zarf/src/pkg/utils/io.go):

if config.ActionsCommandZarfPrefix != "" {
    return fmt.Sprintf("%s %s", executablePath, config.ActionsCommandZarfPrefix), nil
}
if config.ActionsUseSystemZarf {
    return "zarf", nil
}
return executablePath, nil

So an action either runs the bare name zarf resolved on PATH, or runs the consumer’s own executable – optionally under a subcommand prefix. There is no way to say “the zarf at this path”.

That matters here because this collection’s modules link zarf in as a library rather than shelling out, so executablePath is the module binary. That binary dispatches on argv[0] and reads its parameters as WANT_JSON on stdin; it accepts no zarf command line at all. A package action running ./zarf tools kubectl ..., or any component with a data injection, therefore fails outright under ActionsUseSystemZarf = false – and fails inside zarf’s own subprocess, where the module cannot improve the diagnosis.

It also undercuts the role’s zarf_binary key. An operator who points the modules at /opt/zarf/bin/zarf has said where zarf is, and that answer cannot be passed on: an action still gets zarf from PATH, or nothing.

Proposed change

Let a library consumer name the binary – an ActionsZarfPath, or an ActionsCommandZarfPrefix that may carry an absolute path of its own – so a consumer that is not a zarf CLI can point actions at a real zarf without putting it on PATH under that exact name.

What we did instead

We leave config.ActionsUseSystemZarf at the true that src/config initialises it to, which is the deliberate delta: zarf’s own main sets it false, and the obvious reading of “we linked zarf in so there is no child process” says a library consumer should do the same. It must not. internal/zarfmod/libraryrunner.go restates the true in its init rather than relying on the default, so a zarf release that flips the default does not silently break every package action, and the management-node image installs zarf at /usr/local/bin/zarf so the bare name resolves.

Checked and not a finding

The deployed version string is the package’s own metadata.version. Worth recording because a consumer that skips work already done needs to compare versions, and it was not obvious that the two came from the same place. zarf package list builds its output with Version: pkg.Metadata.Version (vendor/github.com/zarf-dev/zarf/src/cmd/package.go, packageListOptions.run), which is the same field zarf package inspect definition reports, so the two are equal by construction rather than by convention. A package’s flavour appears in the tarball’s file name and in build.flavor, not in the version the cluster reports.

Package prerequisites

A Zarf package can state what it needs of a cluster, in a document a machine can read, carried inside the package itself. This page is the format, where it lives, and what reads it.

The failure this exists for

A bundle split into independently versioned packages has an order, and nothing enforces it. Deploy a GitOps workload package onto a cluster whose helm-controller is absent and every step succeeds: zarf applies the OCIRepository and the HelmRelease, the API server accepts both, and zarf package deploy reports a successful deployment. Nothing reconciles them. The only symptom is a HelmRelease that never becomes ready, in a cluster nobody is watching, hours later.

The same shape of failure covers a missing CRD, a cluster too old for a chart’s API versions, and a PersistentVolumeClaim with no provisioner to bind it. In each case the package deployed and the workload did not arrive.

What makes it hard to prevent is that the requirement is known – by whoever built the package – and written somewhere nothing can parse. A README says “requires FluxCD 2.9 or newer”. A package variable is called flux.version in one team’s packages and fluxcd in another’s. An orchestrator cannot check any of it, so it deploys and hopes.

The document

The contract is a colonel-byte.dev/v1alpha1 PackagePrerequisites document. It is YAML, it is validated by a generated JSON Schema, and the Go types it is generated from are in api/colonel-byte.dev/v1alpha1/prerequisites.

requires is required and provides is not, which is the asymmetry the format is for: a package that states nothing it requires has no reason to ship the document, while a leaf workload that provides nothing downstream is the common case.

# yaml-language-server: $schema=../../schema/zarf-v1alpha1-package-prerequisites-schema.json
---
apiVersion: colonel-byte.dev/v1alpha1
kind: PackagePrerequisites
metadata:
  package: operator-external-secrets
  version: 2.12.0
  description: Prerequisites for the External Secrets Operator and its webhook

requires:
  kubernetes:
    version: ">=1.30.0"
  fluxcd:
    version: ">=2.9.0"
  controllers:
    - name: source-controller
      namespace: flux-system
      minAvailable: 1
    - name: helm-controller
      namespace: flux-system
      minAvailable: 1
  crds:
    - helmreleases.helm.toolkit.fluxcd.io
    - ocirepositories.source.toolkit.fluxcd.io
  packages:
    - name: app-fluxcd
      version: ">=2.9.0"

provides:
  capabilities:
    - name: external-secrets-engine
      version: "2.12.0"
  crds:
    - externalsecrets.external-secrets.io
    - secretstores.external-secrets.io
    - clustersecretstores.external-secrets.io
  namespaces:
    - external-secrets

The $schema line is what makes an editor complete and validate the file while it is being written. Point it at the schema wherever it resolves for you: a relative path in a checkout, a URL otherwise.

Why the Flux section is three things

A version alone catches the least interesting failure. A cluster can hold the right Flux release with its controllers scaled to zero, or hold the controllers and not the CRDs a package’s own HelmRelease needs. Each of those is the silent case, so each is stated separately.

minAvailable is about availability rather than readiness. A replica that is ready and not yet available has passed its probes and not satisfied minReadySeconds, and a controller in that state is in the middle of its own rollout.

Why provides is declared rather than discovered

So that a bundle can be ordered without deploying any of it. A planner that reads every package’s document can satisfy each requires against another package’s provides and sort the bundle; one that can only read a live cluster has to deploy to find out. The reference packages are shaped for exactly this reason – app-fluxcd and app-argocd both provide the gitops-engine capability, and each also provides an engine-specific name for a package that genuinely needs one engine’s own API. Note what requires cannot yet do with that: there is no requires.capabilities, so a consumer names the package it needs by name in requires.packages. The capability is declared for a planner reading a bundle, and nothing in this repository matches one.

Where it lives: zarf’s documentation: mapping

The document goes inside the package, under zarf’s own top-level documentation: key. That key is a mapping of a name to a path in the package’s source tree, and zarf packages every file it names into a documentation layer of the built package.

kind: ZarfPackageConfig
metadata:
  name: operator-external-secrets
  version: 2.12.0
  annotations:
    colonel-byte.dev/requires-fluxcd: ">=2.9.0"
    colonel-byte.dev/delivery-mode: gitops-helmrelease

documentation:
  prerequisites: docs/prerequisites.yaml
  schema: schema/zarf-v1alpha1-package-prerequisites-schema.json
  readme: README.md

That is the whole reason to use documentation: rather than a file in the repository: the document travels with the package. A tarball copied onto a USB stick and carried into an air-gapped facility carries its own requirements, and zarf package inspect documentation extracts them without deploying anything or reaching a network.

The annotations beside it are a deliberate duplicate of two facts from the document. They are readable from zarf package inspect definition alone, which is one command and no extraction, so a consumer that only needs to order a bundle can read them. The document is for the consumer that is going to check a cluster against it, and it is the source of truth.

Reading it

Three things in this collection read a package’s documentation, and they do different amounts of work.

The mapping alone tells you the package declares a prerequisites document and where it was in the source tree. It does not get you the contents – the file is inside the tarball, and zarf package inspect documentation is what takes it out.

- name: Report the documentation a package carries
  colonel_byte.zarf.zarf_package_inspect:
    package: /srv/staging/zarf-package-operator-external-secrets-amd64-2.12.0.tar.zst
  delegate_to: localhost
  run_once: true
  register: pkg

- name: Read the prerequisites document out of the package
  colonel_byte.zarf.zarf_package_documentation:
    package: /srv/staging/zarf-package-operator-external-secrets-amd64-2.12.0.tar.zst
    keys:
      - prerequisites
  delegate_to: localhost
  run_once: true
  register: docs

- name: Report the Flux version it needs
  ansible.builtin.debug:
    msg: "{{ docs.prerequisites.requires.fluxcd.version }}"
  when: docs.prerequisites is defined

prerequisites is returned only when the extracted file declares the right apiVersion and kind. A package is free to use that documentation key for a prose file, and returning that as a prerequisites document would hand the role a mapping with no requires for the most boring possible reason.

Enforcing it

The prerequisites role is what a deployment actually calls. It extracts the document, reads the cluster with zarf_cluster_info, and fails the play before zarf_package_deploy runs.

- name: Check the cluster against what this package requires
  ansible.builtin.include_role:
    name: colonel_byte.zarf.prerequisites
  vars:
    zarf_package: "{{ package }}"
    zarf_public_key: /etc/zarf/colonel-byte-zarf-packages.pub
    zarf_kubeconfig: /etc/rancher/rke2/rke2.yaml

The cluster is read through zarf tools kubectl, which is a whole vendored kubectl, so the role needs no kubernetes.core – the management node is typically air-gapped, and a second Galaxy collection to stage onto it is a cost every consumer would pay.

Check each package as its turn comes rather than checking the whole bundle up front. The first package in a bundle is often what satisfies the second one’s prerequisites, so a check of everything at the start refuses a deployment that is about to be correct. ansible/examples/playbooks/zarf-prerequisites.yml loops a bundle that way.

What it cannot check, and says so

Two things come back in zarf_prerequisites_skipped rather than being counted as satisfied.

A compound version range. The role reads one comparator and one version, which covers >=2.9.0. Evaluating >=1.30.0 <1.34.0 needs a semver library and Jinja is not one, and a half-parsed range would silently check only its first half. The format allows the range; the role names it as unevaluated.

requires.storage.accessModes. What a provisioner will actually bind is not discoverable without creating a PersistentVolumeClaim, which is not a thing a read-only check does.

Print that list even on a successful run. A run that reported only its successes would read as though everything in the document had been verified.

Where the engine version comes from

The version under requires.fluxcd or requires.argocd is checked against the app.kubernetes.io/version label on the engine’s own workloads. For Flux that is the flux2 release the manifests were built from – not the controllers’ own versions, which are a different set of numbers entirely: a document naming >=2.9.0 is not talking about source-controller v1.9.5.

Neither engine’s upstream manifests carry that label. flux install adds it from its own build, and Argo CD’s install.yaml has none at all – so a package that vendors either has to add it. app-fluxcd and app-argocd both do it in a generated kustomization, which is most of why those files are generated. A cluster whose engine carries no version label has the constraint reported as unchecked – honestly, and uselessly.

A version without an engine

requires.controllers[].version is a SemVer range on that workload’s own app.kubernetes.io/version label, and it is how a package requires a version of something that is not a GitOps engine.

requires:
  controllers:
    - name: cert-manager
      namespace: cert-manager
      version: ">=1.14.0"

No fluxcd or argocd key anywhere: the package needs that workload at that version and has no opinion about what reconciles the cluster. Without it, the only version a document could put on a workload was the engine’s, so a package needing cert-manager v1.14 had to require an engine it does not use purely to get a version field – a prerequisite that refuses clusters for no reason.

It is the same label and the same comparison the engine sections use, so the same two caveats apply: one comparator and one version, and a workload carrying no version label has the constraint reported as unchecked rather than satisfied. Most charts following the Kubernetes recommended labels set it; a workload nobody stamped does not.

Which engine, and why the format asks

An entry under requires.packages names its prerequisite with name, or with names when the package is published under more than one – the upstream init reports itself as init and a rebuild of it as init-slim, and a package needing an initialised cluster needs either. Several names rather than several entries, because the version constraint belongs to the requirement and not to each name: restating it per name is a constraint that gets bumped in one place and not the other. An entry carries one of the two keys and never both, which the schema refuses and the role refuses again for a document handed to it directly – the role reads names in preference to name, so an entry carrying both would say one thing and be checked as another.

requires:
  packages:
    - names:
        - init
        - init-slim
      version: ">=0.85.0"

When alternative packages have different version histories, group them with id so each alternative keeps its own version constraint. See Grouping alternative package prerequisites with id for details.

The engine is named by the key – requires.fluxcd or requires.argocd – and the two are mutually exclusive, which the generated schema states as well as the role. The key names the engine rather than a field inside it because the two do not number together: Flux is on 2.x while Argo CD is on 3.x, so a version sitting beside a separate engine field could disagree with it, and the dangerous direction of that disagreement is silent. A document meaning “Flux >= 2.9” that named argocd by mistake is satisfied by Argo CD 3.5, because 3.5 >= 2.9 – the constraint passes against an engine it was never about. Naming the engine with the key leaves nothing for the version to contradict.

Only two things about checking an engine are engine-specific at all.

The first is the namespace a controller defaults to: flux-system for Flux, argocd for Argo CD. A document whose controllers name their namespaces never needs the engine for this; one that leaves them off does.

The second is the kind. Flux’s controllers are all Deployments. Argo CD runs argocd-application-controller as a StatefulSet and the rest of itself as Deployments, which is why requires.controllers[].kind exists at all – a check that assumed Deployment would report the component doing all of Argo CD’s reconciling as absent from a perfectly healthy cluster. It defaults to Deployment, so only the Argo CD controller has to say it.

requires:
  argocd:
    version: ">=3.5.0"
  controllers:
    # No namespace on either: the argocd key makes both default to the argocd namespace.
    - name: argocd-application-controller
      kind: StatefulSet
    - name: argocd-server
  crds:
    - applications.argoproj.io
    - appprojects.argoproj.io

Everything else – “these workloads must be available”, “these CRDs must be established” – is the same question whatever reconciles the cluster, which is why requires.controllers and requires.crds sit beside the engine section rather than inside it.

The reference packages

examples/packages holds three packages that declare documents: two providers of the same capability, and a consumer of it.

None of them is built or deployed by anything here – zarf package create on either reaches the network. They are to be read, linted and inspected:

go run ./magefiles/core examples:lint

app-fluxcd is packaged from the upstream flux2 release manifests, vendored from a pinned tag by examples:vendor and with its image lists discovered by examples:images. See the mage reference for those targets, and magefiles/pkg/pkgvendor for what they do.

When a document changes

The schema is generated from the Go types, so changing the format is a change to spec.go followed by:

go run ./magefiles/core generate:schema

The descriptions in the schema are the doc comments on those types, which is why they are written as prose for a package author rather than as notes for a Go reader: an editor with YAML schema support shows them while the document is being written.

Two tests hold the pieces together. TestCommittedSchemasAreCurrent fails when the committed schema stops matching the types – a stale schema does not fail, it validates the wrong document, which is worse than not validating one. TestExampleDocumentsMatchTheTypes decodes each reference package’s document into the types with unknown fields rejected, so an example written against an older shape of the format fails rather than quietly becoming the thing readers copy.

The version is v1alpha1 and means it: the format is a proof alongside ZEP-0072, not a published API.

Grouping alternative package prerequisites with id

This guide describes how to use id groups in colonel-byte.dev/v1alpha1 PackagePrerequisites documents. It covers how id groups organize alternatives with different version requirements and how the colonel_byte.zarf.prerequisites role evaluates them.

The problem: alternatives with different versions

A package often accepts more than one upstream provider to satisfy a requirement. For example, a package may run on either Rook Ceph or Longhorn for persistent storage.

When alternatives share the same version history, use names:

requires:
  packages:
    - names:
        - init
        - init-slim
      version: ">=0.85.0"

The names list applies a single version constraint across every name in the list.

When alternatives have different version histories, a single version constraint cannot describe both packages. Rook Ceph may need >=1.21.0 while Longhorn needs >=1.13.0.

The solution: id groups

Set an optional id string on each alternative entry under requires.packages:

requires:
  packages:
    - name: csi-rook-ceph
      version: ">=1.21.0"
      id: storage
    - name: csi-longhorn
      version: ">=1.13.0"
      id: storage

Entries with the same id value form a single disjunctive (OR) requirement group.

Evaluation rules

The colonel_byte.zarf.prerequisites role evaluates id groups with the following rules:

1. Local scope: The id key applies only to requires.packages. It does not affect other requirement types.
2. Group satisfaction: The cluster satisfies a group if at least one entry in the group is deployed and meets its version constraint.
3. Multiple deployed packages: When a cluster has multiple packages from the same group installed, the check passes if at least one package meets its constraint. A failing sibling does not fail the group.
4. Independent entries: Entries without an id remain independent requirements. The cluster must satisfy all ungrouped entries.
5. A group is stricter than a plain entry. A plain requires.packages entry whose version is a range this role cannot read – a compound range such as >=1.30.0 <1.34.0, which the format’s one-comparator limit covers – is reported in zarf_prerequisites_skipped and does not fail the play, because the package’s presence was proven by a separate check and only its version is unknown. Inside a group there is no such separate proof: the group’s whole contract is “one of these, at an acceptable version”, and a constraint that cannot be read proves nothing about any branch. So a group whose only deployed branch carries an unreadable range fails, and says which of the three things went wrong – see the messages below. Keep constraints inside a group to one comparator.
6. Names per branch: An entry inside an id group can still use names to cover aliases for that branch:

requires:
  packages:
    - names:
        - csi-rook-ceph
        - csi-ceph
      version: ">=1.21.0"
      id: storage
    - name: csi-longhorn
      version: ">=1.13.0"
      id: storage

Error reporting

When no branch in an id group meets its requirements, the role records one failure in zarf_prerequisites_unmet. The message names every option in the group, and then says which of three things happened – because “nothing is deployed”, “the wrong version is deployed” and “the constraint could not be read” are different problems with different fixes.

Nothing in the group is deployed:

one of the alternative zarf packages for storage (csi-rook-ceph >=1.21.0 or csi-longhorn >=1.13.0) is required, and none of them is deployed to this cluster

Something is deployed, and its version was compared and did not satisfy the constraint:

one of the alternative zarf packages for storage (init >=9.0.0 or csi-longhorn >=1.13.0) is required, and what is deployed does not satisfy its version constraint

Something is deployed, and its constraint is a range this role cannot read, so no branch could be verified. The range is named in zarf_prerequisites_skipped as well, which is the half that tells you what to simplify:

one of the alternative zarf packages for storage (init >=0.85.0 <1.0.0 or csi-longhorn >=1.13.0) is required, and what is deployed carries a version range this role cannot evaluate (the deployed zarf package init), so no alternative could be verified

If packages from the group are not installed, the role reports:

one of the alternative zarf packages for storage (csi-rook-ceph or csi-longhorn) is required and is not deployed to this cluster

Driving zarf from Ansible

The colonel_byte.zarf collection initialises a cluster with zarf and deploys zarf packages onto it from a playbook. It is a proof of ZEP-0072, which proposes native Ansible module support for zarf. It began in colonel-byte/cargoship, because the pattern it proposes is the one colonel_byte.cargoship already runs there.

This guide covers the two modules that converge a cluster: what they are, how to install them, the two-module walk that a cluster with no storage provider needs, and what the result holds. The collection also ships five read-only modules and three roles that wrap three of them – zarf_cluster_info, zarf_package_documentation, zarf_package_info, zarf_package_inspect and zarf_state_info – which are not part of the walk and are documented on the collection reference; choice-zarf-info-modules records why they are built differently. The full parameter set of each module, and the zarf flag each parameter renders, is on its reference page: zarf_init and zarf_package_deploy, both generated from the collection. For how the modules reach zarf – each links zarf’s command tree in and executes it in its own process – see choice-zarf-ansible-module.

The two converging modules

Both run on the node the task is delegated to, not on the cluster’s nodes. Zarf reaches the cluster through a kubeconfig, so there is no inventory to project and nothing to copy to a managed node: a task supplies a kubeconfig, a package, and the parameters the zarf command takes.

Neither module is a Python module. Each is a compiled binary that renders the zarf command line from the task’s parameters, executes zarf’s own command in its own process, follows the deployment component by component, and writes one JSON object back. Zarf is linked in rather than installed: neither module needs a zarf on the node, and the zarf that does the work is the version the binary was built against. choice-zarf-ansible-module records why.

Installing

The release .rpm, .deb, and .apk install the collection to /usr/share/ansible/collections/ansible_collections/colonel_byte/zarf/ – on Ansible’s default collections path – along with /usr/bin/zarfmod, a /usr/bin/zarf_<module> symlink per module, and the same names under the collection’s plugins/modules/. One binary answers as all of them: the name it is invoked under is what selects the module. Installing the package is all a management node needs, beyond a zarf of its own for the read-only modules.

From a checkout, build the module files and put them on the path:

go run ./magefiles/core test:endToEndCluster   # builds the wrapper, then runs the e2e suite
# or just the module files:
go run ./magefiles/core build:modules          # build/zarfmod, with a symlink per module name

A management node needs ansible-core, the collection on Ansible’s collections path, the module wrapper with its names linked to it, and a zarf of its own for the read-only modules. ghcr.io/colonel-byte/zarf-ansible-modules is all four: it installs the release .rpm on top of ansible-core and carries the zarf binary out of zarf’s own agent image. One tag per release, and no latest – a management node is meant to pin the image it runs, the way it pins everything else it carries through the airlock.

docker run --rm -v "$PWD:/workspace" -v "$HOME/.kube:/home/nonroot/.kube:ro" \
  ghcr.io/colonel-byte/zarf-ansible-modules:v0.88.0 \
  ansible-playbook -i management.ini playbooks/zarf-bundle-order.yml

The image is built on every pull request that touches it, because its build asserts the module files’ dispatch contract – see test-build-containers.yaml.

A task can name a wrapper outright rather than relying on the path, which is what the e2e suite does:

- name: Initialise the cluster
  colonel_byte.zarf.zarf_init:
    wrapper_binary: /home/operator/ansible-collection-zarf/build/zarf_init
    init_package: /srv/staging/zarf-init-amd64-v0.85.0.tar.zst
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true
  no_log: true

Set run_once: true and delegate_to on every task. One zarf init initialises the whole cluster, so a play over the cluster’s own inventory would otherwise initialise it once per host. Set no_log: true on any task carrying a credential: a binary module has no per-parameter no_log, so the task-level setting is what suppresses the arguments and the result.

Naming the init package

init_package is the package zarf deploys, passed as the positional package source zarf init [ PACKAGE_SOURCE ] takes. A path to a staged tarball under any name, an oci:// reference, or an https:// URL all work, and so does a directory – which leaves zarf to search inside it the way it does when given no source at all, for a file named after zarf’s own version.

zarf package pull oci://ghcr.io/zarf-dev/packages/init:v0.85.0 --architecture amd64 \
  --output-directory /srv/staging

A package pulled that way keeps the zarf-init-<arch>-<version>.tar.zst name, and the version has to be the version of the zarf that will deploy it: zarf version on the node that runs the play is what picks the tag. A package built in-house is under no such constraint about its name, because nothing looks it up by name.

The positional argument arrived in zarf v0.72.0; before it, the only way to choose a package was to run zarf in the directory holding a file named after zarf’s own version. That is no longer a compatibility question an operator can get wrong: the zarf that does the work is linked into the module, so the version is whatever the module was built against and is reported as zarf.version in the result. There is nothing to install and no floor to trip over.

directory is now a separate thing: the working directory zarf runs in, which is where a relative init_package and a relative path inside a zarf_config file resolve from. The two may be set together.

- name: Initialise the cluster from a package built in-house
  colonel_byte.zarf.zarf_init:
    init_package: /srv/staging/our-init-package.tar.zst
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true
  no_log: true

- name: Initialise the cluster from a package held in an internal registry
  colonel_byte.zarf.zarf_init:
    init_package: oci://registry.bubbles.test/zarf/init:v0.85.0
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true
  no_log: true

The walk a cluster with no storage needs

A cluster whose distribution ships no storage provider cannot be initialised in one task, and the reason is worth knowing before a playbook is written: zarf’s registry claims a 20Gi PersistentVolumeClaim during the seed phase. With no StorageClass to satisfy it the claim stays Pending and the Helm install times out. Disabling the claim does not help – the registry chart then renders no storage configuration at all and the container exits.

What does work is deploying a storage provider first, with connected: true. A connected deploy does not push the package’s images, so it needs no registry, and so needs no initialised cluster. That is the one zarf operation available on a cluster nothing has touched, and it is why this collection has a deploy module as well as an init one:

- name: Deploy the storage provider onto the uninitialised cluster
  colonel_byte.zarf.zarf_package_deploy:
    package: oci://ghcr.io/colonel-byte/zarf/csi-local-path-provider:0.0.37-upstream
    components: local-path-images,local-path-chart
    kubeconfig: "{{ zarf_kubeconfig }}"
    public_key: /etc/zarf/colonel-byte-zarf-packages.pub
    verify: always
    connected: true
  delegate_to: localhost
  run_once: true

- name: Initialise the cluster
  colonel_byte.zarf.zarf_init:
    init_package: /srv/staging/zarf-init-amd64-v0.85.0.tar.zst
    kubeconfig: "{{ zarf_kubeconfig }}"
    registry_mode: proxy
  delegate_to: localhost
  run_once: true

This is the shape the init-local-path uds-bundle carries, which lists the init package and the storage provider twice each and alternates between them. test/e2e/cluster/testdata/provider-first.yml and bundle-order.yml are that bundle written as playbooks, and both are run by mage test:endToEndCluster.

Two parameters earn their keep on the second pass over an already-provisioned cluster:

• take_ownership: true lets zarf adopt objects it did not create. A cluster that already has a local-path StorageClass – k3s ships one – makes Helm refuse the provider package without it.
• force_conflicts: true lets server-side apply overwrite fields another controller owns, which is the next thing that refuses after ownership is settled.

Registry proxy mode

registry_mode: proxy is worth preferring on a small or local cluster. Zarf’s default nodeport mode publishes the registry on a Kubernetes NodePort and pushes every image through it; on a k3d cluster that push is slow enough to time out and be retried, which adds minutes to an initialisation. Proxy mode puts a DaemonSet on a host port instead. The mode zarf settled on is recorded in registryInfo.registryMode inside the zarf-state secret.

What the result holds

{
  "changed": true,
  "msg": "initialised the cluster",
  "zarf": {
    "module": "init",
    "command": ["zarf", "init", "--confirm", "--no-color", "--log-format", "json", "--registry-mode", "proxy"],
    "directory": "/srv/staging",
    "componentsRan": ["zarf-injector", "zarf-seed-registry", "zarf-registry", "zarf-agent"],
    "changedSignal": "partial",
    "changedUndeclared": ["zarf-injector", "zarf-seed-registry", "zarf-registry", "zarf-agent"],
    "statusFile": "/tmp/zarf-status-m56nictp.json"
  }
}

componentsRan is what a playbook has in place of per-component tasks, and command is the command line to reproduce the run by hand – with credential values replaced, so it is safe to print.

changedSignal is partial for every successful run, and that is honest rather than unfinished. Zarf reports which components it deployed and does not report whether any of them found the cluster already as it wanted, so the modules report changed: true and name every component as undeclared. A module inside zarf, with access to the deployer, is what could report complete.

Live progress

Zarf deployments take minutes, and an Ansible task prints nothing until it returns. The wrapper writes each component boundary to a status file and the action plugin displays it, so a play run with -v reports progress as it happens:

[zarf] component 1/4: zarf-injector [running]
[zarf] component 2/4: zarf-seed-registry [running]
[zarf] component 3/4: zarf-registry [running]

The count on the left is only known when the task names components; otherwise the display reports component 3 rather than component 3/4, which is what an honest unknown looks like. The monitor polls twice a second, so a component that finishes and is replaced inside that window shows its [running] line and not its [done] line.

Check mode and failures

Check mode reports the task as skipped. Neither zarf init nor zarf package deploy has a dry run, and running anyway during a --check play is the one thing check mode must not do.

A failure is reported inside the result, with the process still exiting 0, so Ansible shows the module’s message rather than its own MODULE FAILURE diagnosis. Zarf’s own output goes to stderr, where Ansible captures it as module_stderr: standard output carries the one JSON object and nothing else.

- name: Deploy a package
  colonel_byte.zarf.zarf_package_deploy:
    package: /srv/staging/zarf-package-monitoring-amd64.tar.zst
    kubeconfig: "{{ zarf_kubeconfig }}"
  register: deploy
  failed_when: deploy.failed

Credentials

Parameters carrying credentials – the registry and git push and pull passwords, the registry secret – are rendered as flags, which puts them in the process table for the life of the run. Keep them out of it by putting them in a zarf config file and naming it with zarf_config, which the module passes as ZARF_CONFIG:

- name: Initialise with credentials zarf reads from its own config file
  colonel_byte.zarf.zarf_init:
    init_package: /srv/staging/zarf-init-amd64-v0.85.0.tar.zst
    kubeconfig: "{{ zarf_kubeconfig }}"
    zarf_config: /etc/zarf/zarf-config.yaml
  delegate_to: localhost
  run_once: true
  no_log: true

The parameters themselves never reach disk on the path this collection’s action plugins take: they are piped to the wrapper on stdin. A connection that cannot carry data on stdin – SSH without pipelining – falls back to a 0600 arguments file in a 0700 directory, removed when the task ends. Both are conventions ZEP-0072 weighs; the wrapper reads either.

Reproducing a module run by hand

Both wrappers answer as the module named by the basename they were invoked under, or by ZARF_ANSIBLE_MODULE, and read their parameters from a JSON file named as the single argument or from stdin:

printf '%s' '{"init_package":"/srv/staging/zarf-init-amd64-v0.85.0.tar.zst","kubeconfig":"/etc/rancher/rke2/rke2.yaml"}' \
  | ZARF_ANSIBLE_MODULE=init zarf_init

This is the invocation the action plugin makes, so it is the one to reach for when a task fails and the question is whether the module or the playbook is wrong.

Testing

mage test:endToEndCluster builds both wrappers and runs the e2e suite, which stands up two k3d clusters – one with a storage provider and one without – and walks the uds-bundle’s four packages across each. It needs k3d, zarf, ansible-playbook, Docker, and the network to pull the packages once; each test skips with a reason when one is missing. mage test:cleanClusters removes the clusters a killed run left behind.

The colonel_byte.zarf collection

Zarf runs inside a playbook as an Ansible collection: zarf init initialises a cluster and zarf package deploy puts a package on it, both driven from the node the task is delegated to. Those two modules are one compiled wrapper binary invoked under the names zarf_<action>, and the name selects the action. zarf_package_pull, zarf_package_mirror_resources, zarf_package_remove, zarf_update_creds and zarf_kubectl are five more names on the same binary: staging a package on disk, seeding a registry and git server from one, taking a deployed package off a cluster, rotating the credentials the cluster holds, and passing an argument vector to the kubectl zarf vendors. There is no Python module behind any of them, and nothing is installed on the cluster’s nodes.

Five further modules read a cluster rather than converging it – the cluster’s own facts, a package’s documentation, the deployed packages, a package definition, and zarf’s own state – and three roles wrap three of them. Those are plain action plugins rather than wrapper binaries, and choice-zarf-info-modules is the record of why a read-only command earns no binary.

This collection is a proof of ZEP-0072, which proposes this pattern for zarf itself. It began in colonel-byte/cargoship, because the pattern it copies is the one colonel_byte.cargoship already runs there. What becomes of it is tied to that proposal’s outcome – see choice-zarf-collection-removal.

Zarf is not an SSH orchestrator. It reaches one cluster through a kubeconfig, so Ansible supplies no inventory here: a task supplies a kubeconfig, a package, and the parameters the zarf command takes. For the seven modules that act, the zarf that does the work is the one linked into the binary; the read-only modules run the zarf on the node’s path.

What is here

Which to reach for

No module picks another’s work up. A playbook writes the action it wants: zarf_init for the cluster itself, zarf_package_deploy for anything deployed onto it afterwards, zarf_package_pull to stage a package before either, zarf_package_mirror_resources to seed the registry it deploys from, zarf_package_remove to take one away, zarf_update_creds to rotate what the cluster holds, and zarf_kubectl for the one resource a playbook has to read or patch directly.

The two are usually written together rather than separately. A cluster whose distribution ships no storage provider cannot be initialised until one is deployed – zarf’s registry claims a volume during the seed phase – and a package cannot normally be deployed until the cluster is initialised. The way out is a connected deploy, which pushes no images and so needs no registry; the module guide covers the ordering.

For reading a cluster, reach for the role rather than the module. state, packages and package_inspect exist to set what the module returned as a fact under a predictable name, and the state role additionally defaults no_log on both of its tasks – which the module cannot do on the playbook’s behalf. A module called directly is the right answer when a play wants the result registered rather than as a fact, or wants two of them in one play.

Reference pages are generated

Every module_*.md page here is generated from the collection itself – the DOCUMENTATION and EXAMPLES blocks in each action plugin – and every role_*.md page from a role’s meta/argument_specs.yml, by mage generate:document. Edit the collection and regenerate; an edit to a generated page does not survive the next commit.

This page, modules.md and roles.md are written by hand; everything else in this directory is generated by mage generate:document and will not survive the next run. The contract the generators implement is in ansible/colonel_byte/zarf/AGENTS.md: every option declares a type, a description, and a cli_flag, and every description entry is double-quoted.

Modules

Twelve modules ship. Seven of them act, and five read a cluster.

zarf_init and zarf_package_deploy are the converging pair: they are the proof of ZEP-0072, each has its parameters and command line in Go, and choice-zarf-ansible-module records how. Package creation is not among them – it builds an artifact from a definition in a repository, which belongs in a pipeline rather than in a convergence run – and neither is package removal, which nothing here has needed yet.

zarf_package_remove is the other direction of a deploy: it takes a deployed package, or some of its components, off a cluster. ZEP-0072 names it, and a playbook that can only add is not reconciling anything. zarf_package_pull acts on a directory rather than a cluster – it fetches a package to disk, which is how a package reaches an air-gapped management node before a deploy reads it – and it is the one module here whose changed is an observation: it reports the package files that were not in the output directory before the run.

zarf_package_mirror_resources pushes a package’s images and git repositories into a registry and a git server without deploying anything, which is the step a connected: true deploy needs to have happened first. zarf_update_creds rotates the credentials an initialised cluster holds: it is the write half of zarf_state_info, and it is one zarf subcommand per service, so a task names the service it is rotating.

zarf_kubectl is the last that acts, and it is not a convergence module: it passes an argument vector to the kubectl zarf vendors and hands back what it printed. It exists because that kubectl is already inside the module file, so a management node reading or patching one resource on an air-gapped cluster needs no kubectl of its own. For declarative resource management, kubernetes.core.k8s is the module to reach for.

All seven are the same binary, invoked under seven names. It dispatches on argv[0], which is what ZEP-0072 proposes zarf itself do, so each module file under plugins/modules/ is a symlink rather than a file of its own.

zarf_cluster_info, zarf_package_documentation, zarf_package_info, zarf_package_inspect and zarf_state_info read. They change nothing, they answer during a check-mode play rather than skipping, and they are plain action plugins with no wrapper binary behind them, for the reasons in choice-zarf-info-modules and – for the two added for the prerequisites format, which are the first to push past the threshold that record names – in choice-package-documentation-module.

The surfaces differ because the commands do

zarf init configures the registry and git server the cluster will use, so it takes the registry and git parameters and no package. A removal takes seven flags where a deploy takes twenty-eight, because there is no signature to verify and nothing to push once the thing is being taken away; a pull takes neither a kubeconfig nor --confirm, because it reaches no cluster and the command has no such flag; and zarf_update_creds takes only the credentials of the service it names, because zarf has one subcommand per service and each accepts only its own – a credential given for another service is an error naming it. zarf package deploy names a package outright and takes none of that configuration, but takes connected, namespace and shasum, which only a deploy has. A parameter offered to the wrong module is an error naming it, rather than a flag quietly dropped.

Each module’s page lists its full parameter set and the flag each parameter renders. A parameter that renders no flag is listed as None: it is either the command’s positional argument, something passed to zarf in its environment, or something the action plugin consumes before the command line is built.

What every task needs

A parameter left unset is left unset: the module passes no flag for it, so whatever zarf’s own configuration file sets still applies. That is why plain_http: false and omitting plain_http are different.

Set run_once: true and delegate_to on every task. One zarf init initialises the whole cluster, one deploy puts the package on all of it, and one read answers for all of it, so a play over the cluster’s own inventory would otherwise do the work once per host. Set no_log: true on any task carrying a credential: a binary module has no per-parameter no_log, so the task-level setting is what suppresses the arguments and the result.

Credentials given as parameters are rendered as flags, which puts them in the process table for the life of the run. zarf_config names a zarf configuration file instead, which zarf reads them from.

What the result says

The two converging modules report changed: true with changedSignal: partial on every successful run, and name every component they deployed in changedUndeclared. Zarf reports which components ran and not whether any of them found the cluster already as it wanted, so partial is the honest answer rather than an unfinished one. Check mode reports the task as skipped, because neither command has a dry run.

zarf_package_mirror_resources reports changed: true with changedSignal: unknown and no component list: a mirror announces the images it pushes rather than the component they came from, so there is nothing to follow. zarf_update_creds reports changed: true with changedSignal: unknown: zarf says it updated the service, not whether it already held what was given, so a credential set twice is reported as changed both times.

zarf_package_remove reports changed: true with changedSignal: unknown: zarf logs a removal as one line and names no component on the way, so there is nothing that distinguishes a package that was there from one that was already gone – a removal that finds nothing fails in zarf. zarf_package_pull is the exception in the collection: it reports changedSignal: complete, because the package file either arrived in the output directory or it did not, and the filenames are in zarf.packagesPulled.

zarf_kubectl reports changed: true with changedSignal: unknown, because the argument vector is kubectl’s and nothing in the module reads it: a successful run says the command worked and nothing about whether the cluster moved. A read wants changed_when: false on the task. Check mode skips it for the same reason – nothing there can tell a get from an apply.

The five read-only modules report changed: false, which is not an approximation, and they run in check mode rather than skipping: answering a question during a run an operator asked to be told about is the honest thing to do.

zarf_state_info withholds what it reads. The zarf-state Secret carries the registry, git and artifact server credentials and the agent webhook’s TLS private key, so the returned state has them removed and redacted names the paths that went. include_credentials: true returns them and censors the task’s own output in the same breath; the task that reads the result afterwards needs its own no_log, because set_fact prints what the module hid.

The two that exist for the prerequisites format

zarf_package_documentation and zarf_cluster_info are the two halves of a check that runs before a deployment. A package states what it needs of a cluster in a PackagePrerequisites document carried in its own documentation: mapping; the first extracts that document, the second reports the cluster facts it is checked against, and the prerequisites role is what holds the two together and fails the play. Reach for the role rather than the modules unless you are checking something the format does not describe.

zarf_cluster_info reads the cluster through zarf tools kubectl, which is a whole vendored kubectl, rather than through kubernetes.core. That keeps this collection’s only requirement the zarf binary it already requires: the management node a playbook runs from is typically air-gapped, and a second Galaxy collection to stage onto it is a cost every consumer of this one would pay. zarf_state_info reads the zarf-state Secret the same way, and for the same reason.

The module guide covers installing the modules, the ordering a cluster with no storage provider needs, the live progress display, and the shape of the result.

cluster_info

zarf_cluster_info

Report the cluster facts a package’s prerequisites are checked against.

Write it as colonel_byte.zarf.zarf_cluster_info. It runs zarf on the node the task is delegated to; nothing is installed on the cluster’s nodes.

Added in colonel_byte.zarf 0.33.0.

Reads the cluster through zarf tools kubectl and returns the facts a colonel-byte.dev/v1alpha1 PackagePrerequisites document can require: the Kubernetes version, the established CRDs, the StorageClasses, and the readiness of every Deployment and StatefulSet.

Needs no kubernetes.core: everything is read through the zarf binary this collection already requires, which carries its own kubectl. The management node is typically air-gapped, and a second collection to stage onto it is a cost every consumer would pay.

The answers are digested rather than returned whole. crds is a list of names, not of objects: a cluster with two hundred CRDs answers with megabytes of schemas that no prerequisite check reads.

Runs five kubectl queries. The first one to fail fails the task, and query names which it was.

Runs on the node the task is delegated to (typically localhost), reaching the cluster through a kubeconfig.

Parameters

Notes

• Set run_once: true and delegate_to: localhost on the task.
• The module runs in check mode and reports changed: false, because reading a cluster changes nothing.
• requires.storage.accessModes in a prerequisites document is not reported here and cannot be: what a provisioner will actually bind is not discoverable without creating a PersistentVolumeClaim, which is not a thing a read-only module does. The prerequisites role skips that check and says so.
• deployments and statefulsets cover every namespace, and carry the same five fields each. A cluster running hundreds of workloads returns hundreds of entries.
• StatefulSets are reported separately because a GitOps engine is not all Deployments. Argo CD runs argocd-application-controller as a StatefulSet and the rest of itself as Deployments, so a check that read only deployments would report the component doing all of the reconciling as absent from a healthy cluster.

Example

- name: Read the cluster's prerequisite-bearing facts
  colonel_byte.zarf.zarf_cluster_info:
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true
  register: cluster

- name: Require a cluster new enough for the bundle
  ansible.builtin.assert:
    that:
      - cluster.kubernetes_semver is version('1.30.0', '>=')
    fail_msg: "the cluster is {{ cluster.kubernetes_version }}, and the bundle needs 1.30.0 or newer"

- name: Require an available helm-controller
  ansible.builtin.assert:
    that:
      - cluster.deployments | selectattr('namespace', 'equalto', 'flux-system')
        | selectattr('name', 'equalto', 'helm-controller')
        | selectattr('available', 'ge', 1) | list | length > 0
    fail_msg: no available helm-controller, so a HelmRelease would never be reconciled

- name: Require an available argocd-application-controller, which is a StatefulSet
  ansible.builtin.assert:
    that:
      - cluster.statefulsets | selectattr('namespace', 'equalto', 'argocd')
        | selectattr('name', 'equalto', 'argocd-application-controller')
        | selectattr('available', 'ge', 1) | list | length > 0
    fail_msg: no available argocd-application-controller, so an Application would never be reconciled

- name: Report what is there
  ansible.builtin.debug:
    msg:
      kubernetes: "{{ cluster.kubernetes_version }}"
      defaultStorageClass: "{{ cluster.default_storage_class or 'none' }}"
      fluxCRDs: "{{ cluster.crds | select('search', 'toolkit.fluxcd.io$') | list | length }}"

init

zarf_init

Initialise a cluster with a staged zarf init package.

Write it as colonel_byte.zarf.zarf_init. It runs zarf on the node the task is delegated to; nothing is installed on the cluster’s nodes.

Added in colonel_byte.zarf 0.0.1.

Runs zarf init against the cluster a kubeconfig names, and follows the init package’s deployment component by component.

The module is a compiled binary with zarf linked into it. It renders the zarf init command line from these parameters, executes zarf’s own init command in its own process, reads the component boundaries back out of zarf’s log stream, and reports one JSON result. It needs no zarf on the node.

This is a proof of ZEP-0072 (https://github.com/zarf-dev/proposals/pull/73), which proposes putting this dispatch inside the zarf binary itself.

Parameters

Notes

• Set run_once: true and delegate_to on the task. One zarf init initialises the whole cluster, so a play over the cluster’s own inventory would otherwise initialise it once per host.
• Set no_log: true on the task. A binary module has no per-parameter no_log, so the task-level setting is what suppresses the arguments and the result.
• Check mode reports the task as skipped. zarf init has no dry run, so there is nothing honest to report instead.
• A parameter left unset renders no flag, so whatever zarf’s own config file sets still applies. That is why plain_http: false and omitting plain_http differ.
• Credentials given as parameters are rendered as flags, which puts them in the process table for the life of the run. Put them in a zarf config file and name it with zarf_config to keep them out of it.
• The zarf that does the work is linked into the module, so there is no version to check and nothing to install. It is reported as zarf.version in the result.

Example

- name: Initialise the cluster
  colonel_byte.zarf.zarf_init:
    init_package: /srv/staging/zarf-init-amd64-v0.85.0.tar.zst
    kubeconfig: /etc/rancher/rke2/rke2.yaml
    storage_class: local-path
    components: zarf-registry,zarf-agent
    timeout: 30m
  delegate_to: localhost
  run_once: true
  no_log: true
  register: init_result

- name: Report which components the init package deployed
  ansible.builtin.debug:
    var: init_result.zarf.componentsRan

- name: Initialise the cluster with credentials zarf reads from its own config file
  colonel_byte.zarf.zarf_init:
    init_package: /srv/staging/zarf-init-amd64-v0.85.0.tar.zst
    kubeconfig: /etc/rancher/rke2/rke2.yaml
    zarf_config: /etc/zarf/zarf-config.yaml
  delegate_to: localhost
  run_once: true
  no_log: true

- name: Initialise the cluster from a custom init package under a name of its own
  colonel_byte.zarf.zarf_init:
    init_package: /srv/staging/our-init-package.tar.zst
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true
  no_log: true

- name: Initialise the cluster from an init package held in a registry
  colonel_byte.zarf.zarf_init:
    init_package: oci://registry.bubbles.test/zarf/init:v0.85.0
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true
  no_log: true

kubectl

zarf_kubectl

Run a kubectl command through the kubectl zarf carries.

Write it as colonel_byte.zarf.zarf_kubectl. It runs zarf on the node the task is delegated to; nothing is installed on the cluster’s nodes.

Added in colonel_byte.zarf 0.88.0.

Runs zarf tools kubectl with the arguments given, against the cluster a kubeconfig names, and reports what it wrote to standard output.

The module is a compiled binary with zarf linked into it, and zarf vendors kubectl. It needs no zarf and no kubectl on the node, which is what it is for: a management node that drives an air-gapped cluster can read and apply resources without a kubectl of its own.

It is a passthrough and renders nothing into args. Zarf already vendors kubectl and already resolves which cluster to talk to, so projecting kubectl’s flags into Ansible parameters would be reimplementing kubectl in YAML. For declarative resource management prefer kubernetes.core.k8s, which this is not a replacement for.

Parameters

Notes

• Set run_once: true and delegate_to on the task. One command runs against the whole cluster, so a play over the cluster’s own inventory would otherwise run it once per host.
• The task always reports changed: true, with zarf.changedSignal set to unknown. The argument vector is kubectl’s, so the module knows the command succeeded and nothing about whether it altered anything. Set changed_when: false on a read.
• Check mode reports the task as skipped, for the same reason: nothing here can tell a get from an apply.
• Standard output comes back as zarf.output, with its trailing newline removed. Use -o json or -o yaml and from_json or from_yaml when a playbook has to read the result rather than show it.
• A failure comes back as failed with kubectl’s own message in msg and its exit status in zarf.exitCode, including the failures kubectl would normally exit the process over – a cluster it cannot reach is one.
• A command kubectl does not have is refused by name before anything runs, rather than printing kubectl’s help and reporting success.
• kubectl plugins are not reachable, deliberately: the handler that resolves them can replace this process with the plugin, or exit it without a result.

Example

- name: Read the nodes of an air-gapped cluster without installing kubectl
  colonel_byte.zarf.zarf_kubectl:
    args:
      - get
      - nodes
      - -o
      - json
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true
  changed_when: false
  register: nodes_result

- name: Report how many nodes are ready
  ansible.builtin.debug:
    msg: "{{ (nodes_result.zarf.output | from_json)['items'] | length }} nodes"

- name: Restart the zarf agent after rotating its certificate
  colonel_byte.zarf.zarf_kubectl:
    args:
      - rollout
      - restart
      - deployment/agent-hook
      - -n
      - zarf
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true

package_deploy

zarf_package_deploy

Deploy a zarf package onto an initialised cluster.

Write it as colonel_byte.zarf.zarf_package_deploy. It runs zarf on the node the task is delegated to; nothing is installed on the cluster’s nodes.

Added in colonel_byte.zarf 0.0.1.

Runs zarf package deploy against the cluster a kubeconfig names, and follows the deployment component by component.

The module is a compiled binary with zarf linked into it. It renders the zarf package deploy command line from these parameters, executes zarf’s own command in its own process, reads the component boundaries back out of zarf’s log stream, and reports one JSON result. It needs no zarf on the node.

It is the companion of colonel_byte.zarf.zarf_init. A cluster whose init package has no StorageClass to claim cannot finish initialising until a storage provider is deployed, and the provider cannot be deployed until the cluster is initialised far enough to hold it, so a playbook alternates between the two.

Parameters

Notes

• Set run_once: true and delegate_to on the task. One deploy puts the package on the whole cluster, so a play over the cluster’s own inventory would otherwise deploy it once per host.
• Set no_log: true on the task when any parameter carries a credential. A binary module has no per-parameter no_log.
• Check mode reports the task as skipped. zarf package deploy has no dry run, so there is nothing honest to report instead.
• A parameter left unset renders no flag, so whatever zarf’s own config file sets still applies.
• The zarf that does the work is the one linked into the module, and its version is reported as zarf.version, so a failed run says which zarf produced it. There is no installed version to be too old.

Example

- name: Deploy a storage provider so the registry has a StorageClass to claim
  colonel_byte.zarf.zarf_package_deploy:
    package: oci://ghcr.io/colonel-byte/zarf/csi-local-path-provider:0.0.37-upstream
    components: local-path-images,local-path-chart
    kubeconfig: /etc/rancher/rke2/rke2.yaml
    public_key: /etc/zarf/colonel-byte-zarf-packages.pub
    verify: always
  delegate_to: localhost
  run_once: true

- name: Deploy a staged package
  colonel_byte.zarf.zarf_package_deploy:
    package: /srv/staging/zarf-package-monitoring-amd64.tar.zst
    kubeconfig: /etc/rancher/rke2/rke2.yaml
    set_variables:
      DOMAIN: bubbles.test
  delegate_to: localhost
  run_once: true
  no_log: true
  register: deploy_result

- name: Report which components the package deployed
  ansible.builtin.debug:
    var: deploy_result.zarf.componentsRan

package_documentation

zarf_package_documentation

Extract and parse the documentation a Zarf package carries.

Write it as colonel_byte.zarf.zarf_package_documentation. It runs zarf on the node the task is delegated to; nothing is installed on the cluster’s nodes.

Added in colonel_byte.zarf 0.33.0.

Runs zarf package inspect documentation against a local package tarball, an oci:// reference, or a deployed package name, and returns the extracted files.

Zarf packages every file named in its zarf.yaml’s top-level documentation: mapping into a documentation layer, which this module extracts without deploying anything.

Each extracted file is returned as text in documents, keyed by its documentation key, and parsed into parsed when it is YAML or JSON. A file that is neither is returned as text only, rather than failing the task.

The prerequisites key is the one this collection has a format for. When the package carries one and it declares apiVersion: colonel-byte.dev/v1alpha1 and kind: PackagePrerequisites, it is also returned as prerequisites; see the prerequisites role, which checks a cluster against it.

Signature verification is zarf’s if-possible unless verify says otherwise, so an unsigned package extracts cleanly even when public_key is given.

Runs on the node the task is delegated to (typically localhost).

Parameters

Notes

• Set run_once: true and delegate_to: localhost on the task.
• The module runs in check mode and reports changed: false. It writes files, but only the ones it extracts, and only under destination or a temporary directory of its own - nothing on the cluster or in the package changes.
• Left unset, destination is a temporary directory that is removed before the task returns, so files names paths that no longer exist. Set it when a later task has to read a file rather than its contents.

Example

- name: Read the prerequisites a package declares
  colonel_byte.zarf.zarf_package_documentation:
    package: /srv/staging/zarf-package-operator-external-secrets-amd64-2.12.0.tar.zst
    keys:
      - prerequisites
    public_key: /etc/zarf/colonel-byte-zarf-packages.pub
    verify: always
  delegate_to: localhost
  run_once: true
  register: pkg_docs

- name: Report the Flux version the package requires
  ansible.builtin.debug:
    msg: "{{ pkg_docs.prerequisites.metadata.package }} requires Flux {{ pkg_docs.prerequisites.requires.fluxcd.version }}"
  when: pkg_docs.prerequisites.requires.fluxcd is defined

- name: Keep the extracted documentation for a later task to read
  colonel_byte.zarf.zarf_package_documentation:
    package: oci://ghcr.io/colonel-byte/packages/app-fluxcd:2.9.5
    destination: /srv/staging/documentation
  delegate_to: localhost
  run_once: true
  register: pkg_docs

- name: Show where it landed
  ansible.builtin.debug:
    msg: "{{ pkg_docs.directory }} holds {{ pkg_docs.files | dict2items | map(attribute='value') | list | join(', ') }}"

package_info

zarf_package_info

List all Zarf packages deployed to a cluster.

Write it as colonel_byte.zarf.zarf_package_info. It runs zarf on the node the task is delegated to; nothing is installed on the cluster’s nodes.

Added in colonel_byte.zarf 0.31.0.

Retrieves the list of packages deployed to the cluster via zarf package list --output-format json and returns structured package records.

Runs on the node the task is delegated to (typically localhost), reaching the cluster through a kubeconfig.

Parameters

Notes

• Set run_once: true and delegate_to: localhost on the task.
• The module runs in check mode and reports changed: false, because reading a cluster changes nothing.

Example

- name: List deployed zarf packages
  colonel_byte.zarf.zarf_package_info:
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true
  register: zarf_packages_result

- name: Display package names and versions
  ansible.builtin.debug:
    msg: "{{ zarf_packages_result.packages | map(attribute='package') | list }}"

package_inspect

zarf_package_inspect

Inspect and parse the definition of a Zarf package.

Write it as colonel_byte.zarf.zarf_package_inspect. It runs zarf on the node the task is delegated to; nothing is installed on the cluster’s nodes.

Added in colonel_byte.zarf 0.31.0.

Runs zarf package inspect definition against a local package tarball, an oci:// reference, or a deployed package name.

Signature verification is zarf’s if-possible unless verify says otherwise, so an unsigned package inspects cleanly even when public_key is given.

Parses the resulting YAML definition into structured output containing package metadata, build data, and components.

Runs on the node the task is delegated to (typically localhost).

Parameters

Notes

• Set run_once: true and delegate_to: localhost on the task.
• The module runs in check mode and reports changed: false, because reading a package definition changes nothing.

Example

- name: Inspect a staged Zarf package tarball, rejecting it when unsigned
  colonel_byte.zarf.zarf_package_inspect:
    package: /srv/staging/zarf-package-csi-rook-ceph-amd64-v1.20.7-upstream.tar.zst
    public_key: /etc/zarf/colonel-byte-zarf-packages.pub
    verify: always
  delegate_to: localhost
  run_once: true
  register: pkg_inspect

- name: Report the documentation the package carries
  ansible.builtin.debug:
    msg: "the package declares {{ pkg_inspect.package_documentation.keys() | list | join(', ') or 'no documentation' }}"

- name: Display package version and flavor
  ansible.builtin.debug:
    msg:
      name: "{{ pkg_inspect.definition.metadata.name }}"
      version: "{{ pkg_inspect.definition.metadata.version }}"
      flavor: "{{ pkg_inspect.definition.build.flavor | default('default') }}"

package_mirror_resources

zarf_package_mirror_resources

Push a package’s images and repositories into a registry and git server.

Write it as colonel_byte.zarf.zarf_package_mirror_resources. It runs zarf on the node the task is delegated to; nothing is installed on the cluster’s nodes.

Added in colonel_byte.zarf 0.88.0.

Runs zarf package mirror-resources, which pushes the images and git repositories a package carries into a registry and a git server without deploying anything.

The module is a compiled binary with zarf linked into it. It renders the zarf package mirror-resources command line from these parameters, executes zarf’s own command in its own process, and reports one JSON result. It needs no zarf on the node.

It is the air-gap seeding step. A cluster that deploys with connected: true pulls the images itself, which means its registry has to hold them first, and before this module a playbook had no way to say so.

Parameters

Notes

• Set run_once: true and delegate_to on the task. One mirror fills the registry the whole cluster pulls from.
• Set no_log: true on the task when any push credential is given as a parameter, or name a zarf config file with zarf_config and keep them out of the argument vector.
• The task reports changed: true with zarf.changedSignal set to unknown, and names no components. Unlike a deploy, a mirror announces the resources it pushes rather than the component they came from – mirroring images, pushing image – so there is no component list to report and nothing that says whether the registry already held what was pushed.
• Check mode reports the task as skipped. The command has no dry run.
• images and repos are mutually exclusive, as they are in zarf. Setting both is an error naming them; leaving both unset mirrors everything.
• A parameter left unset renders no flag, so whatever zarf’s own config file sets still applies.

Example

- name: Seed the cluster's registry so a connected deploy can find the images
  colonel_byte.zarf.zarf_package_mirror_resources:
    package: /srv/staging/zarf-package-monitoring-amd64.tar.zst
    kubeconfig: /etc/rancher/rke2/rke2.yaml
    public_key: /etc/zarf/colonel-byte-zarf-packages.pub
    verify: always
  delegate_to: localhost
  run_once: true
  register: mirror_result

- name: Deploy without pushing the images again
  colonel_byte.zarf.zarf_package_deploy:
    package: /srv/staging/zarf-package-monitoring-amd64.tar.zst
    connected: true
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true

- name: Push only the images, to a registry outside the cluster
  colonel_byte.zarf.zarf_package_mirror_resources:
    package: oci://ghcr.io/colonel-byte/zarf/monitoring:1.0.0
    images: true
    registry_url: registry.bubbles.test
    registry_push_username: zarf-push
    registry_push_password: "{{ registry_password }}"
  delegate_to: localhost
  run_once: true
  no_log: true

- name: Report the command the mirror ran
  ansible.builtin.debug:
    var: mirror_result.zarf.command

package_pull

zarf_package_pull

Fetch a zarf package to disk.

Write it as colonel_byte.zarf.zarf_package_pull. It runs zarf on the node the task is delegated to; nothing is installed on the cluster’s nodes.

Added in colonel_byte.zarf 0.88.0.

Runs zarf package pull to fetch a package from an oci:// registry or an https:// URL and write it to a directory as a tarball.

The module is a compiled binary with zarf linked into it. It renders the zarf package pull command line from these parameters, executes zarf’s own command in its own process, and reports one JSON result. It needs no zarf on the node.

It is the staging half of an air-gapped workflow: a connected node fetches the package, something carries the file through the airlock, and colonel_byte.zarf.zarf_package_deploy reads it from the path it landed at. Nothing it does reaches a cluster.

Parameters

Notes

• Set run_once: true and delegate_to on the task. The package is fetched once, to the node holding the staging directory.
• changed is an observation here rather than a convention: the task reports changed only when the pull left a package file the output directory did not already hold, with zarf.changedSignal set to complete and the filenames in zarf.packagesPulled. Zarf derives the filename from the package’s own metadata, so that list is how a playbook learns what to deploy.
• Check mode reports the task as skipped. zarf package pull has no dry run.
• A package that is a local path is refused: there is nothing to fetch. Use the path directly with colonel_byte.zarf.zarf_package_deploy.
• A parameter left unset renders no flag, so whatever zarf’s own config file sets still applies.

Example

- name: Stage a package for the airlock
  colonel_byte.zarf.zarf_package_pull:
    package: oci://ghcr.io/colonel-byte/zarf/csi-local-path-provider:0.0.37-upstream
    output_directory: /srv/staging
  delegate_to: localhost
  run_once: true
  register: pull_result

- name: Deploy what was staged
  colonel_byte.zarf.zarf_package_deploy:
    package: "/srv/staging/{{ pull_result.zarf.packagesPulled | first }}"
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true
  when: pull_result.zarf.packagesPulled | length > 0

package_remove

zarf_package_remove

Remove a deployed zarf package from a cluster.

Write it as colonel_byte.zarf.zarf_package_remove. It runs zarf on the node the task is delegated to; nothing is installed on the cluster’s nodes.

Added in colonel_byte.zarf 0.88.0.

Runs zarf package remove against the cluster a kubeconfig names, taking a deployed package – or some of its components – off it.

The module is a compiled binary with zarf linked into it. It renders the zarf package remove command line from these parameters, executes zarf’s own command in its own process, and reports one JSON result. It needs no zarf on the node.

It is the other half of colonel_byte.zarf.zarf_package_deploy. A playbook that can only deploy is not reconciling a cluster, and removing a component of a package while leaving the rest is the one thing a deploy cannot undo.

Parameters

Notes

• Set run_once: true and delegate_to on the task. One removal takes the package off the whole cluster, so a play over the cluster’s own inventory would otherwise run it once per host.
• --confirm is always rendered. There is no terminal on the other end of a playbook task to confirm at, and this is the command where that matters most: the task itself is the confirmation that something should be deleted.
• The task reports changed: true with zarf.changedSignal set to unknown. Zarf logs a removal as one line and names no component on the way, so there is nothing that distinguishes a package that was there from one that was already gone – a removal that finds nothing fails in zarf, which is the only signal there is.
• Check mode reports the task as skipped. zarf package remove has no dry run, and removing a package during a run an operator asked to be told about is worse than deploying one.
• A parameter left unset renders no flag, so whatever zarf’s own config file sets still applies.

Example

- name: Remove a deployed package by name
  colonel_byte.zarf.zarf_package_remove:
    package: csi-local-path-provider
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true

- name: Remove one component and leave the rest of the package in place
  colonel_byte.zarf.zarf_package_remove:
    package: monitoring
    components: grafana-dashboards
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true
  register: remove_result

- name: Report the command that ran
  ansible.builtin.debug:
    var: remove_result.zarf.command

state_info

zarf_state_info

Fetch and parse the Zarf state secret from a cluster.

Write it as colonel_byte.zarf.zarf_state_info. It runs zarf on the node the task is delegated to; nothing is installed on the cluster’s nodes.

Added in colonel_byte.zarf 0.31.0.

Retrieves the zarf-state Secret from the cluster via zarf tools kubectl and parses the base64-encoded state JSON into structured output.

Runs on the node the task is delegated to (typically localhost), reaching the cluster through a kubeconfig.

The Secret holds the registry, git and artifact server credentials and the agent’s TLS private key, so by default they are removed from state and the paths that were removed are returned in redacted. Set include_credentials: true to get them.

Parameters

Notes

• Set run_once: true and delegate_to: localhost on the task.
• The returned state is redacted unless include_credentials is set, because zarf writes eight secrets into that one Secret: the registry push, pull and seed secrets, the git server push and pull passwords, the artifact server token, and the agent webhook’s TLS private key.
• A run with include_credentials: true censors its own task output, as though no_log: true had been set on the task. The registered variable still holds the real state; it is the display that is suppressed. Note that a later task which templates the state - set_fact in particular - prints what this one hid, so it needs no_log: true of its own.
• The module runs in check mode and reports changed: false, because reading state changes nothing.

Example

- name: Fetch and inspect the zarf state
  colonel_byte.zarf.zarf_state_info:
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true
  register: zarf_state_result

- name: Display registry mode
  ansible.builtin.debug:
    msg: "{{ zarf_state_result.state.registryInfo.registryMode }}"

- name: Report which paths were withheld
  ansible.builtin.debug:
    var: zarf_state_result.redacted

- name: Read the registry push credentials to log in with them
  colonel_byte.zarf.zarf_state_info:
    kubeconfig: /etc/rancher/rke2/rke2.yaml
    include_credentials: true
  delegate_to: localhost
  run_once: true
  register: zarf_state_credentials

update_creds

zarf_update_creds

Rotate the credentials an initialised cluster holds.

Write it as colonel_byte.zarf.zarf_update_creds. It runs zarf on the node the task is delegated to; nothing is installed on the cluster’s nodes.

Added in colonel_byte.zarf 0.88.0.

Runs zarf tools update-creds <service> against the cluster a kubeconfig names, rewriting the credentials it holds for its registry, its git server or its agent.

The module is a compiled binary with zarf linked into it. It renders the zarf tools update-creds command line from these parameters, executes zarf’s own command in its own process, and reports one JSON result. It needs no zarf on the node.

It is the write half of colonel_byte.zarf.zarf_state_info, which reads the same zarf-state Secret and withholds what it finds. Rotating a password without it means doing in several steps what zarf does in one: the Secret, the service’s own configuration, and every deployed package that pulls through it.

Zarf has one subcommand per service, each taking only that service’s credentials, and the bare zarf tools update-creds that takes all of them warns that it is deprecated. This module renders the subcommand, so service is required and a credential given for another service is an error naming it.

Parameters

Notes

• Set run_once: true and delegate_to on the task. The credentials belong to the cluster, so a play over its own inventory would otherwise rotate them once per host.
• Set no_log: true on the task. Every parameter here but service is a credential or a path to one, and a binary module has no per-parameter no_log. Better still, put them in a zarf config file and name it with zarf_config, which keeps them out of the argument vector and so out of the process table.
• --confirm is always rendered: there is no terminal on the other end of a playbook task, and the command asks before it rewrites anything.
• The task reports changed: true with zarf.changedSignal set to unknown. Zarf reports that it updated the services, not whether any of them already held what was given, so a credential set twice is reported as changed both times.
• Check mode reports the task as skipped. The command has no dry run.
• The artifact server is the fourth service zarf keeps credentials for and has no subcommand of its own. It is deprecated upstream, so this module does not update it.
• One service per task: a playbook rotating the registry and the git server writes two tasks.

Example

- name: Rotate the registry's push password
  colonel_byte.zarf.zarf_update_creds:
    service: registry
    registry_push_password: "{{ new_registry_password }}"
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true
  no_log: true

- name: Rotate the git credentials from a zarf config file, keeping them out of the process table
  colonel_byte.zarf.zarf_update_creds:
    service: git
    zarf_config: /etc/zarf/rotation.yaml
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true

- name: Install a new agent certificate
  colonel_byte.zarf.zarf_update_creds:
    service: agent
    agent_tls_ca: /etc/zarf/ca.pem
    agent_tls_cert: /etc/zarf/tls.pem
    agent_tls_key: /etc/zarf/tls.key
    kubeconfig: /etc/rancher/rke2/rke2.yaml
  delegate_to: localhost
  run_once: true
  no_log: true

Roles

deploy_packages

deploy_packages deploys a sequence of Zarf packages while inspecting each package definition and checking the cluster’s deployed components to avoid unnecessary deployments.

Skipping settles for a package that one item deploys. It does not settle for a package that several items deploy in phases – the bundle’s two-phase init, for instance – because zarf package list reports only the most recent deploy’s components, so each phase sees the other’s and both re-deploy on every run. Re-deploying reconciles rather than fails, but it is never reported as unchanged. choice-deploy-packages-drift-scope has the reasoning and the alternatives that were declined.

An item is also only skipped when the cluster’s version of the package matches the one the item carries, so an upgraded tarball whose component names did not change is not mistaken for what is already deployed.

Skipping is per component, and an item that names no components is deployed every run. That is a limit rather than an oversight: with no components named, zarf picks the set itself – every non-optional component, plus any optional one marked default, with required and optional carrying inverted defaults between the v1alpha1 and v1beta1 package schemas – and the role does not reimplement that choice to guess what it would be. Naming the components is what turns drift detection on.

The role sets zarf_deploy_packages_changed to whether any item reached zarf, and refreshes the zarf_packages fact once at the end when one did.

Set zarf_result_dir and each item’s result is written to <position>-<package file name>.json inside it. The position leads the name because the package does not identify the item – a bundle names the same init package twice, once for the seed phase and once for the rest – and it puts a listing of the directory in deployment order. log_dest on an item overrides the whole path.

registry_mode, connected, take_ownership and force_conflicts are settings the role passes through rather than decides. Each has a role-level default (zarf_registry_mode, zarf_connected, zarf_take_ownership, zarf_force_conflicts) that a package item can override, and each is unset by default – which leaves zarf’s own behaviour alone. Turning force_conflicts on means a deploy takes fields another controller owns, so the role will not do it on an operator’s behalf.

The boolean item keys below take true or false. Their role-level counterparts are typed str with an empty choice instead, because a role default needs a third state – empty means “say nothing and leave zarf’s own behaviour alone”, which false cannot express.

Each dictionary entry in zarf_packages_to_deploy supports:

Example:

- name: Deploy package stack
  ansible.builtin.include_role:
    name: colonel_byte.zarf.deploy_packages
  vars:
    zarf_kubeconfig: /etc/rancher/rke2/rke2.yaml
    zarf_packages_to_deploy:
      - path: "{{ zarf_package_dir }}/zarf-init-amd64-v0.88.0.tar.zst"
        package_type: init
        components:
          - k3s
      - path: "{{ zarf_package_dir }}/zarf-package-csi-rook-ceph-amd64-v1.21.0-upstream.tar.zst"
        components:
          - operator-images
          - cluster-images
          - operator-seed-chart
          - cluster-seed-chart
          - registry-bucket
        package_type: package
        set_variables: "{{ zarf_storage_set_variables }}"

packages

packages queries installed Zarf packages and exposes them as a structured list under the zarf_packages fact:

- name: List installed Zarf packages
  ansible.builtin.include_role:
    name: colonel_byte.zarf.packages
  vars:
    zarf_kubeconfig: /etc/rancher/rke2/rke2.yaml

- name: Report installed packages
  ansible.builtin.debug:
    msg: "{{ zarf_packages | map(attribute='package') | list }}"

package_inspect

package_inspect parses a package tarball, OCI reference, or cluster package definition and sets facts.

Naming zarf_public_key requires a signature. That is the role being stricter than the module, which passes verify through and leaves zarf’s own if-possible default alone – and if-possible accepts an unsigned package whether or not a key was given. Set zarf_verify explicitly to override, including to if-possible to get zarf’s default back.

- name: Inspect storage package definition, rejecting it when unsigned
  ansible.builtin.include_role:
    name: colonel_byte.zarf.package_inspect
  vars:
    zarf_package: /srv/staging/zarf-package-csi-rook-ceph-amd64-v1.20.7-upstream.tar.zst
    zarf_public_key: /etc/zarf/colonel-byte-zarf-packages.pub

- name: Report package details
  ansible.builtin.debug:
    msg:
      name: "{{ zarf_package_name }}"
      version: "{{ zarf_package_version }}"
      flavor: "{{ zarf_package_flavor }}"

prerequisites

prerequisites is the check that belongs in front of zarf_package_deploy. It reads the PackagePrerequisites document a package carries in its documentation: mapping, reads the cluster, and fails the play when the cluster does not satisfy it.

It exists for one failure in particular. A workload package delivered as a HelmRelease needs a running helm-controller to do anything at all; on a cluster without one, zarf writes the manifests, reports the deployment as a success, and nothing is ever reconciled. The only symptom is a HelmRelease that never becomes ready.

Three facts come out of it, and the third matters as much as the other two. zarf_prerequisites_checked is what was checked and held, zarf_prerequisites_unmet is what did not, and zarf_prerequisites_skipped is what could not be checked at all – a compound version range the role will not half-evaluate, or an accessModes requirement that cannot be verified without creating a PersistentVolumeClaim. A check that cannot be made is named rather than quietly passed.

Every unmet prerequisite is reported at once, from one assert at the end, so an operator who has to install Flux and a storage provider is told both rather than told one, fixing it, and being told the next.

- name: Refuse to deploy onto a cluster that cannot run this package
  ansible.builtin.include_role:
    name: colonel_byte.zarf.prerequisites
  vars:
    zarf_package: /srv/staging/zarf-package-operator-external-secrets-amd64-2.12.0.tar.zst
    zarf_public_key: /etc/zarf/colonel-byte-zarf-packages.pub
    zarf_kubeconfig: /etc/rancher/rke2/rke2.yaml

- name: Report what could not be checked
  ansible.builtin.debug:
    msg: "{{ zarf_prerequisites_skipped }}"
  when: zarf_prerequisites_skipped | length > 0

A package that carries no document is not a failure: nothing is checked, zarf_prerequisites_checked is empty, and the play continues. Set zarf_prerequisites_enforce: false to report rather than enforce, which is what a survey across a fleet of clusters wants and what a deployment does not. ansible/examples/playbooks/zarf-prerequisites.yml deploys a two-package bundle this way.

state

state is the convenient way to retrieve and parse Zarf’s cluster state. It runs zarf tools kubectl get secret zarf-state -n zarf -o jsonpath='{.data.state}', decodes the payload, and sets the resulting dictionary as the zarf_state fact.

The credentials in that Secret – the registry push, pull and seed secrets, the git server passwords, the artifact server token, and the agent webhook’s TLS private key – are withheld. The paths withheld are set as zarf_state_redacted, and zarf_include_credentials: true returns them instead. The role defaults no_log on both of its tasks, which is also why reaching for the role beats calling the module directly here: set_fact prints what the module censored, so the module alone cannot keep a credential out of a -v transcript. See choice-zarf-info-modules.

- name: Load cluster zarf state
  ansible.builtin.include_role:
    name: colonel_byte.zarf.state
  vars:
    zarf_kubeconfig: /etc/rancher/rke2/rke2.yaml

- name: Inspect cluster registry mode
  ansible.builtin.debug:
    msg: "Cluster registry mode is {{ zarf_state.registryInfo.registryMode }}"

- name: Report which paths were withheld
  ansible.builtin.debug:
    var: zarf_state_redacted

deploy_packages

deploy_packages

Deploy a list of Zarf packages to a cluster with component-level drift detection.

Include it as colonel_byte.zarf.deploy_packages.

Added in colonel_byte.zarf 0.31.0.

Iterates over a list of Zarf packages to deploy, inspecting each package definition and checking installed cluster components before deploying.

An item that names components is skipped when every one of them is already deployed on the cluster at the same package version, and no excluded component is. An item that names none is always deployed: the role compares component names, and with none to compare it has nothing to go on.

Skipping settles for a package that one item deploys. It does not settle for a package that several items deploy in phases, because zarf package list reports only the most recent deploy’s components – so each phase sees the other’s and both re-deploy every run. That reconciles rather than fails; see the choice-deploy-packages-drift-scope page in the book.

When deployment is executed, invokes zarf_init for packages of type init, or zarf_package_deploy for standard packages. Both verify the package against zarf_public_key when one is named.

Optionally writes deployment results as JSON files to zarf_result_dir, refreshes the zarf_packages fact once the stack is deployed, and sets zarf_deploy_packages_changed to whether any item reached zarf.

Every task carries run_once: true and delegates to zarf_delegate_to.

Variables

package_inspect

package_inspect

Inspect a Zarf package definition and expose structured metadata facts.

Include it as colonel_byte.zarf.package_inspect.

Added in colonel_byte.zarf 0.31.0.

Runs zarf package inspect definition against a package source and parses the YAML definition.

A zarf_public_key with no zarf_verify requires a signature, which is stricter than the module’s own default. See zarf_verify.

Sets the zarf_package_definition, zarf_package_name, zarf_package_version, zarf_package_flavor, and zarf_package_documentation facts.

zarf_package_documentation is the package’s documentation: mapping: documentation keys to paths inside the package’s own source tree. It is empty for a package that declares none. Reading the file one of them names is colonel_byte.zarf.zarf_package_documentation, and checking a cluster against a prerequisites key is the prerequisites role.

Every task carries run_once: true and delegates to zarf_delegate_to.

Variables

packages

packages

List all Zarf packages deployed to a cluster into an Ansible fact.

Include it as colonel_byte.zarf.packages.

Added in colonel_byte.zarf 0.31.0.

Queries the cluster via zarf package list and sets the resulting package list as the zarf_packages fact.

Every task carries run_once: true and delegates to zarf_delegate_to.

Variables

prerequisites

prerequisites

Check a cluster against the prerequisites a Zarf package declares.

Include it as colonel_byte.zarf.prerequisites.

Added in colonel_byte.zarf 0.33.0.

Reads a colonel-byte.dev/v1alpha1 PackagePrerequisites document out of a package’s documentation layer, reads the cluster with colonel_byte.zarf.zarf_cluster_info, and fails the play when the cluster does not satisfy the document.

This is the check that belongs in front of colonel_byte.zarf.zarf_package_deploy. A workload package deployed onto a cluster whose GitOps engine is absent is accepted, reported as a success, and never reconciled; the point of the document is to turn that into a refusal.

Supply the package in zarf_package and the role extracts its document, or supply the document directly in zarf_prerequisites for a package that carries none.

Sets the zarf_prerequisites_document, zarf_prerequisites_checked, zarf_prerequisites_unmet, and zarf_prerequisites_skipped facts. A package carrying no document is not a failure: nothing is checked and zarf_prerequisites_checked is empty.

requires.storage.accessModes is reported in zarf_prerequisites_skipped rather than checked. What a provisioner will bind is not discoverable without creating a PersistentVolumeClaim, and a check that cannot be made is better named than quietly passed.

Every task carries run_once: true and delegates to zarf_delegate_to.

Variables

state

state

Fetch and parse the Zarf state secret from a cluster into an Ansible fact.

Include it as colonel_byte.zarf.state.

Added in colonel_byte.zarf 0.31.0.

Queries the cluster for the zarf-state Secret via zarf tools kubectl and sets the parsed state as the zarf_state fact.

The credentials the Secret holds are withheld unless zarf_include_credentials is set. The paths that were withheld are set as the zarf_state_redacted fact.

Every task carries run_once: true, no_log from zarf_no_log, and delegates to zarf_delegate_to.

Variables

End-to-end tests

There are two suites, split by whether they need a cluster. Both drive a real ansible-playbook, because every module here reaches zarf through an action plugin and only Ansible can run one. Neither runs under a bare go test ./...: both are behind mage targets, and every test skips with a reason when a prerequisite is missing rather than failing.

test/e2e/noncluster

Needs ansible-playbook, and a zarf on PATH for the tests that build a real package. It covers the five read-only modules, the state, package_inspect and prerequisites roles, and every acting module’s refusals and check-mode answers.

The one impure step in most read-only modules is running zarf, so the suite stands a stub in its place: testdata/zarf-state-stub.sh prints the base64 of a recorded zarf-state Secret, and zarf-package-list-stub.sh prints a recorded package list. The playbook passes the stub as the module’s zarf_binary.

testdata/zarf-cluster-stub.sh is the exception that reads its arguments. zarf_cluster_info asks the cluster four questions – the Kubernetes version, the CRDs, the StorageClasses and the Deployments – and a stub that printed one document whatever it was asked would show that the module can parse a document, not that it asked four questions and kept the answers apart. It fails loudly on a command it does not recognise, for the same reason: a silent empty answer is a cluster that looks empty, and an empty cluster satisfies no prerequisite.

The tests that need a real zarf build a package rather than committing one. testdata/zarf-inspect-fixture is the smallest package that builds – a files-only component needs no registry, no cluster and no network – and it carries a documentation: mapping with a PackagePrerequisites document and a Markdown file, which is the fixture for zarf_package_inspect, zarf_package_documentation and the prerequisites role. A .tar.zst is a binary blob nobody can review; the document in that directory is what the tests are actually about.

testdata/acting-modules.yml is the part that needs no stub, because the modules it drives are the wrapper binary with zarf inside it. It runs all seven acting modules through the paths that reach no cluster: what each one refuses before it runs anything – a pull given a local path, a mirror given both images and repos, a credential given for the wrong service, a kubectl command that does not exist – and what each one does in check mode. zarf_kubectl is the exception that runs for real: kubectl version --client reads nothing but the binary it is in, so the module whose answer is a vendored tool’s output is covered end to end here rather than only against a cluster.

That playbook is also where argv[0] dispatch is exercised through Ansible for every module at once: each task names a different zarf_<module> symlink, so a dispatch entry that went missing fails as the module for it. The suite builds the wrapper, and the test skips with a reason when it is not there, so go test over the package still works without the mage target.

The prerequisites test is the one combination worth knowing: the cluster is the stub, and the stub hands package inspect documentation to the real zarf. The role takes one zarf binary for both the extraction and the cluster reads, so it cannot be given a real one for half of them – and faking the extraction would leave the role’s own extraction branch untested, which is the branch an operator uses.

go run ./magefiles/core test:endToEndNonCluster

Two things about these tests are deliberate rather than incidental, and both are easy to undo by accident:

• They count the assertions that reported passing. A renamed fact or a skipped block leaves every remaining assertion passing, and failed=0 would call that a success. The expected count is read out of the playbook rather than written in the Go file, so adding an assertion does not quietly stop the count from meaning anything.
• They read the playbook, the fixture and the plugin before running anything. Go’s test cache keys on the files the test process opens, and everything these tests exercise is opened by the ansible-playbook child. Without those reads, editing a plugin leaves a passing result cached – which is the shape of mistake that gets a credential leak marked as fixed.

test/e2e/cluster

Needs k3d, a zarf on PATH, ansible-playbook, Docker, and the network on the first run – to pull three packages, and to build a fourth from examples/packages/app-fluxcd, whose four Flux controller images come from ghcr.io. It builds the module wrapper first, stands up two k3d clusters, and walks zarf_init and zarf_package_deploy across each. The other five acting modules run against the cluster the first walk leaves behind.

go run ./magefiles/core test:endToEndCluster

Both walks deploy the two packages of the init-local-path uds-bundle, pinned in main_test.go. The init package is that bundle’s slim init, whose five components are all optional, and that is what shapes the walk: each step names the components it wants, so the bundle’s alternation of seed phase, storage provider, the rest of init, storage provider again is four steps rather than two. bundle-order.yml adds a fifth the bundle has not got – the second init again, unchanged – because a playbook that cannot be re-run is not a playbook. See choice-zarf-ansible-module for the two tag traps that came with those packages.

Two clusters, because the walk differs by what storage the cluster has. testdata/k3d-config.yaml is the config from colonel-byte/uds-bundles copied verbatim, so that what the modules initialise is the cluster shape that repository already deploys onto; k3d-config-local-storage.yaml is the same file with k3s local-storage left enabled, which is what lets a walk begin with init at all. Each fixture carries a header saying which walk it serves and why. The reasoning behind needing both is in choice-zarf-ansible-module.

app-fluxcd is the one package the suite builds rather than pulls, because nothing publishes it: it is a reference package in this repository, and until now mage examples:lint was all that read it. It earns the build by being the only package here with a single component that is required: true, which is what lets a deploy that names no components deploy exactly one known thing – the deploy_packages role’s omitted-components path, which has no other way to be exercised against a real zarf. The suite finds what the build wrote by globbing for the architecture rather than naming the version, because metadata.version there is machine-maintained from resources.json.

Packages are cached under build/tmp/zarf-e2e between runs, so the downloads happen once. The init package is cached in a subdirectory of its own, which matters: zarf package pull names a ZarfInitConfig after its kind and version, so the slim init and the standard zarf-dev one both land as zarf-init-<arch>-v0.88.0.tar.zst, and a flat cache that had ever held the standard one would serve it to every walk. test:cleanClusters removes the clusters a run interrupted before its own cleanup left behind.

The acting modules the walks do not cover

testdata/acting-modules.yml runs after the bundle-order walk, on the cluster that walk initialised, and covers the half of those five modules that cannot be faked:

The order in that file is load-bearing and the Go side runs it last on that cluster: the removal takes away a package the read-only module tests expect to find, and the rotation rewrites credentials the cluster is using, so it goes last of all. Nothing may be added after it without a cluster of its own.

zarf_package_remove is measured by asking the cluster rather than by reading the module’s report, and that is the point of running it here: it reports changedSignal: unknown because zarf names no component on the way out, so zarf package list no longer holding the package is the only evidence there is. The check-mode rotation is checked the same way – the cluster’s registry credentials are still there and are not the value the task offered.

The rotation is asked for in check mode because neither subcommand can complete against the cluster this suite builds, and both reasons are zarf’s rather than the module’s. update-creds git updates the gitea users zarf-state names, and the read-only user it names is never created by the uds-bundle’s slim init: it fails with failed to update git user "zarf-git-read-user": unexpected status code 404 and its rollback fails after it, reporting the cluster may be in an inconsistent state. update-creds registry fails with error finding init package with registry service, because the walk’s second init records only the components that step deployed, so the deployed package zarf looks up for the registry is not the one that owns it. What a real cluster is left to prove is that the module reaches it and refuses to act; the rendering of all three subcommands is held against zarf’s own commands by TestUpdateCredsArgsAreKnownFlags, and the refusals by the non-cluster suite.

Two findings from this suite are worth knowing before changing anything it covers:

• --verify must be rendered as --verify=always. It is a flag with a no-argument default, so pflag reads it the way it reads a boolean: the space form leaves always as a positional argument and zarf rejects the command for having one argument too many.
• Local hooks do not reproduce the lint CI runs. .pre-commit-config.yaml pins the same golangci-lint as the workflow, v2.15.0, and prek run golangci-lint --all-files still passed a discarded type-assertion ok that the lint job rejected under errcheck. Treat a green hook as necessary and not sufficient, and read the lint check on the pull request.
• A mirror announces resources, not components. zarf_package_mirror_resources was written expecting the component announcements a deploy makes, so that the progress reader could follow it. This suite is what showed that it cannot: what zarf logs on the way through a mirror is mirroring images, pushing image and done pushing images, never the component they came from. The module reports changedSignal: unknown with no component list for that reason, and the e2e assertion is that the list is empty – a list appearing there would mean the progress reader had started matching something it should not.
• The init package is tracked under its own metadata name. zarf package list reports the uds-bundle’s slim init as init-slim, not as the init the zarf-dev package records, so the fixtures name that.
• A two-phase init needs git-server in the first phase. Zarf records the git server’s credentials in zarf-state only when the git-server component is in the set an init was asked for, and zarf init carries --git-push-username=zarf-git-user as a flag default whether or not anybody set it – a viper default another command registers, which zarf’s own root.go warns about leaking. A second init onto a cluster whose state holds no git server therefore reads as changing the git server and is refused with cannot change git server information after initial init. The walks deploy git-server in their seed step for that reason, which is a deviation from the bundle’s own split and is commented as one in both playbooks. Pinning the credentials does not help: after a seed-only init there is nothing in state for them to match.
• Registry proxy mode is what makes a local cluster usable. Zarf’s default nodeport mode pushes every image through a Kubernetes NodePort, and on k3d that push times out and retries for minutes. registry_mode: proxy puts a DaemonSet on a host port instead, and the same initialisation finishes in about a minute.

Mage targets

Build and test targets run through mage, not a Makefile. magefiles/ does not build with a plain go build – it needs mage’s own build tag – so drive it through the mage CLI, or through go run ./magefiles/core <namespace>:<target>. The second form needs only the Go toolchain and works on a host with no mage binary installed.

go run ./magefiles/core -l    # list every target

The dependency tree is vendored, so every target builds from vendor/ and needs no module proxy. CI passes -mod=vendor with GOPROXY=off explicitly, so that an accidental fetch fails fast instead of quietly succeeding on a machine that happens to be online. After changing go.mod, run go mod vendor and commit the result – check-go-mod.yaml fails the pull request otherwise.

Build

One binary under several names is what ZEP-0072 describes, and the names come from zarfmod.ModuleFiles() rather than a list in the target, so a module added to the dispatch table gets its symlink with nothing to keep in step. It is built for the host rather than for a release target list, because what drives it is the end-to-end suites and an operator reproducing a module run. The release builds come from .goreleaser.yaml. No version is stamped in: a module file reports what zarf did, and has no version of its own to report.

Test

See e2e-tests for what each suite covers.

Generate

generate:document rewrites generated pages in place. The hand-written overviews beside them – docs/ansible/collection.md, modules.md, roles.md – are left alone, which is why only the module_* and role_* patterns are cleared before a run.

generate:schema writes the two published JSON Schemas: the colonel-byte.dev/v1alpha1 PackagePrerequisites schema, from api/colonel-byte.dev/v1alpha1/prerequisites, and the schema of a reference package’s resources.json, from magefiles/pkg/pkgvendor. The descriptions in both are the doc comments on those types, so a reworded comment is a schema change. TestCommittedSchemasAreCurrent fails when a committed schema stops matching its type – a stale schema does not fail at build time, it validates the wrong document.

There is no generate:zarfFlags any more. It used to record the flag surface of the zarf commands the modules wrap into internal/zarfmod/testdata/, so their tests could check the rendered flags against a list. Both modules now link zarf’s command tree in, so TestInitArgsAreKnownFlags and TestDeployArgsAreKnownFlags parse what they render against the real commands and pflag decides whether a flag exists – which also catches a flag zarf renamed, the one thing a recorded list could not. The zarf they check against is the github.com/zarf-dev/zarf version in go.mod, which Dependabot watches. See choice-zarf-ansible-module.

generate:dnfPins moves the two dnf package pins the management node image installs. It fetches repomd.xml and primary.xml.gz from the AlmaLinux 10 AppStream and BaseOS repodata over HTTP, picks the newest ansible-core and bash-completion by RPM epoch-version-release ordering, and rewrites the ARG lines in containers/ansible/Dockerfile along with the zarf-ansible build_args in .goreleaser.yaml. Both files carry the same two versions on purpose – the ARG defaults are what a local build gets and the build_args are what the release build passes – so one run writes both, and a rewriter that no longer finds what it edits fails the target rather than leaving one of them behind. Dependabot’s docker ecosystem moves the two FROM digests and nothing else, which is why these pins need a target at all.

Read the diff rather than committing it unseen: a pin that jumped a minor line is a change to what the image installs, and the image’s own build is what proves it still works.

Examples

The reference packages under examples/packages are illustrative, and the parts of them that track an upstream release are machine-maintained. These are the targets that maintain them. Nothing in CI runs them: examples:vendor reaches the network, and examples:images and examples:lint need a zarf.

A package opts in by holding a resources.json, which declares the upstream repository, the pinned tag, where to read the manifests from, and which files go into which subdirectory and zarf component. app-fluxcd and app-argocd hold one; operator-external-secrets ships its own manifests and vendors nothing.

The two differ in where the manifests live, which is why both are here. Flux attaches manifests.tar.gz to every release, so app-fluxcd/resources.json names an asset. Argo CD attaches only CLI binaries and SBOMs and documents applying raw.githubusercontent.com/argoproj/argo-cd/<tag>/manifests/install.yaml instead, so app-argocd/resources.json names a tree – a directory in the tagged source tree, whose match entries are paths relative to it. Exactly one of the two is set, and a declaration naming both or neither is refused.

Moving a pin is a manual edit, which is the third kind of owner magefiles/AGENTS.md allows and the one these pins have: write the new tag into resources.json, then run the two targets below in that order and read the diff. Nothing here discovers the upstream’s latest release on its own – the pin is on a reference package nothing deploys, and moving it is a reviewed change rather than a thing that happens to a tree.

The order matters when a pin moves: examples:vendor writes the manifests, and examples:images discovers images from the manifests that are now there.

go run ./magefiles/core examples:vendor    # after editing tag in resources.json, or to re-apply it as it is
go run ./magefiles/core examples:images
go run ./magefiles/core examples:lint

magefiles/pkg/pkgvendor is where that lives, and it is a Go port of the shell in colonel-byte/zarf-pkg-fluxcd’s .github/hack. ZARF_ANSIBLE_FLAGS_BINARY names a zarf that is not on PATH. The name is the one the retired generate:zarfFlags read; these targets kept it rather than asking an operator to learn a second variable.

Check for go mod changes

Holds go.mod, go.sum and vendor/ to each other. Runs on every pull request and on the merge queue.

The job runs go mod tidy and fails on a diff to go.mod or go.sum, then runs go mod vendor and fails on a diff to vendor/.

The second half is the one worth knowing about. The vendored tree is committed and every build reads from it, so a go.mod change that was not re-vendored leaves the two disagreeing – and nothing local notices, because go build uses vendor/ quite happily with a dependency list that no longer matches the module file. The failure only surfaces when somebody drops vendor/ and gets a different build. This workflow is the only check that catches it, which is why a go.mod change is not finished until go mod vendor has run and the result is committed.

Dependabot’s gomod updates trip it until the branch is re-vendored.

CodeQL

GitHub’s static analysis, feeding the repository’s security tab. Runs on pushes to main, on pull requests, and weekly on a schedule – the scheduled run is what keeps the tab current for advisories published after a commit landed.

The merge queue is deliberately not a trigger. CodeQL gates no merge, so the queue does not wait for it and deletes the gh-readonly-queue ref as soon as the entry merges, leaving the slower Go analysis nowhere to upload its results; the pull request and the push to main cover the same commits either side of the queue.

Three languages are analysed in a matrix:

The Go build names its packages explicitly – go build -mod=vendor ./cmd/... ./internal/... – rather than using ./.... magefiles/ is package main behind //go:build mage, and mage supplies the main function at runtime, so go build cannot compile it at all, with or without -tags mage. Those targets are dev tooling and ship in no binary, so they are left unanalysed. CodeQL’s paths-ignore cannot express this: it does not apply to Go, which is analysed by being built, so the exclusion has to live in the build command.

The build runs with GOPROXY=off and GOFLAGS=-mod=vendor, matching unit-tests.yaml and e2e.yaml, so accidental network access fails fast.

PR Title Check

Lints the pull request title against commitlint, using the config in .github/commitlint/. It runs on a pull request opened, reopened, synchronized, edited or milestoned, and on the merge queue.

It lints the title, not the individual commits: release-please squash-merges, so the pull request title becomes the final commit message and is the thing that has to parse as a conventional commit. wagoid/commitlint-github-action lints commit messages instead, which is why it is not a drop-in replacement for the step here.

milestoned is in the trigger list as a workaround for release-please not triggering pull request workflows – adding its pull request to a milestone is what gets the check to run.

Under merge_group there is no pull request to read, so every step is skipped and the job reports success. That is deliberate: title_check is a required status check, so the merge queue has to see a result on the merge_group ref too, and the title was already linted on the pull request that got queued.

Dependabot gets .commitlintrc.dependabot.yaml, the same config with header-max-length turned off. It cannot shorten a module path, and ten of the modules in go.mod overflow a 100-character title. Skipping the job for Dependabot instead would leave title_check without a result, and that is a required check on main. See choice-dependabot-title-length.

dependabot validate

Validates .github/dependabot.yaml against Dependabot’s schema, and posts the result as a sticky comment on the pull request.

It is path-filtered: it runs only when the pull request touches .github/dependabot.yaml or the workflow itself.

This exists because GitHub reports a malformed Dependabot configuration on the repository’s Insights tab and nowhere a pull request author would look. A typo that stops an ecosystem from being updated is otherwise silent – the updates simply never arrive.

The comment step runs with if: always(), so a failed validation is reported in the comment rather than only in the job log.

GitHub Pages

Builds the mdBook and publishes it to the gh-pages branch, which is what serves https://colonel-byte.github.io/ansible-collection-zarf/. It runs on every push to main.

The job runs mdbook build over docs/ and pushes ./book with peaceiris/actions-gh-pages. That push is the only write the workflow makes, so contents: write sits on the job rather than at the top of the file; nothing here touches a pull request.

Two things follow from the book being published this way:

• The generated pages have to be committed. The build does not run generate:document, so whatever is in docs/ at the time of the push is what gets published – a generated page that was not regenerated is published stale.
• A renamed or moved page breaks whatever links to it from outside the repository, and mdBook does not notice: the old URL simply 404s. A move comes with an entry in [output.html.redirect] in book.toml. See docs/AGENTS.md.

E2E cluster

Runs test/e2e/cluster – the suite that walks the converging modules across two real k3d clusters, then runs the other five acting modules against the cluster the first walk initialised.

It is not a check on every push. It stands up two clusters, pulls three zarf packages, and runs a real zarf init against each, which makes it the slowest and least hermetic thing in this repository; the first run on a cold cache pays for the package pulls. Everything it covers that can be covered without a cluster already is, by the unit tests and by e2e.yaml – which includes every acting module’s refusals and check-mode answers, so what is left here is the half that needs a registry, a cluster state or something deployed to take away.

So it runs in two cases only: on demand from the Actions tab (workflow_dispatch), and on a pull request carrying the e2e-cluster label. The job’s if: condition enforces both, which is why adding the label to an open pull request is enough to start a run.

The job installs Go, Python, ansible-core, k3d and zarf, then runs test:endToEndCluster with a 90-minute timeout and tears down any surviving clusters with test:cleanClusters under if: always().

One step is easy to mistake for boilerplate: mkdir -p /tmp/k3d. Both k3d configs bind-mount that directory into the nodes as the local-path provisioner’s root, k3d does not create it, and a missing bind source fails the cluster create before any test runs.

The Go build is fully vendored, but the suite itself does reach the network to pull packages, so GOPROXY=off closes the module proxy only – not the network.

E2E

Runs test/e2e/noncluster – the five read-only modules, the state redaction, and every acting module’s refusals and check-mode answers, driven through a real ansible-playbook, mostly against a stub zarf. It needs no cluster, but it does need a real zarf: one case builds a real package to inspect. Runs on every pull request and on the merge queue.

The target builds the module wrapper, which the acting modules are. That is a large binary – zarf is linked into it – so this job pays a compile the read-only cases do not need; the alternative was leaving seven modules with no coverage outside the suite that needs k3d.

The triggers carry no path filter, deliberately. This is a required status check, and a filtered required check never reports at all on a change that only touches the ignored paths, which leaves the check pending forever.

Installing ansible-core and zarf are steps rather than optional conveniences. Every module in this collection reaches zarf through an action plugin, so only Ansible can run one; without ansible-playbook on PATH the whole suite skips itself with a reason and the job still passes, and without zarf the cases that build a real package skip the same way. That is exactly the failure mode worth preventing here – see the note in e2e-tests.md about a suite that silently stopped running looking just like a suite that passed.

The suite that needs k3d, a real zarf and Docker is not run here; see e2e-cluster.yaml.

github.ref is distinct per merge_group ref, so the concurrency group never lets a queued entry be cancelled by a push to the pull request it came from.

Validate pre-commit hooks

Runs the hooks from .pre-commit-config.yaml over the whole tree, so licensing, formatting and sorting can never drift from the local pre-commit setup. Runs on every pull request and on the merge queue.

It uses prek rather than pre-commit: same config, same hooks, faster. That matches the local instruction to prefer prek.

It passes --config .pre-commit-config.yaml, and that flag is load-bearing rather than explicit-for-its-own-sake. prek discovers nested .pre-commit-config.yaml files and runs each as a separate project, and two vendored dependencies ship one of their own. Without the flag this job runs their hooks over their code and fails on their toolchain – a shellcheck that is not installed, a golangci-lint built against a different Go – rather than on anything in this repository. Naming the config turns discovery off. The local command carries the same flag for the same reason.

Two hooks are skipped:

Every hook that walks files excludes vendor/. That is not tidiness: addlicense would stamp a header onto other people’s code and the whitespace fixers would rewrite it, in both cases making the tree differ from what go mod vendor produces – which check-go-mod.yaml then fails on. If a hook starts touching vendor/, the fix is an exclude. An exclude does not cover the nested-config case above, though, because it filters the files a hook walks rather than which configs prek finds.

Release Please

Maintains the release pull request: it accumulates conventional commits landed on main, writes the changelog, bumps the version, and tags the release when the pull request merges. Runs on every push to main, and on demand from the Actions tab.

Configuration lives in release-please-config.json and the current version in .release-please-manifest.json. The changelog-sections block decides which commit types appear in the changelog and under what heading; perf and test are hidden.

extra-files is the part that needs keeping current. Release-please bumps ansible/colonel_byte/zarf/galaxy.yml, and it will bump a Markdown page the same way – but only when the page is annotated with x-release-please-start-version markers and listed in extra-files. Both halves are required: an annotated page that is not listed is never read, and a listed page with no annotation is read and left alone. See docs/AGENTS.md for the annotation form and the hazard of a region holding a second dotted-decimal string.

The workflow declares no write permission anywhere. It mints a GitHub App token with actions/create-github-app-token and hands that to release-please, which does every write with it, so GITHUB_TOKEN is only along for the ride. The app’s credentials come from the RELEASE_PLEASE_GITHUB_APP_ID and RELEASE_PLEASE_GITHUB_APP_SECRET secrets.

goreleaser

Builds and publishes a release. Triggered by pushing a tag, which is what release-please.yaml does when the release pull request merges.

GoReleaser builds the module binaries and the archives per .goreleaser.yaml. Its before hook shells out to ansible-galaxy to build the collection tarball, which is why the job installs ansible-core with pipx – without it the release fails at its first step.

The job asks for four writes, each for one reason:

The dockers_v2 pipe builds containers/ansible/Dockerfile for linux/amd64 and linux/arm64 and pushes ghcr.io/colonel-byte/zarf-ansible-modules, so the job also installs QEMU and Buildx and logs into ghcr.io with the job token. The same image is built without being pushed on every pull request that touches it – see test-build-containers.yaml.

Like that workflow, this one sets GOTMPDIR before GoReleaser runs: the module binary carries zarf’s whole dependency tree and is built for three architectures at once, and the compiler scratch that generates overruns the 7.9GB tmpfs mounted at /tmp long before the root filesystem is anywhere near full. Without it the release fails with disk quota exceeded and a df showing plenty of room.

Beyond the archives, the workflow publishes provenance. actions/attest attests the checksums file, and a second attestation covers an AI-BOM: .github/ai-bom.json is rendered with the tag, commit and timestamp filled in by jq, then attested and uploaded alongside the release as dist/ai-bom.json. Both attestation bundles are uploaded as .intoto.jsonl files with gh release upload --clobber.

Signing uses cosign with the key and password in the COSIGN_PRIVATE_KEY and COSIGN_PASSWORD secrets, and Syft is installed for SBOM generation.

Validate Lint

Runs golangci-lint over the tree, and hadolint over containers/ansible/Dockerfile. Runs on every pull request and on the merge queue.

The interesting part is how the version is chosen. golangci-lint-action installs the latest release when it is given no version, so CI once ran 2.14.0 against a tree whose hooks pinned 2.13.2 – a lint failure that reproduced nowhere locally.

So the workflow reads the version out of .pre-commit-config.yaml instead of carrying a pin of its own. An awk program finds the golangci/golangci-lint repo block, takes its rev:, and exports it as a step output the action then consumes. Leaving that block without having seen a rev exits rather than reporting the next repo’s pin back, and an empty result fails the step with a message.

Reading rather than restating is the point: Dependabot’s pre-commit ecosystem bumps that rev, and its github-actions ecosystem updates uses: references, not inputs inside a with: block. A second pin here would be one nothing keeps in step with the first – the same “every pin needs an owner” rule that magefiles/AGENTS.md states.

The containers job runs hadolint. Its path is listed explicitly rather than globbed: the action’s recursive mode globs from the repository root, which would drag in the vendored Dockerfiles under vendor/. hadolint checks syntax and never builds the image – the build runs in test-build-containers.yaml.

Scorecards supply-chain security

Recomputes the repository’s OpenSSF Scorecard and uploads the result to the code scanning dashboard. Borrowed from zarf’s own workflow.

It runs on pushes to main, weekly on a schedule, and on a branch_protection_rule event. Only the default branch is supported, so there is no pull request trigger – which is the thing to know about it: a workflow change can move the score without anything failing in CI, and the published score only refreshes after a merge.

Score the working tree instead, before opening the pull request, and name the checks you ran in the description:

ver=$(gh release view --repo ossf/scorecard --json tagName -q .tagName)
curl -fsSL "https://github.com/ossf/scorecard/releases/download/${ver}/scorecard_${ver#v}_linux_amd64.tar.gz" | tar xz scorecard
./scorecard --local . --checks=Token-Permissions,Dangerous-Workflow,Pinned-Dependencies --show-details

--local reads the working tree, so it scores a change that is not yet committed. Prefer the release binary over gcr.io/openssf/scorecard, which is not reliably pullable. For an honest before and after, take the baseline from a clean export – git archive HEAD | tar -x -C "$(mktemp -d)" – rather than by stashing.

.github/AGENTS.md records the handful of checks this repository has actually tripped: granting a permission on the job rather than at the top of the file, never taking a checkout ref: from the event payload, pinning every action to a full commit SHA, and passing an untrusted value through env: rather than into run:.

The workflow itself declares permissions: read-all at the top and takes security-events: write on the job to upload the SARIF, plus id-token: write to receive the badge. Its checkout sets persist-credentials: false.

Test build containers

Builds the management node image on a pull request that touches it, and on the merge queue. Paths-filtered to containers/**, .goreleaser.yaml, the collection, cmd/**, internal/zarfmod/**, and the workflow itself.

This is a test rather than a packaging check. The smoke-check RUN block in containers/ansible/Dockerfile is the only place in the tree where the collection, the installed module files, a zarf and an ansible-core exist at once, so it is the only place the argv dispatch contract is checked end to end: every module file name must be a link to the same binary, a module file invoked under a zarf_ name it does not answer as must exit non-zero, one given parameters it cannot read must still write one JSON object and exit 0, check mode must report the task as skipped, a deploy against a bad kubeconfig must come back with failed and a zarf.exitCode, and zarf_kubectl must return the vendored kubectl’s output in zarf.output. Each of those fails the image build.

The Dockerfile cannot be built with a plain docker build: it COPYs the nfpm-produced *.rpm out of a ${TARGETPLATFORM} directory that only GoReleaser stages. So the job runs goreleaser release --snapshot --clean --skip=sign,sbom. --snapshot skips publish, validate and announce – nfpm still builds the rpm and the dockers_v2 pipe still runs buildx build – which is why the job needs no registry login and no packages: write. --skip=sign,sbom avoids failing on the cosign key the workflow has no reason to hold.

QEMU and Buildx are installed because the pipe builds linux/amd64 and linux/arm64 in one pass, and ansible-core because GoReleaser’s before hook shells out to ansible-galaxy for the collection tarball.

The job sets GOTMPDIR, and has to. The snapshot links zarf’s dependency tree into a binary of around 269MB and builds it for amd64, arm64 and riscv64 at once, which puts more than 8GB of scratch in the Go compiler’s temporary directory – and /tmp on the hosted runner image is a 7.9GB tmpfs rather than part of the 145GB root filesystem, so the build dies reporting disk quota exceeded against /tmp/go-build while df reports around 88GB free on /. That is the trap: the message names a quota, the obvious reading is the disk, and freeing disk space changes nothing. Pointing GOTMPDIR at $RUNNER_TEMP moves the scratch onto the work volume where the free space is – exported through $GITHUB_ENV, because the runner context a job-level env: block would need is not available there and a workflow that references it does not load. The step’s df covers every mount rather than / alone, so the next failure of this kind names the filesystem that ran out. release.yaml builds the same artifacts for real and does the same thing for the same reason.

scan-lint.yaml runs hadolint over the same Dockerfile. That checks syntax and never builds, so a base image bump that breaks the build is caught here and nowhere else.

Unit tests

Runs test:unit – everything but the two suites that have runners of their own. Runs on every pull request and on the merge queue.

This is where the collection and the Go code are held against each other. The four contract tests in internal/zarfmod all run here, and all of them need nothing installed:

The last pair parses what each module renders against zarf’s own command tree, which is linked in, so pflag decides whether a flag exists and a zarf release that renamed one fails the job. There is nothing to refresh by hand; the zarf they check against is the github.com/zarf-dev/zarf version in go.mod.

The suites under ./test/... need Ansible, k3d and Docker, so they are excluded here and run in e2e.yaml and e2e-cluster.yaml instead. See Test.Unit in magefiles/test.go for the package selection.

The step runs go run -mod=vendor ./magefiles/core test:unit rather than mage: same target, and it needs no mage binary on the runner. GOPROXY=off makes accidental network access fail fast rather than quietly succeed on a machine that happens to be online.

AI usage in this collection’s development

This document discloses how AI coding agents are used to build the colonel_byte.zarf collection. It is not a CycloneDX ML-BOM or SPDX AI profile - those describe AI/ML models and datasets shipped inside a product, and this one has none: it’s an Ansible collection wrapping a Kubernetes packaging tool, with no inference or model-serving code of its own. What follows instead is a disclosure of the tools used to write it, in the same spirit as the supply-chain artifacts described in the security policy - an SBOM discloses what’s in a release; this discloses how the release was produced.

Tools in use

Claude Code (Anthropic, Claude models) is used across the repository for code changes, documentation, and the design records under this directory. Its operating rules for this repository live in the various AGENTS.md files scattered through the tree - the repository root’s, and the per-directory ones it points to - which tell an agent how this codebase expects to be worked in, not just what the code does.

Human review

AI-authored and human-authored changes go through the same gate: a reviewed pull request, and the same CI checks - CodeQL, linting, the test suites, OpenSSF Scorecard - regardless of who or what wrote the diff. Nothing here grants an AI-authored change an exemption from review or from CI.

Where the provenance trail lives

Two records carry this forward:

• choice-*.md files in this directory record non-obvious design decisions and the tradeoffs behind them, agent-driven or not - see AGENTS.md for why they’re treated as constraints rather than history.
• Every tagged release publishes ai-bom.json as a release asset, attested the same way as the release’s SBOM and provenance. It’s a per-release snapshot of this document’s claims - which tools, what they’re used for, and the human-review statement above - rather than a list of what’s in that particular release.

Why Dependabot gets a commitlint config of its own

title_check in commitlint.yaml lints the pull request title rather than the commits in it, because release-please squash-merges and the title becomes the final commit message. It is also a required status check on main.

Dependabot’s titles break one rule in that config and cannot be made to stop. This is the record of what was done about it.

The problem

@commitlint/config-conventional sets header-max-length to 100. Dependabot writes build(deps): bump <module> from X to Y, and it cannot shorten a module path – there is no template knob that abbreviates it, and the path is the one piece of information the title exists to carry.

Ten of the modules in go.mod overflow 100 characters that way. The longest, go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc, reaches 117. With a daily gomod schedule, that is a steady stream of pull requests that cannot pass a required check.

Collapsing the gomod ecosystem into one group made most of them titles of the form build(deps): bump the gomod group across 1 directory with N updates, which fits – see choice-dependabot-vendor-grouping.md. It did not remove the problem: Dependabot opens a security advisory update outside the group, naming the one module, and that title is as long as it ever was.

Why not the two easier answers

Skipping the job for Dependabot leaves title_check with no result at all. A required status check that never reports is not a passing check – the pull request stays unmergeable, which is the problem it was meant to solve. The same reasoning is why the job runs on merge_group and reports success with every step skipped: the queue needs to see a result.

Turning header-max-length off everywhere gives up the rule on human titles too, and a human title is a title somebody can shorten. The rule is worth keeping where it can be met.

What is there instead

.github/commitlint/.commitlintrc.dependabot.yaml is .commitlintrc.yaml with header-max-length set to 0, and the lint step selects it by author:

config=.commitlintrc.yaml
if [ "${PR_AUTHOR}" = "dependabot[bot]" ]; then
  config=.commitlintrc.dependabot.yaml
fi

Every other rule still applies, so a malformed type or subject in a Dependabot title fails exactly the way it would in a human’s. The check reports, and the length rule is relaxed for the one author that cannot satisfy it.

PR_AUTHOR and PR_TITLE reach the script through env: rather than being interpolated into the run: block, which is the rule .github/AGENTS.md states for any github.event.* value.

The cost

Two files that must agree. There is no shared base, because commitlint cannot extend a YAML sibling, so a rule added or changed in .commitlintrc.yaml has to be copied into .commitlintrc.dependabot.yaml by hand. Both files carry a comment saying so, and nothing in CI checks it – the configs can drift, and the symptom would be a rule that is enforced on humans and quietly not on Dependabot.

If commitlint ever gains sibling extension, collapsing these two into a base plus one override is the fix.

Why every Go module update arrives in one Dependabot pull request

.github/dependabot.yaml puts the whole gomod ecosystem in a single group matching *. It used to carry seven groups by vendor family – k8s, opentelemetry, sigstore, golang, docker, cloud-kms, openapi – and anything outside them opened a pull request of its own. This is the record of why they were collapsed.

The problem

vendor/ is committed, and every build and CI job reads from it, because the management node this collection runs on is typically air-gapped. That makes the vendored tree the dependency list rather than a cache, and it is the reason check-go-mod.yaml re-vendors and fails on any diff.

Dependabot edits go.mod and go.sum and stops there. It has no notion of a vendored tree, and there is no setting that gives it one. So every gomod branch it opens is inconsistent on arrival, and the failures are not confined to the check that exists to catch it:

vendor/cloud.google.com/go/monitoring/apiv3/v2/auxiliary_go123.go:32:32: method AlertPolicyIterator.All already declared at vendor/.../auxiliary.go:31:32
vendor/github.com/containerd/typeurl/v2/types_gogo.go:24:2: cannot find module providing package github.com/gogo/protobuf/proto: import lookup disabled by -mod=vendor

Unit tests, both e2e suites, lint and CodeQL all fail the same way, because none of them can load a package that vendor/ no longer holds. The fix is one command – go mod vendor, committed to the branch – and it is the same one command whether the branch moved one module or two hundred.

With seven groups, a daily schedule and a 50 pull request limit, that command had to be run by hand on up to a dozen branches a day. The queue of them is what the grouping removes.

Why not the two alternatives

Automating the re-vendor means a workflow that pushes a commit to a Dependabot branch, which needs contents: write on a pull-request-triggered job – a token grant that .github/AGENTS.md keeps off anything a fork can trigger. A workflow_dispatch variant taking a pull request number avoids that, and remains the upgrade if one grouped branch a day still proves to be one too many.

Dropping vendor/ is not available. The air-gapped build is the point of it.

The cost

A grouped update is one pull request and therefore one revert. When a bump breaks something, git revert takes the other two hundred modules back with it, and finding which one was at fault means bisecting within the branch rather than reading the title. The previous layout bought that isolation, and it was paid for daily in re-vendoring; the grouped layout pays only when a bump actually breaks, which is rarer.

Security advisories are unaffected. Dependabot opens those outside the group regardless of this configuration, which is what the open-pull-requests-limit: 5 headroom is for.

Drift detection converges for a package deployed in one phase, and not for one deployed in several

colonel_byte.zarf.deploy_packages skips an item whose components the cluster already reports as deployed. For a package that one item deploys, that works and the second run of a play is a no-op. For a package that several items deploy in phases – which is what every uds-bundle init is – it never settles, and the role re-deploys on every run.

This is a limit of what the cluster knows, not a bug in the role, and it is written down because the code looks like it ought to work and the obvious fixes are worse than the limit.

What happens

The bundle’s init is two items: the seed phase, then the rest. zarf package list reports the components of the most recent deploy of a package, because NewDeployedPackage in vendor/github.com/zarf-dev/zarf/src/pkg/state/state.go assigns DeployedComponents: components from the deploy that is happening and the secret is written fresh. There is no union and no history; Generation is a single incrementing int.

So on the second run of the same stack:

The two init items hand the record back and forth. There is no run after which both are satisfied.

TestDeployPackagesRoleWalk is what found this, and it asserts the behaviour above rather than the behaviour we wanted, so that the day zarf starts retaining the union the test fails and says so.

Why the alternatives are worse

Verify against the cluster instead of zarf’s record. The appealing one, and it does not reach. Helm releases do accumulate – two deploys of the storage provider leave local-path-provider.v1 and .v2 – but they do not map onto components. In the init package zarf-seed-registry and zarf-registry declare the same release name, zarf-docker-registry, so Helm cannot tell the two phases apart; and zarf-injector declares no chart and no manifest at all, so there is nothing of it to find. Three of the five components are unanswerable this way. Closing the gap would mean a rule per component – “injector means look for these pods” – which is the cluster-specific knowledge this collection exists to avoid, and which would be wrong for the next package.

Union what the role deployed earlier in the same play. Makes the second role invocation in one playbook skip, and changes nothing for the case that matters: a scheduled converge is a fresh run with no memory of the last one. It would turn a test green while leaving the behaviour it claims to prove broken, which is worse than the limit.

A marker the role owns. A ConfigMap recording what the role deployed would genuinely make the phased case idempotent across runs, and the collection already ships a vendored kubectl that could read and write it. It was declined because it is a second source of truth: it goes stale the moment anybody runs zarf package remove, or deploys the same package by hand, and a drift check that trusts a stale marker skips work that was needed. The failure mode of the limit is a wasted deploy; the failure mode of a stale marker is a cluster nobody converged.

What this costs, concretely

• A phased package re-deploys on every run, which is changed: true forever and the deploy’s wall-clock time each converge.
• changed is therefore not a drift signal for such a package. Nothing else in the collection can supply one: the modules report changedSignal: partial on every successful step, because zarf tells them which components it deployed and not whether any of them found the cluster already as wanted – see choice-zarf-ansible-module.
• Nothing is at risk on the cluster. Re-deploying reconciles rather than fails, which test/e2e/cluster/testdata/bundle-order.yml proves on purpose in its fifth step: “a playbook that cannot be run twice is not a playbook”.

Before reversing this

The limit disappears if zarf merges DeployedComponents instead of replacing it, at which point the role’s existing logic is correct with no change. That is drafted as finding 1 of ZEP findings. Check whether it landed before assuming the role needs work.

Why there are two fuzz targets, and why they are not under fuzz/

This repository has two Go fuzz targets, both in internal/zarfmod, beside the code they exercise. This is the record of why those two and not others, and why a fuzz/ directory was considered and rejected.

Why fuzzing at all

The OpenSSF Scorecard’s Fuzzing check reports whether a repository fuzzes anything, and a proof-of-concept collection that scores zero on it invites the question. That is the reason the question was asked; it is not the reason these two targets exist. A target written to satisfy a scanner and nothing else is a test that passes forever and catches nothing, which is worse than no target at all, because it looks like coverage.

So the surface was read for places where fuzzing earns its keep: loosely-shaped input meeting hand-written parsing. Most of this tree does not qualify. command.go renders strings rather than reading them, and its inputs come from the module’s own parameters. version.go hands its input to semver.NewVersion and treats every failure as “unreadable, carry on” by design, so the parsing is upstream’s to fuzz and the failure path is already the lenient one. ModuleName in dispatch.go is a prefix trim against a fixed map. response.go and the stdout*.go files parse nothing. Two places survived.

The two targets

FuzzComponentName is the one worth having. componentName in progress.go is the only real parser in the tree: it reads zarf’s stdout one line at a time, in either slog-JSON or console form, because the operator can override zarf’s log format and the wrapper has no API to ask instead. The property it holds is not “does not crash” but “a line that announces a component names one”. progress.line uses a non-empty current as its “a component is in flight” flag and progress.finish returns early without one, so a ("", true) answer would set current back to empty and silently drop the run’s last heartbeat. Nothing would panic and no existing test would fail; the progress display would just stop reporting. The invariant holds today - the console regex’s capture group cannot match empty, and the message fallback is reached only for a message containing both “component” and “deploy” - and nothing in the tree said so until this target did.

FuzzReadArgs holds the property Args.Params exists for: a parameter the module does not accept is an error naming it, never a value silently dropped. The JSON decode underneath is the standard library’s and will not crash, but the accepted-name walk in jsonFieldNames is hand-written reflection over a struct with embedded fields, and a binary module has no argument_spec - as that function’s own comment says, it is the whole of what stands between a mistyped parameter and a run that ignores it.

Both seed their corpus with f.Add calls in the Go source rather than files under testdata/fuzz/, which keeps .gitattributes untouched and invents no fixture format. A crasher Go writes there during a campaign is a finding to fix, not a file to commit.

Why not fuzz/

A fuzz/ directory is the conventional home for fuzz harnesses in a C or C++ project, and for Go projects that drive go-fuzz or OSS-Fuzz builds. Neither applies here, for two reasons.

Scorecard does not look at paths. Its Go-native detection globs *_test.go across the whole repository and matches func\s+Fuzz\w+\s*\(\w+\s+\*testing.F\) against the contents. Where the file sits is not part of the check, so a directory move buys nothing on the score it was proposed for.

A Go fuzz target lives in the package it tests. Both targets reach unexported identifiers - componentName, controlPrefix, jsonFieldNames, initParams - and there is no exported path to the progress parser at all. A fuzz/ package could only see the exported API, so the move would mean either exporting internals to suit a test, or dropping the target that guards something real. Exporting a function because a test in the wrong directory cannot see it is the test dictating the API, and the API is the thing with the longer life.

What is deliberately not fuzzed

redact_state in plugins/action/zarf_state_info.py is the only place in this repository where a bug leaks a credential, and it is where a fuzzer would most like to go. It is Python, so no Go target reaches it, and fuzzing it means atheris or hypothesis: an unvendored dependency on a tree whose premise is that the build works with no proxy reachable. The function is already written to fail safe - it matches credential-bearing leaf names anywhere in the document rather than a list of paths, so a field zarf adds under a name like those is withheld before anybody reads that file - and the non-cluster suite covers it. More cases there are the cheaper answer than a fuzzing runtime; see choice-zarf-info-modules.md for why the redaction removes keys rather than blanking them.

The cost

Neither target has a CI job, and that is on purpose. Go runs a fuzz target’s seed corpus as an ordinary unit test, so test:unit exercises the seeds and the recorded crashers for free, and that is the part that protects against a regression. A campaign that explores new input is a by-hand run:

go test ./internal/zarfmod -run=FuzzComponentName -fuzz=FuzzComponentName -fuzztime=30s

What that means is that nobody is fuzzing continuously. The seeds run on every push and find nothing new by definition, so these targets protect the two invariants against a future edit rather than searching for a bug that is there now. Thirty seconds of each found nothing when they were written. If one of these parsers grows a real state machine, a scheduled campaign - or ClusterFuzzLite, which Scorecard also recognises and which would want a .clusterfuzzlite/ directory - becomes worth its CI minutes. Today it is not.

Why the GitOps engine is a key under requires, not a field beside a version

PackagePrerequisites first shipped with a requires.fluxcd section holding three things: a version, the controller Deployments that must be available, and the CRDs that must be established. When Argo CD had to be supported, the obvious move was a requires.argocd beside it with the same three fields. That is not what happened, and this records why.

What is actually engine-specific

Almost nothing. “These workloads must be available” and “these CRDs must be established” are the same question whatever reconciles the cluster, and they were only ever spelled requires.fluxcd.* because Flux was the only engine anybody had written a document for. A requires.argocd would have duplicated the struct, the schema, and all three of the role’s checks – and made the third engine worse, not better.

Exactly two things differ per engine:

• The namespace a controller defaults to. flux-system for Flux, argocd for Argo CD. A document whose controllers name their namespaces never needs the engine; one that leaves them off does.
• The kind of the workload. Flux’s controllers are all Deployments. Argo CD runs argocd-application-controller as a StatefulSet and everything else as Deployments.

So controllers and crds were hoisted to requires, ControllerRequirement gained kind, and what was left of the engine – its identity and its version – became a section of its own.

The identity is the key, not a field

That section was first written as requires.gitops holding engine and version side by side, and that was wrong in a way that took a second pass to see. The two engines do not number together: Flux is on 2.x while Argo CD is on 3.x. So a document could say engine: argocd next to version: ">=2.9.0" – an author who meant Flux, editing a copied document – and the two would disagree with nothing to notice it.

The disagreement is not symmetric, which is what makes it dangerous. Checked against a real cluster running Argo CD v3.5.5, >=2.9.0 is satisfied, because 3.5 is greater than 2.9. The constraint passes against an engine it was never about, and the run reports success. The reverse – meaning Argo CD and writing fluxcd – fails loudly against Flux 2.9.5. The quiet direction is the one a reader is most likely to produce, and silent success is the one failure this whole format exists to prevent.

The fix is to leave nothing for the version to contradict. The engine is now the key – requires.fluxcd or requires.argocd – each holding only a version, in that engine’s own numbering. A mismatch is no longer something to validate; it is something that cannot be written down.

The two are mutually exclusive, stated in both places it can be: Requires.JSONSchemaExtend emits not: {required: [fluxcd, argocd]} so an editor refuses it while the document is being written, and the role asserts it too, because a document reaches the role through zarf_prerequisites without ever meeting a validator.

What was lost is that engine as a single enum field was mutually exclusive for free. That is a real cost, paid for a property the enum could not give at all.

Why the identity is still needed

The version is read off the app.kubernetes.io/version label, and the label means different things per engine: for Flux it is the flux2 release the manifests were built from, which is not any controller’s own version. Without knowing the engine, the role cannot default a namespace and cannot say what the version it read is a version of.

A version required of something that is not an engine has its own home: requires.controllers[].version, on the workload itself. Without it the only version a document could put on a workload was the engine’s, so a package needing cert-manager v1.14 had to require Flux or Argo CD purely to get a version field – a prerequisite that refuses clusters for no reason.

An engine the format does not know is now unrepresentable rather than reported: the key is one of two the schema names. A document that requires neither is fine, and means a package with no opinion about what reconciles the cluster.

Why now, and why it was breaking

It was done while v1alpha1 had two consumers in this repository and no known consumers outside it. The alternative – a requires.argocd that never breaks anything – would have locked the engine-shaped mistake in permanently, for a format whose whole value is being readable by tools nobody here wrote.

The additive half was deliberately left for later and then done anyway in the same change: the StatefulSet query in zarf_cluster_info and the app-argocd reference package break nothing and could have waited for a real Argo CD user. They are here because the format change is unprovable without them – a discriminator with one engine behind it is a field, not a decision.

What this cost

zarf_cluster_info runs a fifth query. deployments and statefulsets are separate result keys rather than one workloads list with a kind on each entry, because merging them would have been a breaking change to a module result for no gain; the prerequisites role merges them itself, in one set_fact, and matches on (namespace, name, kind).

Where the prerequisites work departs from the shape already here

The colonel-byte.dev/v1alpha1 PackagePrerequisites work adds read-only modules, a role, a Go API, a schema generator, and a vendoring layer for one of the reference packages. Most of that follows the shape already here. The things that do not are recorded here as they arrive.

choice-zarf-info-modules is the record to read first: it is where the read-only modules’ Python implementation is argued for, and it names a threshold that this change crosses.

zarf_package_documentation renders three parameter-controlled flags, and is not a wrapper

The threshold in that record reads: a read-only module that grows a second or third flag, or any flag whose value attaches in a way pflag cares about, wants a wrapper binary rather than a better regular expression. zarf_package_documentation renders --keys, --key and --verify=, all three controlled by a parameter, and --verify is the attaching one. By the letter of it, this module wants a wrapper.

It did not get one, for three reasons.

The threshold was already crossed, and the record had not noticed. It says the three read-only modules “render one flag between them that any parameter controls – --key on zarf_package_inspect”. That stopped being true when verify was added to zarf_package_inspect: it renders --key and --verify= today. The threshold as written would have made that module a wrapper too, and nobody thought so at the time – which is a sign the number was the wrong thing to count.

What the number was standing in for is coverage, and the coverage here is better than a wrapper’s. The worry behind the threshold is spelled out plainly in that record: TestInfoPluginDocsMatchRenderedFlags reads flag spellings out of Python as string literals, so it sees which flags are named and not which parameter they are rendered for, in what order, or whether a value attaches with a space or an =. A wrapper’s test is better because it populates a struct and renders a real command line.

But --verify=always – the spelling that record calls the mistake the weak test would miss – is covered here by something stronger than either: TestZarfPackageDocumentationExtractsTheLayer runs the module against a real unsigned package built by the test and asserts that zarf refused it for having no signature. A module that rendered --verify always fails that assertion, because zarf rejects the command for a surplus positional argument instead and says so. The same test covers --keys by extracting one key from a package carrying two, and --output by extracting into a destination it then stats. Every flag is exercised against a real zarf. That is what a wrapper’s command-line test would have bought, and more.

equalsFlags in internal/zarfmod/infoplugins_test.go also catches the --verify spelling statically, in both modules, and did so before this change.

A wrapper is the wrong artifact for this command anyway. The three reasons that record gives for a wrapper – a flag surface worth projecting, progress worth following, a changed worth signalling – are still all absent. Extracting a documentation layer returns in under a second, has no components, and changes nothing. What a wrapper would add is a third main package, a third binary in every release, and a third thing an operator installs on the management node before a playbook runs, for a collection choice-zarf-collection-removal has scheduled for deletion once ZEP-0072 resolves.

The threshold is worth restating as what it was standing in for, rather than raised: a read-only module needs its rendered command line covered against a real zarf, or it needs to be a wrapper. Counting flags was a proxy for that, and a bad one.

ZarfInfoActionBase runs a list of commands, for one module

zarf_cluster_info asks the cluster four questions – the Kubernetes version, the CRDs, the StorageClasses and the Deployments – and no amount of flags makes them one kubectl command. The base class ran exactly one command and handed its stdout to interpret.

The alternatives were worse than extending it:

• Four modules. A prerequisite check would become four tasks that can each half-fail, and the role’s job would turn into assembling them.
• One command. kubectl get crd,storageclass,deployments -A -o json is one call and megabytes of answer: a CRD object carries the full OpenAPI schema of every version of its resource, and a cluster with a few operators on it answers with tens of megabytes, all of which would land in a registered variable and then in a fact. The version is a separate call regardless.
• Overriding run() in the subclass. That is a copy of the binary resolution, the environment handling and the failure reporting, which is the whole of what the base class is for.

So the base gained zarf_argvs and interpret_outputs, and they are the only path it runs. zarf_argvs defaults to the single command zarf_argv returns, and interpret_outputs defaults to handing that command’s stdout to interpret, so the four single-command modules are unchanged and did not have to declare that they are. Overriding the pair is the whole declaration: there is no flag saying which shape a module is, and a subclass that overrode one of the two and not the other would fail its task on the first run rather than quietly run one command and report as though it had run four.

Two consequences worth knowing:

• command_parts in zarf_cluster_info returns the first command, and command_parts_all returns all four. TestInfoPluginDocsMatchRenderedFlags was taught to read both and to read every match rather than the first, because otherwise three of the four commands’ flags would be invisible to the one test that looks at them – which is the state that file exists to prevent.
• A failed query names itself in the result’s query key. A message naming only the command line leaves the reader to work out which of four kubectl calls that was.

The role checks one comparator, and names what it cannot evaluate

requires.kubernetes.version is a SemVer range in the format. The role evaluates one comparator and one version, which covers >=1.30.0, and reports anything else – a compound range like >=1.30.0 <1.34.0 – in zarf_prerequisites_skipped.

Evaluating a real range needs a semver library. Masterminds/semver is already a dependency of this repository and is no use here: the check runs in Jinja, inside a role, on a management node. The options were a half-parse that silently checks the first comparator and calls it satisfied, or naming the constraint as unevaluated. The second is the only honest one, and it is why zarf_prerequisites_skipped exists as a returned fact rather than as a log line – a caller can assert on it, and the example playbook prints it on successful runs.

requires.storage.accessModes is in the same list for a different reason: what a provisioner will bind cannot be known without creating a PersistentVolumeClaim, which is not a thing a read-only check does.

A role that reported success for a document it mostly ignored would be worse than no role, so the rule is: anything not checked is named.

Where this diverges from the proposal

choice-zarf-ansible-module says it is the one place the proof deliberately diverges from ZEP-0072. This is the second, for the prerequisites proposal, and the divergences are small:

The first row is the one that matters. An annotation is a duplicate of a fact in the document, and a check that reads the duplicate is a check that passes when the two disagree. Keeping both is deliberate – they answer different questions, and the asymmetry is stated where they are written – but the document wins.

app-fluxcd is vendored by a mage target, not written by hand

The reference package was first written against the Flux Operator chart, and is now the upstream flux2 release manifests, vendored from a pinned tag. magefiles/pkg/pkgvendor is a Go port of the three shell scripts in colonel-byte/zarf-pkg-fluxcd’s .github/hack, which is where that pattern already runs.

Three simplifications against the original, each affordable because what lives here is a reference package rather than a published one: one flavor rather than upstream and minimus; one copy of the pinned tag, which the target writes into both zarf.yaml and the kustomizations rather than three copies that must agree; and generated kustomizations, which is what lets the version label be stamped from the pin.

The image lists are rewritten by line surgery rather than by a YAML round trip. zarf.yaml there is a hand-written teaching artifact full of comments, and a marshaller would reformat it and drop every one of them. The risk of that choice – a pattern that reaches past the component it was given – is what TestReplaceImagesRewritesOneComponent is for, and the blank-line case in it is a regression it already caught: a list pattern that also matched blank lines ate the separator between components, which reflowed the file on the first run and was a no-op on every run after, so the damage was invisible in the second diff.

The pin’s owner is a documented manual bump – edit tag, run examples:vendor and examples:images – which is the third answer magefiles/AGENTS.md allows. A target that read the upstream’s latest release and wrote it back was written and then dropped: it is a reference package nothing deploys, and the pin moving without being asked is the opposite of what pinning it was for. The renovate annotation in resources.json is carried over from the repository that has Renovate configured; nothing here reads it.

Ordering a directory of Zarf packages into a deployment DAG from prerequisites

This document records the architectural decisions, constraints, and trade-offs for automatically ordering a directory of Zarf package archives into a Directed Acyclic Graph (DAG) for deployment, using each package’s embedded prerequisites configuration.

The problem and goal

Today, playbooks deploying stacks of Zarf packages (e.g. through deploy_packages) require authors to specify an explicit deployment list or numeric step prefix. When operators receive a directory of arbitrary Zarf package tarballs, manually sorting them requires unpacking or inspecting each package to deduce dependency order (e.g. init packages, CRDs, service meshes, storage, application workloads).

Because packages in this ecosystem carry PackagePrerequisites (colonel-byte.dev/v1alpha1) inside their embedded documentation.prerequisites mapping, a bundle planner can extract these documents without deploying anything, build a dependency graph, verify that the graph is acyclic, and emit a topological deployment order.

Static inspection vs runtime deployment

The primary virtue of the PackagePrerequisites design is offline planning:

• zarf package inspect documentation extracts the embedded docs/prerequisites.yaml directly from the package archive without access to a Kubernetes cluster or network.
• zarf_package_documentation already implements non-destructive inspection and YAML decoding of documentation keys for single packages.
• A directory-level planner reads every archive in the target directory, extracts its prerequisites document (or handles packages without one), and constructs the dependency model entirely in memory before any cluster mutation occurs.

Inspecting many large archives introduces disk I/O and process overhead from running zarf package inspect documentation across tarballs. A realistic planner caches extracted documents by archive checksum/mtime to avoid redundant unpacking across runs.

Dependency resolution mechanics

Each inspected package carries:

• Identity: metadata.package and metadata.version in PackagePrerequisites (mapping to the concrete archive path in the directory).
• Provides: provides.capabilities, provides.crds, provides.namespaces.
• Requires: requires.packages (with name, names, and optional group id), requires.crds, requires.storage_classes, and engine/controller requirements.

In a directory bundle, edges in the DAG are constructed primarily via package dependencies:

1. Direct package edges (requires.packages[].name): If package $B$ requires package $A$ by name, directed edge $A \to B$ is added ($A$ must deploy before $B$). If a version constraint is declared, the bundle planner verifies whether $A$’s version in the directory satisfies $B$’s constraint.

2. Alias edges (requires.packages[].names): If package $B$ allows names: [init, init-slim], and the directory contains a package matching an alias (e.g. init), an edge is added from that package to $B$. If multiple satisfying packages are present in the directory, the planner flags the bundle as ambiguous unless disambiguated by explicit configuration.

3. Alternative group edges (requires.packages[].id): When a consumer package declares alternative requirements sharing an id in requires.packages (e.g. csi-rook-ceph vs csi-longhorn grouped under id: storage), the requirement is a disjunction on the consumer side. If exactly one provider from that group exists in the directory, an edge is created from that provider to the consumer. If multiple candidate providers in the group are present in the directory, the planner refuses the ambiguity unless the operator explicitly selects the target provider.

4. Transitive CRD and capability resolution: Packages can also declare requires.crds. If package $A$ lists CRD $C$ in provides.crds and package $B$ lists $C$ in requires.crds, an implicit dependency edge $A \to B$ exists even if $B$ does not explicitly name $A$ in requires.packages.

Edge cases and failure modes

• Cycles: If circular dependencies exist ($A \to B \to A$), topological sort fails. The planner reports the cycle trace clearly and aborts before deploying any package.

flowchart TD
    PkgA[Package A<br/>requires: B] --> PkgB[Package B<br/>requires: A]
    PkgB --> PkgA

• Phased component limits: DAG planning at the archive level cannot resolve intra-package cyclic bootstrap sequences where the same package archive is deployed in multiple component phases (such as init seed components $\to$ storage provider $\to$ init full components, as recorded in choice-deploy-packages-drift-scope.md). Such bootstrap sequences remain the domain of explicit multi-phase playbooks.
• Missing prerequisites: If package $B$ requires a package or CRD not provided by any package in the directory, the planner distinguishes two modes:
  1. Standalone bundle mode: Reject the plan because the bundle is self-contained and incomplete.
  2. Cluster-aware planning mode: Check if the missing prerequisite is already satisfied on the target cluster (via zarf_package_info / zarf_cluster_info). If already satisfied on the cluster, the requirement is satisfied and does not need a local provider in the DAG.

flowchart TD
    PkgC[Package C<br/>requires: cluster-storage] -.->|Satisfied on Cluster?| Clust[(Target Cluster<br/>zarf_package_info)]
    PkgC -.->|Missing in Bundle?| Fail[Abort or Query Cluster]

• Packages without prerequisites: A package in the directory with no prerequisites document or an empty requires has in-degree 0 (no prerequisites) and out-degree 0 (unless another package requires it). Independent nodes are sorted deterministically using lexical package name ordering as a tie-breaker.
• Multiple packages in the same id group: When a directory contains multiple provider archives that satisfy the same consumer id group (e.g. both csi-rook-ceph and csi-longhorn for id: storage):
  1. Default fail-fast rule: The planner refuses the plan and errors out immediately. Attempting to deploy both mutually exclusive providers, or arbitrarily picking one without operator intent, risks conflicting storage classes or incorrect infrastructure.
  2. Cluster precedence (existing installation preference): If one candidate from the group is already deployed on the cluster (e.g. csi-longhorn is already installed and healthy), cluster-aware mode prefers the existing installation. The requirement is satisfied by the cluster, avoiding an unnecessary redeployment or duplicate provider rollout.
  3. Priority / weight ranking: If offline bundles must choose between multiple candidate providers without operator intervention, package requirements or inventory configuration can declare an explicit weight or priority (e.g. priority: 100 on rook-ceph vs priority: 50 on longhorn). In the absence of an explicit priority or existing cluster deployment, equal-priority candidates remain a hard ambiguity failure.

flowchart TD
    Alt1["Package X<br/>csi-rook-ceph (priority: 100)"]
    Alt2["Package Y<br/>csi-longhorn (priority: 50)"]
    Consumer["Package Z<br/>requires id: storage"]
    Cluster[("Installed on Cluster:<br/>csi-longhorn")]
    Alt1 -.->|"Both in folder: Ambiguous failure<br/>unless prioritized or installed"| Consumer
    Alt2 -.->|"Prefer existing install or weight"| Consumer
    Cluster -.->|"Precedence to installed"| Consumer

• Multiple versions of the same package: When the directory holds multiple versions of a single package (e.g. operator-cert-manager v1.13.0 and v1.14.0):
  1. Default selection: The planner selects the highest version present.
  2. Constraint conflict risk: If a dependent package requires a specific or constrained lower version (e.g. <=1.13.5), selecting the highest available version (v1.14.0) causes the version check to fail.
  3. Planner resolution options: The planner can evaluate version constraints against available candidate archives before picking the latest version that satisfies all consumers’ constraints, or fail fast and require the operator to prune superseded archives from the directory.

flowchart TD
    VLow["Package M v1.13.0"]
    VHigh["Package M v1.14.0<br/>Highest candidate"]
    Dep["Package N<br/>requires: M <=1.13.5"]
    VHigh -.->|"Default to highest"| Dep
    Dep -.->|"Constraint Fails!"| Error["Version Conflict"]

• Multiple architectures in the bundle directory: Air-gapped delivery directories often stage archives for multiple CPU architectures side-by-side (e.g. zarf-package-app-amd64-v1.0.0.tar.zst and zarf-package-app-arm64-v1.0.0.tar.zst). Keying graph nodes strictly by package identity creates duplicate node collisions unless architecture is pruned first.
  1. Architecture disambiguation: The planner filters candidate archives by target architecture before building vertices. In cluster-aware mode, the architecture is queried from the cluster nodes (kubernetes.io/arch); in offline mode, it is provided explicitly (defaulting to the delegate host architecture).

flowchart TD
    ArchA["Package A (amd64)"]
    ArchB["Package A (arm64)"]
    Planner{"Target Arch Filter<br/>(amd64)"}
    NodeA["DAG Node: Package A<br/>path: app-amd64.tar.zst"]
    Ignored["Ignored Archive"]
    ArchA --> Planner
    ArchB --> Planner
    Planner -->|Matches| NodeA
    Planner -->|Mismatches| Ignored

• Package flavors and variant selection: Packages can be built under distinct flavors (e.g. flavor: upstream vs flavor: hardened or vendor-specific variants). Unlike architecture, flavor selection may be an intentional deployment requirement for a cluster stack.
  1. Bundle ambiguity: When multiple flavors of the same package and version exist in the folder, the planner requires an explicit flavor selector; otherwise, it fails fast on ambiguity.
  2. Upstream Zarf state limitation & RFC: Today, Zarf’s cluster state secret (zarf package list) does not record or report the deployed package flavor. A cluster-aware planner cannot determine which flavor is currently running on the cluster. Recording and exposing deployed flavor data requires an upstream Zarf RFC (ZEP) to include build flavor in deployed package state. Until upstream tracks flavor on-cluster, cluster-aware checks cannot assert flavor parity without external package inspection.

flowchart TD
    FlavorA["Package A (flavor: hardened)"]
    FlavorB["Package A (flavor: upstream)"]
    Selector{"Flavor Selector / Override"}
    Chosen["Selected for DAG"]
    StateCheck{"Cluster-Aware Check:<br/>zarf package list"}
    RFC["Upstream RFC Needed:<br/>State secret lacks flavor"]
    FlavorA --> Selector
    FlavorB --> Selector
    Selector --> Chosen
    Chosen --> StateCheck
    StateCheck -.->|Flavor not in state!| RFC

• Asynchronous CRD and StorageClass readiness races: While Zarf deploys package archives sequentially, Kubernetes registers CRDs and CSI storage drivers asynchronously. If package $A$ installs a CRD or CSI driver and package $B$ immediately creates Custom Resources or PersistentVolumeClaims, the Kubernetes API server may reject $B$’s manifests with webhook timeouts or missing resource errors.
  1. Topological ordering vs readiness barrier: The DAG planner establishes deployment order ($A \to B$), but execution must enforce a readiness barrier between dependent steps (e.g. awaiting CRD Established condition or CSI controller availability) before initiating $B$’s deployment.

flowchart TD
    StepA["Deploy Package A<br/>installs CRD / StorageClass"]
    Barrier{"Readiness Barrier<br/>wait CRD Established<br/>wait CSI Available"}
    StepB["Deploy Package B<br/>applies CR / PVC"]
    StepA --> Barrier
    Barrier -->|Ready| StepB
    Barrier -.->|Timeout / Not Ready| Abort["Abort Deploy"]

• Partial or unhealthy cluster state: In cluster-aware planning mode, a package may appear in zarf package list, but a previous deployment failed midway or omitted required components.
  1. Health and component completeness check: Rather than assuming package presence in zarf package list satisfies an edge, cluster-aware planning must verify that required components are deployed and controller workloads meet availability constraints (minAvailable >= 1). If unhealthy or incomplete, the edge must remain active in the local deployment DAG to re-reconcile the package.

flowchart TD
    ClusterList["zarf package list:<br/>Package A present"]
    HealthCheck{"Health & Component<br/>Completeness Check"}
    Satisfied["Edge Satisfied<br/>Skip local deploy"]
    Reconcile["Keep Edge Active<br/>Re-reconcile Package A"]
    ClusterList --> HealthCheck
    HealthCheck -->|Healthy & Complete| Satisfied
    HealthCheck -->|Degraded or Incomplete| Reconcile

• Directory hygiene and non-archive artifacts: Staging directories frequently hold non-package artifacts (.sha256 checksums, .sig Cosign signatures, README.md, or unrelated files). Furthermore, Zarf packages already embed internal checksums and metadata specifications.
  1. Archive filtering: The planner must strictly filter for valid archive extensions (*.tar.zst, *.tar) and ignore non-tarball companion files.
  2. Graceful warning on non-Zarf tarballs: When an archive in the directory cannot be parsed by Zarf (e.g. an arbitrary tarball missing zarf.yaml or failing internal checksum validation), the planner warns or ignores the file rather than aborting the entire bundle plan.

Output shape and execution contract

The DAG planner emits a linear, topologically ordered list of package items compatible with deploy_packages (zarf_packages_to_deploy), mapping each resolved node to its concrete archive path.

Because Zarf mutates cluster state sequentially through a single kubeconfig and Ansible tasks are serialized with run_once: true on the delegated host, execution against the cluster is serialized in topological order rather than run concurrently.

Visualizing the dependency graph

flowchart TD
    subgraph Phase1 [Phase 1: Foundation]
        Init[init / init-slim]
    end

    subgraph Phase2 [Phase 2: Platform & Engine]
        Flux[app-fluxcd<br/>provides: gitops-engine]
    end

    subgraph Phase3 [Phase 3: Shared Infrastructure]
        CertManager[operator-cert-manager<br/>provides: cert-manager CRDs]
        Storage[csi-rook-ceph<br/>satisfies id: storage]
    end

    subgraph Phase4 [Phase 4: Workloads & Addons]
        ExtSecrets[operator-external-secrets]
        App[workload-app]
    end

    Init --> Flux
    Flux --> CertManager
    Init --> Storage
    CertManager --> ExtSecrets
    Flux --> ExtSecrets
    Storage --> App
    ExtSecrets --> App

Why prerequisite packages accept name or names, and how id groups alternatives

PackagePrerequisites (colonel-byte.dev/v1alpha1) allows each entry under requires.packages to specify either a single package name or a list of acceptable names. This records why names exists beside name, why they are mutually exclusive, and how grouping by id extends this to alternative packages that carry distinct version constraints.

Why names exists beside name

A prerequisite package often exists under multiple published names that provide identical cluster capability. The canonical example is upstream Zarf’s init package versus rebuilds such as init-slim: a downstream package requiring an initialized cluster can run on either.

The expressive alternative – stating capabilities in provides and requiring a capability – cannot be evaluated on cluster because Kubernetes and zarf package list track package names, versions, and components, not PackagePrerequisites documents. The name is the only token zarf_package_info can read from the cluster without out-of-band package inspection.

Declaring separate requires.packages entries for each alias does not work:

• Multiple requirements would mean all of them must be present, when the author means any of them satisfies the requirement.
• Attaching version constraints across multiple alternative requirements causes duplicate constraints that drift when one is updated and another is forgotten. names keeps the version constraint bound to the requirement itself, applying whichever matching package is found on the cluster.

Why oneOf mutual exclusivity, not precedence or anyOf

A requirement that carries both name and names creates ambiguity:

• Under anyOf, a document with both keys is valid schema, forcing consumers to invent precedence rules. If the role reads names first, an author writing name: cluster-init and names: [init, init-slim] would have name silently ignored.
• Under oneOf, a document declaring both keys or neither key is rejected immediately at schema validation time.

Because playbooks can pass prerequisite documents directly to the prerequisites role without running schema validation first, the role repeats this check explicitly in roles/prerequisites/tasks/collect.yml:

• It asserts that no package entry defines both name and names.
• It asserts that no package entry defines neither name nor names.
• It asserts that names is non-empty and name is non-empty (minLength=1 and minItems=1).

How the role evaluates them today

In collect.yml, the role immediately normalizes every entry into an internal names list:

'names': item.names | default([item.name | default('', true)], true) | select() | list

This normalization means downstream task logic (check_resources.yml and check_versions.yml) only handles a single list structure:

• Resource checks verify that at least one name in names appears in deployed packages on the cluster.
• Version checks apply the constraint against whichever matching package name is deployed.

Grouping alternatives with distinct versions via id

The existing names: [a, b] mechanism binds a single version constraint across all names in that list. When alternatives have different version histories – such as two distinct storage providers satisfying a common requirement:

requires:
  packages:
    - name: csi-rook-ceph
      version: ">=1.21.0"
      id: storage
    - name: csi-longhorn
      version: ">=1.13.0"
      id: storage

A single version field under names: [csi-rook-ceph, csi-longhorn] cannot express >=1.21.0 for Rook Ceph and >=1.13.0 for Longhorn simultaneously.

Why id instead of requires.capabilities

provides.capabilities exists in the format for bundlers and orchestrators that inspect package documents ahead of time. However, on a running cluster, zarf package list only reports deployed package names and versions; it does not read or query capability declarations. id solves this by grouping concrete package names that can be checked directly against the cluster’s installed package inventory, while keeping separate version ranges per package.

Semantics of id grouping

• Local scope: id is scoped strictly within requires.packages. It groups alternative entries in that list and does not collide with other requirement types.
• Group satisfaction: An id group represents a disjunction (OR). A group is satisfied if at least one branch in the group has its package deployed on the cluster and satisfies its specific version constraint.
• Handling multiple installed packages: If multiple packages from the same id group are deployed on the cluster (e.g. both Rook Ceph and Longhorn are present), the requirement is satisfied if at least one satisfies its constraint. An obsolete or failing sibling does not fail the group if another branch meets its constraint.
• Ungrouped requirements: Entries without an id remain independent requirements that must each be present on the cluster (the existing behavior).
• Structure per entry: Each entry in an id group may use either name or names. For example, an entry in the group can still use names: [csi-rook-ceph, csi-ceph] if a specific branch has aliases sharing a single version constraint.
• Schema & Go types: ID string (minLength=1) is an optional field on PackageRequirement. The existing oneOf between name and names remains intact.
• Error reporting: If no branch in an id group is satisfied, the failure is reported as a single grouped unmet prerequisite naming all candidate packages and their constraints, rather than separate missing package failures.

Why the collection’s version tracks zarf’s

colonel_byte.zarf is versioned to loosely follow the version of zarf it is written against, rather than counting its own releases from zero. The collection released as 0.87.0 is the one to install beside zarf v0.87.x.

This is a deliberate break from how the collection was versioned in colonel-byte/cargoship, where it inherited the cargoship tag and reached 0.32.1 – a number that said something true about cargoship and nothing at all about zarf.

Why not start from 0.1.0

Two reasons, and the first is not negotiable.

0.32.1 is already published. Cargoship’s v0.32.1 release ships colonel_byte-zarf-0.32.1.tar.gz and the zarf_ansible_modules_Linux_* archives. Those are in the wild under this collection’s name. A first release here at 0.1.1 would be a lower version of the same collection: ansible-galaxy resolves semver, so the new line would read as older than what has already shipped, and a >=0.32.0 pin would never pick it up. Any starting point has to be above 0.32.1.

A number that means nothing is worse than no number. Having to pick something above 0.32.1 anyway, the choice was between an arbitrary 0.33.0 and a number that answers the question an operator actually has: which zarf does this work with? zarf_init links a specific zarf in and zarf_package_deploy renders a command line for the zarf it finds – see choice-zarf-ansible-module – so “which zarf” is the single most load-bearing fact about a given release. For zarf_init it is not even a statement of intent: the zarf in go.mod is the zarf that runs.

What “loosely” means

Loosely, because the two cannot be locked together:

• Not every zarf release needs one here. A zarf release that changes nothing this collection renders produces no release, and the version simply lags. A lag is honest: it says the last zarf this was checked against.
• This collection can release without zarf doing so. A fix to a role, an action plugin, or the redaction is a patch bump on the current line. That is why the patch component is this collection’s to spend, and why 0.87.1 means “the first fix against zarf 0.87”, not “zarf 0.87.1”.
• Nothing enforces it. Release-please computes the next version from conventional commits, so a feat: on the 0.87.x line produces 0.88.0 whether or not zarf 0.88 exists. Moving the line to a new zarf release is a manual edit to .release-please-manifest.json and galaxy.yml, made deliberately when the zarf dependency is bumped.

So the collection’s own version is a statement of intent rather than a guarantee, and the guarantee lives where it can be checked: the github.com/zarf-dev/zarf version in go.mod is the zarf both modules execute. That is not a claim about compatibility – it is the code that runs.

What has to move together

Bumping the line to a new zarf is more than the two version strings.

The dependency moves first, and it is the one that cannot be forgotten. zarf_init links zarf in, so the zarf it runs is whatever go.mod names. Nothing about the collection’s own version number changes it:

go get github.com/zarf-dev/zarf@vX.Y.Z
go mod vendor
go run ./magefiles/core test:unit

vendor/ is committed and is the dependency list rather than a cache, so the go mod vendor is part of the change and not a follow-up – see CLAUDE.md. TestInitArgsAreKnownFlags then parses what the module renders against the new zarf’s own init command, so a flag the new release renamed or moved fails that run. Read the failure rather than working around it: it is naming a module parameter that no longer reaches zarf.

The state fixture in test/e2e/noncluster/testdata/zarf-state.json is a recording of the shape zarf writes, and it goes stale the same way. It is the thing most likely to drift without failing anything, because the non-cluster suite checks the module against the recording rather than against zarf.

How the zarf Ansible modules reach zarf

ZEP-0072 proposes native Ansible module support for zarf: the zarf binary dispatches on argv[0], so a zarf_init symlink to /usr/bin/zarf is an Ansible module, with a thin Python action plugin alongside it. That is the design colonel_byte.cargoship already runs, and choice-ansible-module is the record of why.

This repository holds a proof of that proposal for the two commands that converge a cluster, zarf init and zarf package deploy, plus zarf tools kubectl as zarf_kubectl: the collection in ansible/colonel_byte/zarf/, the module logic in internal/zarfmod/, and one module file binary built from cmd/zarfmod/. The collection also ships read-only modules, which are built as plain action plugins rather than as wrapper binaries and are not part of this proof – see choice-zarf-info-modules. It is a proof of concept and is meant to be removed once the proposal is resolved – see choice-zarf-collection-removal.

Both modules link zarf in and execute its own command tree. How that came about, and what it bought, is the point of this note.

Both modules execute zarf’s command tree in this process

zarf_init and zarf_package_deploy import github.com/zarf-dev/zarf/src/cmd, build zarf’s own command tree with cmd.NewZarfCommand(), and execute init and package deploy in their own process – internal/zarfmod/libraryrunner.go. Each therefore enters the same command an operator’s command line enters: the real flag parsing, and for init the real validateInitFlags, the real agent TLS load, the real existing-state guard and the real init package search. None of that is reachable as API – every identifier in zarf’s src/cmd/initialize.go is unexported – which is why going through the command tree is the only honest way in, and why it proves something a reimplementation would not.

They did this one at a time, and init went first. For a while deploy still ran the installed zarf as a child, which was worth keeping as a side-by-side comparison of the two designs. That stopped being worth its cost once the comparison had been made: internal/zarfmod is one package, so zarf was linked into the deploy binary either way and the child process bought nothing but a second code path and a recorded flag list to maintain. The Runner interface survives with one implementation, because it is what lets the unit tests prove an argument vector without reaching zarf at all.

What linking zarf in bought

The flag surface stopped being a copy. This was the proof’s worst divergence. internal/zarfmod/init.go and deploy.go spell their flags out, and the two tests that check them used to hold those flags against recorded lists under internal/zarfmod/testdata/ – recordings that could catch a flag zarf never had, but not a flag zarf renamed. With the command tree linked in, TestInitArgsAreKnownFlags and TestDeployArgsAreKnownFlags resolve the subcommand with Find and parse the rendered argv with ParseFlags, so pflag itself answers. A renamed flag fails them. So does a flag that moved between the root and the subcommand, and so does --verify written in the space form – the mistake recorded below – which shows up as a second positional argument. The recordings and the generate:zarfFlags target that wrote them are gone.

The zarf the modules run gained an owner. It is the github.com/zarf-dev/zarf version in go.mod, which Dependabot’s gomod ecosystem already watches. magefiles/AGENTS.md asks every pin to have one and prefers Dependabot where it fits; a recorded flag list needing a human to run a mage target was the weaker half of that rule.

The version stopped being read back out of a subprocess. It is config.CLIVersion, compiled in.

What it did not buy, and what that tells upstream

Component progress is still read out of a log stream. ZEP-0072 has the deployer emit heartbeats, where the component list and the phase boundaries are already known. root.ExecuteContext returns an error, not a result, so internal/zarfmod/progress.go still reads component boundaries out of zarf’s log output – which is why the modules render --log-format json by default, and why the total component count is reported as 0 (an honest unknown) unless the operator named --components. What changed is only where the lines come from: logger.DestinationDefault is pointed at a writer the module owns instead of a child’s pipe being scanned.

changedSignal can never be complete. Zarf reports which components it deployed, not whether a component found the cluster already in the state it wanted. initOptions.run discards its DeployResult and state.DeployedComponent.Status records Succeeded or Failed rather than already-in-desired-state, so a successful run reports changed: true, changedSignal: partial, and every component named in changedUndeclared.

Both of these are the argument for the proposal rather than defects to engineer around here. A module layer living inside zarf sits beside initOptions.run, where the component list and the DeployResult are both in scope – which no external caller can reach at any price, however it gets in.

What it cost

vendor/ went from 66M to 386M, and the module file binary is about 280M. The proof is air-gapped-first and vendored, so that tree is the dependency list rather than a cache – see CLAUDE.md. Accepted deliberately: a proof that argues for code living inside zarf has to link zarf, and inside zarf the cost is zero because the binary is already there.

Two things the linking made dead were removed rather than left to rot, and both are breaking changes to the converging modules’ parameter surface.

The zarf_binary parameter and the ZARF_ANSIBLE_BINARY environment variable selected which zarf to execute. Nothing executes a zarf by name any more, so a task that sets either was choosing something that could not be honoured – worse than no parameter at all. The command a result reports now begins with the literal zarf, which is still what an operator would type to reproduce the run. The five read-only modules are plain action plugins that genuinely do shell out, so they keep their own zarf_binary; this was never a collection-wide parameter.

The version floor went with it. MinZarfVersion was 0.72.0, the release where zarf init gained the positional package source, and internal/zarfmod/version.go read zarf version before every init so that an older installed zarf failed naming both versions instead of rejecting an argument it had never heard of. With zarf compiled in, that check asked the binary its own version and compared it against a constant it could not fall below: go.mod is the floor, and check-go-mod.yaml enforces it. zarf.version is still reported – it is config.CLIVersion now, read straight off the linked-in zarf.

Calling packager.Deploy with a packager.DeployOptions instead was considered and rejected. That call is the last of eight steps in initOptions.run, and the other seven are unexported; taking that route means reimplementing init’s orchestration outside zarf, which drifts on every zarf release and argues the opposite of what this proof exists to argue.

One binary under one name per module

Every module file is a symlink to the single binary built from cmd/zarfmod, which answers as whichever zarf_ name it was invoked under. That is the proposal’s own design – in it, /usr/bin/zarf_init is a symlink to /usr/bin/zarf – and here the symlinks point at a wrapper of this repository’s own instead, because a proof built here cannot add the dispatch hook to zarf’s main.

It was a main package per module until linking zarf in made that expensive. Each binary carried its own copy of zarf’s dependency tree – 269MB apiece, 538MB for two – to reach dispatch code that was already shared, since internal/zarfmod was one package the whole time. Collapsing them cost the symlinks and saved a copy per module, which is also what a third module would have cost: zarf_kubectl added one dispatch entry and no binary.

zarfmod.ModuleFiles() is the list of names, derived from the dispatch table, and both the build target and .goreleaser.yaml’s packaging read it rather than keeping lists of their own. The binary’s own name is not one of them, so running it by hand prints what it is for instead of answering as a module.

The root command’s shorthands are pruned, and os.Args is left alone

Cobra merges a parent’s persistent flags into the command being run, so zarf’s root flags land in the flag set of whatever zarf tools kubectl resolves to – and -l is zarf’s --log-level and kubectl’s --selector. pflag panics on that: unable to redefine "l" shorthand. Duplicate long names are fine, because AddFlagSet skips a name the child already has; it is only the letters that collide. LibraryRunner therefore clears the shorthand on every root persistent flag before executing, which costs nothing here because the modules render long flags only. Every letter rather than -l alone: the next zarf release that gives a root flag a letter kubectl already uses would panic in a playbook, and nothing in this repository would have seen it coming.

Zarf’s own CLI avoids the collision differently – it installs no root flags at all for a vendored command, which it decides by reading os.Args – and presenting the rendered command line there was tried first. It works, and it is worse. kubectl’s NewDefaultKubectlCommandWithArgs installs its plugin handler when os.Args names tools kubectl, and that handler calls os.Exit(1) for a subcommand it cannot find and syscall.Execs a plugin it can. Either one leaves the module process without the JSON object Ansible is waiting for, which is a worse failure than the one being avoided, and the second one leaves it running something else entirely. Pruning the shorthands reaches the same place without presenting anything.

Two consequences of not having kubectl’s plugin handler are handled in the module instead. kubectl plugins are not reachable at all, which for an air-gapped management node is the answer that was wanted anyway. And the handler is what diagnoses an unknown subcommand on the real CLI, so without it args: [frobnicate] reached kubectl’s root Run, printed help, and reported success – cobra’s own unknown-command check fires only at a root command, and kubectl’s root has a parent here. validateKubectlCommand does that check: resolve as far as the tree matches, let the resolved command parse the rest, and refuse a leftover positional argument on a command that has subcommands.

zarf_kubectl is a passthrough, and deliberately thin

zarf tools kubectl is vendored kubectl, and the module renders nothing into the argument vector a task gives it: args is kubectl’s own argv, and the only other parameter is kubeconfig, which zarf takes from the environment rather than a flag. Projecting kubectl’s flag surface into Ansible parameters would be reimplementing kubectl in YAML, and kubernetes.core.k8s already exists for declarative resource management.

What it adds is that there is no kubectl to install: the one zarf vendors is already inside the module file, and a management node driving an air-gapped cluster would otherwise carry a second binary through the airlock to read one resource. The prerequisites role already reaches the cluster through zarf tools kubectl for the same reason.

Two things about it are honest rather than convenient. It reports changed: true with changedSignal: unknown, because nothing in the module reads the argument vector: a successful run says the command worked and nothing about whether the cluster moved, and reporting changed: false would tell a playbook to skip the handlers after an apply that did change something. A read is what changed_when: false is for. And check mode skips, because nothing here can tell a get from an apply.

Its failures are reported rather than exited on. kubectl reports a fatal error through cmdutil.CheckErr, which prints and calls os.Exit – correct for a CLI, and the end of a module, since Ansible would report MODULE FAILURE in place of what went wrong. That is the ordinary failure path, not an edge: a cluster that cannot be reached goes through it. cmdutil.BehaviorOnFatal is kubectl’s own hook for a caller that is not its main, and its documentation names panic as the thing to do instead, so runCatchingKubectlExit installs a handler that panics and recovers it into an error with kubectl’s message and exit code.

Its output comes back as zarf.output. kubectl writes to os.Stdout, which it takes when the command tree is built – inside the runner, by which point the stdout guard has pointed os.Stdout at stderr – so runKubectl points it at a temporary file for the length of the run. A file rather than a pipe: nothing drains a pipe while the command runs, and get -o yaml over a large cluster would fill one and hang.

What the five modules beyond the converging pair decided

Each of them had one thing worth recording, and all five are the same binary under another name.

zarf_package_pull is the only module here whose changed is an observation. Everything else in the collection reports changed: true with a signal saying how little that is worth, because zarf reports what it did and not whether the cluster already agreed. A pull writes a file, so the module lists the package files in the output directory before and after and reports the arrivals in zarf.packagesPulled, with changedSignal: complete. That list is also the only way a playbook learns the filename, which zarf derives from the package’s own metadata rather than from anything the task said.

zarf_package_remove reports unknown rather than partial. The converging pair report partial because zarf announces each component as it deploys it, and internal/zarfmod/progress.go reads those announcements. A removal announces nothing until it is over – l.Info("package successfully removed", ...) is the whole of it – so there is no component list behind a signal, and nothing distinguishes a package that was there from one that was already gone. The cluster is what the e2e suite asks instead.

zarf_update_creds renders the subcommand and refuses a mismatched credential. Zarf has update-creds registry, git and agent, each accepting only its own service’s credentials, and the bare zarf tools update-creds that accepts all of them warns that it is deprecated. So service is required, and a credential belonging to another service is refused by name rather than rendered as a flag that subcommand has never heard of. force_conflicts is refused for git for the same reason: zarf puts it on the other two only. The artifact server is the fourth service zarf keeps credentials for, has no subcommand, and is deprecated upstream – so this module does not update it.

zarf_package_mirror_resources reports no components, which was found by running it. It was built with the progress reader wired to it, on the assumption that a mirror walks components the way a deploy does and announces them the same way. The cluster suite showed otherwise: zarf logs mirroring images, pushing image and done pushing images – the resources, never the component they came from. So the module reports changed: true with changedSignal: unknown, carries no heartbeat and takes no status_file, and the assertion in the suite is that the component list is empty.

zarf_package_mirror_resources refuses images and repos together. Zarf marks the two mutually exclusive with MarkFlagsMutuallyExclusive, which is checked when the command runs – after the module has already reported the command it was about to run. Refusing the pair up front makes the message name the parameters rather than the flags. TestMirrorExclusivityIsZarfsOwn holds that refusal against zarf’s own declaration, so a release that stopped marking them exclusive shows up as the module inventing a rule.

zarf_kubectl is covered in its own section above.

What the proof does keep

Everything else in the proposal is implemented as described: argv[0] dispatch with a ZARF_ANSIBLE_MODULE override, the WANT_JSON arguments file, the stdout guard that duplicates fd 1 and points os.Stdout at stderr, exit 0 with failure reported inside the JSON, check mode answered as skipped because zarf init has no dry run, and the heartbeat status file with a polling monitor thread in the action plugin.

The stdout guard earns its keep harder on the linked-in path, and in a way worth recording. cmd.OutputWriter is a *os.File that zarf’s cmd package initialises to os.Stdout when it loads – which is before the guard runs – so it holds the true file descriptor 1 that Ansible parses the module’s JSON from. Reassigning os.Stdout does not move it. LibraryRunner points it at stderr explicitly; left alone, anything zarf printed through it would land in the middle of the result object.

Two of the proposal’s open questions are answered here rather than deferred, because a proof has to pick something:

• Argument delivery. Both options are implemented. The action plugin pipes the parameters on stdin when the connection can carry them, which is the proposal’s leaning and keeps registry and git credentials off disk; it falls back to a 0600 file in a 0700 directory when the connection cannot, which is the proposal’s Option A. The module reads both, so the fallback is a different wire and not a different contract.
• Process cancellation. The module cancels its context on the first signal and exits on the second – which is zarf’s own signal wiring, and is why both main functions here and zarf’s own read alike. The cancelled context is the one zarf executes with. That is the proposal’s leaning – a partial initialisation a later run reconciles beats one still mutating a cluster nobody is watching – without the in-flight rollback the proposal leaves open.

Why there are two modules

zarf init alone cannot be proved against a cluster whose distribution ships no storage provider, and that is not a quirk of the test environment – it is the case the init-local-path uds-bundle exists to handle. Zarf’s registry claims a 20Gi PVC during the seed phase. With no StorageClass the claim stays Pending until the Helm install times out, and disabling the claim does not help: the registry chart then renders no storage configuration at all and the container exits with no storage configuration provided. Both were measured rather than reasoned about, on a k3d cluster from that repository’s own config.

What does work is deploying the storage provider package first with --connected, which pushes no images and so needs no registry, and so needs no initialised cluster. That makes zarf package deploy a prerequisite of zarf init rather than a follow-on, which is why the proof carries both actions and why test/e2e/cluster walks them in both orders – the bundle’s own order on a cluster that has k3s’s StorageClass, and provider-first on one that has none.

Both walks deploy the bundle’s own two packages, pinned in test/e2e/cluster/main_test.go: init-slim:v0.88.0 and csi-local-path-provider:0.0.37-upstream. The init package matters more than it looks. It is the bundle’s slim init, whose five components are all optional, and that is what makes the bundle a four-step alternation rather than two deploys: the first init takes zarf-injector and zarf-seed-registry, and the second takes zarf-registry, zarf-agent and git-server once there is a StorageClass for the registry’s volume. The standard zarf-dev init package has no optional components, so a walk using it deploys everything on the first step and the split means nothing.

Two traps came with that, both recorded in the code. The bundle’s two refs as written – init-slim:v0.88.0-upstream and csi-local-path-provider:v0.0.37-upstream – do not exist; the init publishes no -upstream suffix, and the provider dropped its leading v at 0.0.37. And zarf package pull names a ZarfInitConfig after its kind and version, so the slim init and the standard one both land as zarf-init-<arch>-v0.88.0.tar.zst, which is why the suite caches the slim one in a subdirectory of its own.

The suite adds a fifth step the bundle does not have: the second init, repeated with the same parameters. The bundle’s four steps are all disjoint, so none of them asks whether converging an already-converged cluster succeeds – which for an Ansible module is the question. It needs the git credentials pinned on every init step, and which step matters in a way that took two cluster runs to see. Zarf’s init writes a gitServer block into cluster state whether or not the git-server component was asked for, and validateExistingStateMatchesInput compares field by field with IsNotZeroAndNotEqual – which ignores a field the input leaves empty. So an init that passes no git information is always allowed, and the first init that passes some after one that did not is always refused. Pinning them only on the later steps fails; pinning them from the first init means state carries them and every later init agrees. That is zarf’s guard working as designed, not a fault in the module, but it is a sharp edge for any playbook that converges a cluster more than once.

Two findings from that suite are worth keeping even if everything else here goes away:

• --verify must be rendered as --verify=always. It is a flag with a no-argument default, so pflag reads it the way it reads a boolean: the space form leaves always as a positional argument and zarf rejects the command for having one too many. command.valueFlag exists for exactly this, and --registry-mode does not need it. This cost a cluster run to find. TestInitArgsAreKnownFlags now catches it in milliseconds, because it checks what the real command tree does with the positional arguments – which is the clearest single illustration of what linking zarf in is worth.
• The slim init package requires an adrp.xyz/is-infra=true node. Its registry carries a required nodeAffinity on that label, which is a convention of the infrastructure the uds-bundle deploys onto and is set by neither the bundle nor the uds-bundles k3d config. On an unlabelled k3d cluster the registry pod is unschedulable – 0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector – and the seed phase waits until the Helm install times out, which reads exactly like the StorageClass failure above and is not it. Both k3d-config*.yaml fixtures add the label, and that is the only place they diverge from the upstream config they are otherwise copies of.
• Registry proxy mode is what makes a local cluster usable. Zarf’s default nodeport mode pushes every image through a Kubernetes NodePort, and on k3d that push times out and retries for minutes. registry_mode: proxy puts a DaemonSet on a host port instead and the same initialisation finishes in about a minute.

What the proof does not wire up

The reference pages are generated the same way the cargoship collection’s are: ansibleCollections in magefiles/pkg/gen/docs/ansible.go lists both collections, and this one’s pages land under docs/ansible/zarf/ because the hand-written overviews that introduce them cannot share a filename with the cargoship collection’s. The drift between a Go parameter struct and its Python DOCUMENTATION block is held by internal/zarfmod/plugindocs_test.go, which is the zarf side of internal/ansiblemod/plugindocs_test.go.

The end-to-end suite runs against two k3d clusters. One is k3d-config.yaml in colonel-byte/uds-bundles, copied verbatim, so that what the modules initialise is the cluster shape that repository already deploys onto; the other is the same file with k3s local-storage left enabled, which is what lets a walk begin with init. Both fixtures carry headers saying which walk they serve and why.

The zarf collection is a proof of concept, and is meant to be removed

ansible/colonel_byte/zarf, internal/zarfmod, cmd/, and test/e2e/cluster exist to prove ZEP-0072, which proposes native Ansible module support for zarf. They are not part of what cargoship does. Cargoship builds and installs offline Kubernetes distro packages and runs a cluster lifecycle over SSH; nothing in that needs a zarf module, and no cargoship command calls into any of this.

They live here because the pattern the proposal describes is the one colonel_byte.cargoship already runs – see choice-ansible-module – so the cheapest place to find out whether the pattern holds for zarf was beside the implementation it copies. That was the whole reason, and it expires.

This note exists so that whoever removes it does not have to work out whether it was load-bearing first.

When to remove it

Any one of these is enough:

• ZEP-0072 is accepted and implemented in zarf. Then the real thing exists, dispatch lives inside the zarf binary, and keeping wrappers that shell out to zarf means maintaining a worse copy of a shipped feature.
• ZEP-0072 is rejected, or lands somewhere else. The maintainers’ own feedback on the pull request raised an isolated experimental repository and a plugin architecture as alternatives. If either is where this goes, the proof belongs in that repository rather than this one.
• It stops being maintained here. The modules pin a zarf version in go.mod that someone has to bump and re-vendor (see choice-zarf-ansible-module), and the e2e suite pulls packages from two registries. The first time a zarf release breaks it and nobody is willing to refresh it, delete it rather than skip it.

Nothing about cargoship’s own roadmap forces the removal, and nothing breaks while it stays. It is dead weight rather than a hazard: a release artifact administrators carry through an airlock and never install, and a test suite with prerequisites most contributors do not have.

What to remove

And these edits, each of which is the only thing in its file that mentions zarf-the-tool:

• .goreleaser.yaml: the zarf_init and zarf_package_deploy builds, the zarf_modules archive, the second before hook, the collection tree and two module symlinks under nfpms[0].contents, the two bin/ansible/colonel_byte-zarf-* globs under release.extra_files, and the two wrapper ids in nfpms[0].ids. Leave the ids: [cargoship] on the first archive: it is right either way, and without it the cargoship archive silently absorbs whatever builds exist.
• containers/ansible/Dockerfile: the zarf build stage, the COPY --from=zarf line, the zarf version check, the /home/nonroot/.zarf-cache directory, the SHELL line, and the whole smoke-check RUN block after it. The cargoship version check stays, and so does /home/nonroot/.zarf – cargoship’s own cache paths predate this. Dropping the stage takes about 200MB off the image.
• magefiles/pkg/build/zarfmodules.go, and in magefiles/pkg/testrunner/testrunner.go the RunE2EZarf, K3dClusters and DeleteK3dClusters functions, and in magefiles/test.go the EndToEndZarf and CleanZarfClusters targets.
• hack/ansible-build-collection.sh: the third positional argument and the namespace/name lookup can stay – they are a generalisation that costs nothing and the cargoship path does not care – or revert to the hardcoded collection. Do not leave the script accepting a collection directory that no longer exists while the before hook still names it.
• docs/dev/mage.md, docs/dev/e2e-tests.md, docs/workflows/test-build-containers.md, AGENTS.md’s layout table, and .gitignore’s __pycache__ entry. The .gitignore entry should stay: it is right for the cargoship collection on its own.

Then go run ./magefiles/core generate:document, which rewrites docs/SUMMARY.md, and pre-commit run --all-files.

What not to take with it

• internal/ansiblemod and ansible/colonel_byte/cargoship. The zarf side is modeled on them and shares no code with them. Nothing in internal/zarfmod is imported by anything outside itself and cmd/, and nothing in the cargoship collection refers to the zarf one.
• internal/heartbeat. It predates this work, is used by the cargoship phases, and the zarf side deliberately writes its own rather than reusing it.
• The zarf-dev/zarf dependency in go.mod. Cargoship depends on zarf as a library for package formats, architecture detection and OCI handling, and has since long before this. Removing the collection does not make it unused; go mod tidy will tell the truth about that on its own.

Why this is a record rather than a TODO

The removal is a decision somebody should make deliberately, with the proposal’s outcome in hand, not a chore to be swept up by whoever next tidies the repository. A todo- note would invite the second reading. What makes it safe to remove on short notice is the list above, and what makes it safe to leave is that none of it is reachable from cargoship’s own code paths.

Why the read-only zarf modules are Python rather than wrapper binaries

Every other module in colonel_byte.zarf is a compiled Go binary. plugins/plugin_utils/projection.py opens by saying its Python is “deliberately the whole of” the collection’s Python, and choice-zarf-ansible-module records why: the parameters are a Go struct in internal/zarfmod, the command line is rendered by a Go function, and plugindocs_test.go holds the Python DOCUMENTATION block against both.

zarf_cluster_info, zarf_package_documentation, zarf_package_info, zarf_package_inspect, and zarf_state_info are not built that way. Their whole implementation is the action plugin, sharing ZarfInfoActionBase in plugins/plugin_utils/info.py. This is the record of that decision, and of what it costs.

It was written when there were three of them and one flag between them. Two were added for the PackagePrerequisites format, the flag counting below no longer holds, and choice-package-documentation-module is the record of why the conclusion survived the arithmetic anyway. Read that one with this one.

Why the wrapper buys nothing here

The wrapper exists for three reasons, and a read-only command has none of them.

There is no flag surface worth projecting. zarf init renders more than thirty flags, which is why spelling them out in Go and parsing them against zarf’s own command tree earns its keep. The most any of these renders is three – --keys, --key and --verify= on zarf_package_documentation – and zarf_cluster_info renders none that a parameter controls at all. The rest of each command line is fixed words.

When this was written the claim was “one flag between them, --key on zarf_package_inspect”, and it was already out of date: verify had been added to that module. See the correction in choice-package-documentation-module.

There is no progress to follow. internal/zarfmod/progress.go reads component boundaries back out of zarf’s log stream so that a fifteen-minute init is not a silent play. zarf package list returns in under a second and has no components, and so do the other four. The heartbeat file, the status-file parameter, and the polling monitor thread in ZarfActionBase would all be carried for nothing.

There is no changed to signal. The hardest question in the wrapper – what changedSignal can honestly be when zarf reports which components ran and not whether any of them had work to do – does not arise. These modules always report changed: false, and that is not an approximation.

Against that, building them as wrappers would add a main package under cmd/ and a file in internal/zarfmod each, a binary each in every release, and one more thing each that an operator has to install on the management node before a playbook runs. choice-zarf-collection-removal has the whole collection scheduled for deletion once ZEP-0072 resolves, which is a poor thing to spend a release artifact on.

What is checked instead

The gap this leaves is not that the modules are Python. It is that plugindocs_test.go is a table of modules, so a module absent from the table is a module no test looks at, and nothing says so. That is the state these three were first written in. Two tests in internal/zarfmod/infoplugins_test.go close it:

• TestEveryActionPluginIsCovered globs plugins/action/*.py and requires each plugin to appear in exactly one of two lists: wrapperModuleCases, the table the wrapper modules are checked through, or infoModules, the read-only ones. A plugin in neither fails the test by name. It also fails on a list naming a plugin that no longer exists, since that is a check that passes by checking nothing.
• TestInfoPluginDocsMatchRenderedFlags holds each read-only module’s documented cli_flag values against the flags its command_parts function names, in both directions.

command_parts is a module-level function of the task’s parameters, rather than a few lines inside run(), precisely so that the second test has something to read. zarf_cluster_info asks the cluster four questions and so has a command_parts_all beside it, which that test reads too – otherwise three of its four commands would be invisible to the only test that looks at them. The same technique holds the variable allowlist in the cargoship collection’s projection plugin: see internal/ansibleinv/plugin_test.go.

What that test cannot see

It reads string literals out of Python source, so it is weaker than its wrapper-side counterpart, which populates a struct and renders a real command line. It sees which flags are named. It does not see which parameter a flag is rendered for, in what order, or whether a value is attached with a space or an =. --verify=always – the spelling choice-zarf-ansible-module records as load-bearing, because the space form leaves always as a positional argument and zarf rejects the command – is exactly the mistake it would miss. It also depends on every flag being a literal inside command_parts, which is why none is assembled from a variable.

That is an acceptable trade at one parameter-controlled flag per module. It stops being one well before the flag surface gets interesting, and the threshold is worth naming: a read-only module that grows a second or third flag, or any flag whose value attaches in a way pflag cares about, wants a wrapper binary rather than a better regular expression.

That threshold has since been restated. Counting flags turned out to be a proxy for what actually matters, which is whether the rendered command line is covered against a real zarf – and zarf_package_documentation, which renders three, is covered better by its end-to-end test than a wrapper’s command-line test would cover it. The rule now reads: a read-only module needs its rendered command line exercised against a real zarf, or it needs to be a wrapper. choice-package-documentation-module has the argument.

zarf_state_info withholds the credentials it reads

The zarf-state Secret is not state with some credentials in it. As of zarf v0.85.0 it holds eight: registryInfo.pushPassword, registryInfo.pullPassword, registryInfo.secret, gitServer.pushPassword, gitServer.pullPassword, artifactServer.pushPassword, agentInfo.tls.key, and the deprecated agentTLS.key that duplicates it. The last one is the agent webhook’s TLS private key, which is a different severity from a registry password: a password is scoped to a registry, and that key is the admission webhook’s identity.

So the module redacts by default. include_credentials is false unless set, the secret leaves are removed from state, and the dotted paths removed come back in redacted. An operator who needs a password sees redacted: ["registryInfo.pushPassword", ...] and knows which parameter to set, which is a better discovery path than documentation.

Three decisions inside that are worth the words:

The keys are removed, not blanked. A missing key fails a template with 'dict object' has no attribute 'pushPassword', which names the problem. A sentinel like VALUE_REDACTED templates cleanly into a registry login and fails somewhere else entirely, hours later, as an authentication error against a registry whose password is right in the Secret.

Matching is by leaf name, not by a list of the eight paths. A path list is more precise and goes stale the next time zarf adds a field, and what staleness costs here is a leaked credential rather than a failed task. Leaf-name matching picks up a new field under a name like these without anybody revisiting the file. The cost is over-redaction: a future benign field named *key would be withheld. That is the right direction to be wrong in. agentInfo.tls.ca and .cert are deliberately not matched – they are public certificates, and templating the CA into a trust bundle is a reason to call this module.

A run that asks for the credentials censors its own output. executor/task_result.py honours an _ansible_no_log key in a module’s returned data, not just the task’s own no_log, so the module sets it when include_credentials is true. The registered variable still holds the real state; only the display is suppressed. Asking for the data is not the same as asking for it to be printed, and relying on the playbook author to have remembered no_log is relying on the thing that fails.

set_fact prints what the module hid

This is the part that is easy to get wrong, and it is why zarf_no_log defaults to true on both of the state role’s tasks rather than just the module call. A set_fact that reads a censored result prints the value in its own result:

TASK [set_fact from a censored result] *****
ok: [localhost] => {
    "ansible_facts": {
        "zarf_state": "hunter2"
    },

That is at -v, with the producing task fully censored. no_log on the module alone looks correct in review and leaks on the next line.

One thing that is already safe, and worth not breaking: set_fact defaults cacheable: false (plugins/action/set_fact.py), so the fact stays in the play’s memory and never reaches a configured fact cache plugin. A cacheable: true here would write the cluster’s registry credentials into a jsonfile cache on the control node, outliving the play that read them. Do not add it.

How it is tested

A redaction is a claim about the contents of a returned dictionary, so unlike the flag rendering above it cannot be read out of the source – it has to run. TestZarfStateInfoWithholdsCredentials in test/e2e/noncluster drives a playbook that asserts the redacted paths by name, that no credential value survives anywhere in the document, that every non-credential field is intact, and that include_credentials returns the whole thing.

It needs no cluster. The module’s one impure step is running zarf, so testdata/zarf-state-stub.sh stands in for it and prints the base64 of testdata/zarf-state.json, a copy of the shape zarf v0.85.0 writes with every credential replaced by a FIXTURE- placeholder. That covers the decode, the redaction, the role, and the fact.

Two things about that test are deliberate rather than incidental:

• It counts the assertions that reported passing. A renamed fact or a skipped block leaves every remaining assertion passing, and failed=0 would call that a success. The expected count is read out of the playbook rather than written in the Go file, so adding an assertion does not quietly stop the count from meaning anything.
• It reads the playbook, the fixture and the plugin itself before running anything. Go’s test cache keys on the files the test process opens, and everything this test exercises is opened by the ansible-playbook child. Without those reads, editing the plugin leaves a passing result cached – which is precisely the shape of mistake that gets a credential leak marked as fixed. This was confirmed by sabotaging the redaction: before the reads were added the cached pass survived, and after them the test fails as it should.

One thing that did not generalise

--no-color is rendered by zarf_package_info and zarf_package_inspect, which parse zarf’s stdout as JSON and YAML respectively and cannot have colour codes in it. It is not rendered by zarf_state_info, and not for want of tidiness: zarf tools kubectl hands its arguments to kubectl, which answers unknown flag: --no-color and exits. It is in each command_parts rather than in the shared base class for that reason, with the reason written at the point where its absence looks like an oversight.

TODO: skip an item that names no components

colonel_byte.zarf.deploy_packages decides whether to deploy an item by comparing the components the item asks for against the components zarf package list reports for that package. An item that names no components is deployed every run, because there is nothing to compare.

This is written down rather than fixed because the fix is not the small one it looks like, and because the way it is currently wrong is the safe way: an item that deploys when it did not need to wastes time, and an item that skips when it was needed leaves a cluster the operator believes is converged and is not. Anything done here has to keep that asymmetry.

What this page said before, and why it was wrong

The first version of this page said an item naming no components “is always deployed”. It was not: it killed the play, part-way through the stack, with the earlier items already deployed.

Both the init and the deploy task rendered components as

{{ pkg_item.components | join(',') if (pkg_item.components is iterable and pkg_item.components is not string)
   else (pkg_item.components | default(omit)) }}

and the shape test ran before the | default(omit) that was supposed to cover an absent key. Ansible’s undefined marker raises UndefinedError from is iterable rather than returning false, because Jinja’s iterable test only catches TypeError, so the error escaped the conditional entirely:

Error while resolving value for 'msg': object of type 'dict' has no attribute 'components'

The fix is pkg_item.components is defined leading the test, and Jinja’s and short-circuits so nothing after it is reached. What is worth keeping from the episode is why it survived review: every item in ansible/examples/playbooks/zarf-deploy-packages.yml and in test/e2e/cluster/testdata/deploy-packages-role.yml names components, so no test ever rendered the branch, and the page documenting the limitation described a behaviour nobody had run.

The lesson for whoever picks this up: the omitted-components path needs a test before it needs a feature. A check-mode run of the role against the noncluster stub exercises every expression in it in seconds, with no cluster and no network – that is where this belonged.

Where the behaviour is

ansible/colonel_byte/zarf/roles/deploy_packages/tasks/deploy_package.yml, the Determine if deployment should be skipped task:

_components_already_present: >-
  {{ (_required_components | length > 0) and
     (_required_components | difference(_deployed_components) | length == 0) and
     (_excluded_components | intersect(_deployed_components) | length == 0) }}

_required_components is the item’s components with the --prefixed exclusions removed. The | length > 0 guard is the whole of this issue: with no components named, the expression is false and the item deploys.

The _deployed_version clause beside it is a separate rule and not part of this issue: it refuses to skip when the cluster’s version of the package differs from the one the item carries, so that an upgraded tarball whose component names did not change is not mistaken for what is already deployed.

What a fix has to work out

The question is what zarf would deploy for a package with no --components, and that is selectOptionalComponent and the loop above it in vendor/github.com/zarf-dev/zarf/src/pkg/packager/filters/deploy.go. Non-interactively – which is what the modules are, they pass --confirm – the selected set is:

• every component that is not optional, and
• every optional component carrying default: true, and
• for each component group of more than one, the member carrying default: true; a group with no default is an error rather than an empty selection.

Three things make that hard to rebuild from the inspected definition:

A Jinja expression that got any of those wrong would report a package as fully deployed when it was not, which is the failure direction this must not have.

If it is picked up

Do not write it as one more expression in _components_already_present. Two routes, in order of preference:

1. Ask zarf. The honest answer is the one zarf itself computes. There is no zarf package deploy --dry-run that prints the selection today; if one appears, or if the filter is exposed through a library call the wrapper binary could make, that is the fix – the wrapper already links zarf in, so a module parameter that returned the selection would put the computation in the one place that cannot drift from it.
2. Resolve it in the inspect step, in Go. zarf_package_inspect is a Python action plugin today, so this would mean a new parameter on a wrapper module rather than on it. The same objection applies to anything that reimplements the filter – it has to be the filter, not a copy of it.

Either way it needs a test per schema version, over a package with an optional-and-default component and over one with a group. test/e2e/noncluster/ is where that belongs: the stub serves inspect, so no cluster is needed.

Until then: naming the components in the item is what turns drift detection on, and that is what docs/ansible/roles.md and the role’s argument_specs.yml say.

TODO: the per-item keys of zarf_packages_to_deploy are prose, not a contract

zarf_packages_to_deploy is declared in ansible/colonel_byte/zarf/roles/deploy_packages/meta/argument_specs.yml as type: list, elements: dict, and nothing else. The sixteen keys an item may carry are a list of prose bullets under one description. Ansible validates none of them.

So path is not enforced as required, no key is type-coerced, and package_type, verify, registry_mode and the three boolean pass-throughs get no choices check – even though their role-level counterparts, declared a few lines further down in the same file, have all of that.

What it cost already

Three separate defects traced back to this, all found in review and all now patched defensively in the tasks rather than at the boundary:

Each patch is correct and each is a guard that an options: sub-spec would make unnecessary, with a better error: Ansible names the key and the expected type, where an assert names only what the role thought to check for.

Why it was not just done

The obvious fix is to convert the bullets into an options: block under zarf_packages_to_deploy. Ansible would then validate every item key, and the role’s own guards could go.

The documentation generator would silently drop it. docOption in magefiles/pkg/gen/docs/ansible.go has fields for type, elements, required, default, choices, description and the cli_flag extension – and nothing for options or suboptions. orderedOptions decodes each option body through that struct, so a nested block is parsed and discarded. The generated docs/ansible/role_deploy_packages.md would lose the item keys entirely, which is worse than prose.

Writing both – an options: block for validation and the bullets for the page – would leave the item contract stated twice in one file, which is the duplication ansible/colonel_byte/zarf/AGENTS.md names a single source of truth to avoid.

What a real fix is

Two parts, in this order:

1. Teach the generator suboptions. Add an Options goyaml.MapSlice field to docOption, recurse in orderedOptions, and render a nested option as an indented sub-table or a second table under the parent’s row. magefiles/pkg/gen/docs/ansible_role.go’s writeRoleDoc builds the rows; that is where the shape is decided. The module pages would get the same ability for free, which no module needs today.
2. Convert the bullets, delete them from the description, and then remove the three defensive guards above – the assert on package_type, the | default('', true) | string wrappers, and the is not none clauses – because argument validation covers all three and reports better.

Step 1 is the real work and it is the one with no caller asking for it, which is why this is written down rather than done. Step 2 without step 1 regresses the published page.

Related

• The role’s remaining per-item gap of a different kind is in todo-deploy-packages-component-selection: what an item that names no components should be compared against.
• choice-deploy-packages-drift-scope records why skipping settles for a single-phase package and not a phased one.

TODO: the roles say “once” fifteen times and nothing tests it with two hosts

Every role in this collection delegates its work to one node and marks the tasks run_once: true. Nothing in the repository runs any of them against a play with more than one host, so the whole design is unexercised: every playbook under test/e2e/noncluster/testdata/ and test/e2e/cluster/testdata/, and every example under ansible/examples/playbooks/, is hosts: localhost with connection: local.

This is not a hypothetical shape. The collection’s own scope statement in ansible/colonel_byte/zarf/AGENTS.md says zarf talks to one cluster from the node it runs on, and a management-node play that targets a fleet of clusters – one host per cluster – is the obvious way an operator would use it. That play has never been run.

What is uncertain

colonel_byte.zarf.deploy_packages is the sharpest case. tasks/main.yml loops include_tasks: deploy_package.yml with no run_once, while almost every task inside the included file has one. So the loop is entered once per host and the inner tasks are what stop zarf running once per host.

Under the linear strategy the hosts move in lockstep and a run_once task’s facts propagate to the rest of the play, so this is expected to work. Under strategy: free the hosts desynchronise, and then _components_already_present – a fact, set run_once, read by a task that is itself run_once – is read by whichever host arrived first against whichever item’s value was written last. The same question applies to the three include_tasks in the prerequisites role’s main.yml and the one in its check_versions.yml.

The milder version of the same doubt is cosmetic: if run_once inside a per-host include fires once per expansion rather than once per play, zarf_prerequisites_checked gains a copy of every message per host and the final assert’s message is N times as long. Wrong-looking rather than wrong, since what matters is whether the unmet list is empty.

Why it is written down rather than settled

It needs a two-host play to settle, and a conclusion drawn from one is only as good as the strategy it ran under – linear passing says nothing about free. That is a fixture plus a decision about which strategies the collection claims to support, and nothing claims any today.

There is a second reason to be careful: run_once: true on the include_tasks itself would be smaller than fifteen inner ones and is probably more robust, but it is a behaviour change to how facts propagate, and making it without a multi-host test is trading an untested design for a different untested design.

What a real fix is

1. A fixture with two hosts. The noncluster suite can do it with an inline inventory of two aliases both resolving to localhost, run once under linear and once under strategy: free. That is cheap – the stub serves both – and it is what turns every statement above into a fact.
2. Then decide. Either move run_once up to the includes, or state the supported strategies in meta/argument_specs.yml and leave the tasks alone. Both are small once the fixture exists; neither is safe before it.

Until then, treat the roles as single-host. meta/main.yml claims min_ansible_version: "2.15.0", which widens the uncertainty rather than narrowing it: ansible-core 2.19 rewrote the templating engine, and every templating behaviour this repository has verified was verified on 2.20.