Cargoship
Cargoship is a Go-based CLI for building, distributing, and applying offline Kubernetes distro packages. It is designed to simplify two core workflows:
- Package Creation: Building a self-contained, offline Kubernetes distro package containing everything needed for installation in disconnected or highly-regulated environments.
- Cluster Lifecycle Management: Bootstrapping, upgrading, and managing the cluster on target hosts over SSH using the packaged distribution.
Cargoship bridges the gap between offline distro packaging tools and remote cluster lifecycle managers, supporting OCI image and file packaging, secure publishing to OCI registries, and robust SSH orchestration.
Installation
Pre-built binaries
Every release publishes archives for Linux, macOS, and Windows across amd64, arm64, and riscv64. The name carries the OS and the architecture, with amd64 spelled the way uname -m does, so the one for a 64-bit Linux host is cargoship_Linux_x86_64.tar.gz and the arm64 one is cargoship_Linux_arm64.tar.gz:
VERSION=$(curl -fsSL https://api.github.com/repos/colonel-byte/cargoship/releases/latest | grep -o '"tag_name": *"[^"]*"' | cut -d'"' -f4)
curl -fsSLO "https://github.com/colonel-byte/cargoship/releases/download/${VERSION}/cargoship_Linux_x86_64.tar.gz"
tar -xzf cargoship_Linux_x86_64.tar.gz cargoship
install -m 0755 cargoship /usr/local/bin/cargoship
Linux packages
Releases also carry .rpm, .deb, and .apk packages. Each installs the binary, shell completions for bash, fish and zsh, and the colonel_byte.cargoship Ansible collection onto ansible-core’s default collection path:
sudo dnf install ./cargoship_*_linux_amd64.rpm # or: sudo apt install ./cargoship_*_linux_amd64.deb
Container images
podman run --rm ghcr.io/colonel-byte/cargoship:latest version
| Image | Contents |
|---|---|
ghcr.io/colonel-byte/cargoship | The binary on a minimal base |
ghcr.io/colonel-byte/cargoship-ubi | The binary installed from the .rpm, on AlmaLinux 10 |
ghcr.io/colonel-byte/cargoship-deb | The binary installed from the .deb, on Debian slim |
ghcr.io/colonel-byte/cargoship-ansible | The .rpm plus ansible-core, on AlmaLinux 10 |
The -ubi name is historical: that image was based on Red Hat UBI and is now built on AlmaLinux 10, but the published image name has not changed.
From source
Building needs Go 1.27 or newer.
go install github.com/colonel-byte/cargoship@latest
That reaches the network for the module. To build without it, clone the repository instead: the vendor/ tree is committed, so a clone builds with no network access at all, which is how a management node inside an airlock is staged.
git clone https://github.com/colonel-byte/cargoship.git && cd cargoship
mage build:binary # or: go run ./magefiles/core build:binary
Verifying a download
Release archives and container images are signed with Cosign, using the key published at cosign.pub. Each archive ships a .sigstore.json bundle beside it, and every release carries a checksums.txt:
cosign verify-blob --key cosign.pub --bundle cargoship_Linux_x86_64.tar.gz.sigstore.json cargoship_Linux_x86_64.tar.gz
cosign verify --key cosign.pub ghcr.io/colonel-byte/cargoship:latest
Supported Distributions
Cargoship currently provides native support and integration for the following Kubernetes engines:
- K3s
- RKE2
- Upstream Kubernetes (kubeadm) - installs the
pkgs.k8s.iopackages and containerd, and applies a CNI the package declares
Core Concepts
Cargoship is built around two primary architectural concepts:
1. Offline Distro Packages
A Cargoship package is a single, compressed archive containing all artifacts necessary to spin up or upgrade a Kubernetes cluster in air-gapped networks. It bundles:
- Distro-specific metadata and layout descriptors.
- Required engine configuration templates and system files.
- OCI images and container artifacts.
- Checksums and cryptographic signatures for integrity verification.
The package compilation process loads a local distro definition, gathers all specified assets, and produces a single, verifiable archive.
2. Phase-Based Orchestration
Cluster operations are modeled as ordered sequences of reusable, structured “phases.” This makes high-level actions (apply, prepare, reset, kube-config, engine-config-sync) predictable, easier to debug, and simple to extend.
For example, the apply workflow runs these phases, in this order:
- Connect: Establish secure SSH connections to all target hosts.
- OS Detection & Fact Gathering: Identify host operating systems, network facts, and the engine version already installed.
- Validation: Verify that host nodes meet the prerequisites, and stop if the cluster holds a node the config does not.
- Lock: Take an exclusive host lock, held for the rest of the run, so a second cargoship cannot change the same hosts.
- Prepare: Install OS packages, configure sysctl and environment, SELinux and
fapolicydrules,/etc/hosts, and the node firewall. - Upload: Securely transfer package binaries, RPMs or DEBs, configuration templates, and OCI images to the nodes, then import the images.
- Bootstrap / Upgrade: Render the engine config, initialize control-plane nodes, apply bundled manifests, join workers, and upgrade any node already running an older version.
- Engine Config Sync: Reconcile registry, audit, and pod-security config on any node whose config drifted.
- Kubeconfig Retrieval and Labelling: Fetch the generated admin credentials and label nodes with their profile group.
- Cleanup & Disconnect: Release the lock, clean up temporary artifacts, and close SSH sessions.
The generated reference in docs/phases/ is the authoritative list, including what each phase does under --dry-run.
Usage Examples
Here are the standard workflows for compiling and deploying offline Kubernetes packages with Cargoship.
The commands that change a host - prepare, apply, reset, and engine-config-sync - will not do so without --confirm. Swap it for --dry-run to connect to every host and run the preflight checks for real, reporting what the run would change without changing it.
1. Compile an Offline Package
To build an offline archive containing all required files, binaries, and container images:
# Create a package from the current directory structure
cargoship create .
# Build from a specific path and output to a custom directory
cargoship create ./distro-defs -o ./build/
2. Prepare Target Nodes
Verify and configure OS-level prerequisites (such as kernel modules, firewall ports, fapolicyd rules, and /etc/hosts) across target machines using your cluster configuration inventory:
cargoship prepare ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm
3. Deploy or Upgrade a Cluster
Bootstrap a new cluster or upgrade an existing one from the compiled package:
cargoship apply ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm
4. Fetch the Kubeconfig
Retrieve the admin kubeconfig securely from the primary controller:
cargoship kube-config --config ./cargoship-config.yaml
5. Uninstall or Reset Target Nodes
Stop, uninstall, and completely purge the Kubernetes distro and its state from the target hosts:
cargoship reset --config ./cargoship-config.yaml --distro rke2 --confirm
6. Generate an Inventory from Ansible
Translate an Ansible inventory you already maintain into a cargoship cluster inventory, taking each host’s role from its Ansible groups:
cargoship inventory from-ansible ./resolved.json -o ./inventory.yaml
7. Publish, Pull, Sign, and Verify a Package
A package can live in an OCI registry rather than on a disk, and every command that takes a package takes an oci:// reference wherever it takes a path:
# Push a built package to a registry
cargoship publish ./build/cargoship-distro-amd64.tar.zst oci://ghcr.io/my-org
# Pull one back down to a local directory
cargoship pull oci://ghcr.io/my-org/my-package:1.0.0 -o ./build/
# Sign a package with a private key
cargoship sign ./build/cargoship-distro-amd64.tar.zst --signing-key ./private-key.pem
# Verify the signature before anything installs from it
cargoship verify ./build/cargoship-distro-amd64.tar.zst --key ./cosign.pub
An operator who cannot reach the registry from the management node moves the package across the airlock with Skopeo instead, then converts the resulting OCI archive into a native package:
# On the connected side: export the package as an OCI archive
skopeo copy --override-os multi docker://ghcr.io/my-org/my-package:1.0.0 oci-archive:my-package.tar:1.0.0
# On the disconnected side: convert it into a package apply can install
cargoship import ./my-package.tar -o ./build/
import reads an OCI Image Layout, as a .tar or as an unpacked directory; see Importing Archives for the other tools that write one, and why a docker-archive cannot carry a package.
A cosign keypair is the only signing method - keyless (Fulcio/OIDC) signing and verification were removed, because they need network the management node does not have; see #848. The commands that load a package - apply, prepare, engine-config-sync, publish, pull, validate, schema - verify as part of loading, gated on --verify=never|if-possible|always. verify does nothing but check the signature, which is what makes it the one to run before a playbook.
8. Encrypt Credentials in a Config File
Registry credentials in a cluster inventory are encrypted at rest, with Ansible Vault or with age. The vault command group covers both:
# Encrypt one value for a registry's user/pass/token field
cargoship vault encrypt my-registry-password --vault-password-file ./vault-pass.txt
# Encrypt or decrypt every registry credential in a config file, in place
cargoship vault encrypt-file ./cargoship-config.yaml --vault-password-file ./vault-pass.txt
cargoship vault decrypt-file ./cargoship-config.yaml --vault-password-file ./vault-pass.txt
# Encrypt the values already sitting at given YAML paths, naming each one
cargoship vault encrypt-path ./cargoship-config.yaml '.spec.config.registries[0].auth.pass' --vault-password-file ./vault-pass.txt
# Generate an age key pair, and re-wrap an existing file under a new key
cargoship vault keygen
cargoship vault rekey ./cargoship-config.yaml --new-vault-password-file ./new-pass.txt
encrypt-path only encrypts; cargoship decrypts a registry’s auth.user, auth.pass, auth.token and tls.ca at apply time, and warns when asked to encrypt a path outside that set. decrypt, decrypt-path, decrypt-file and encrypt-file complete the set; cargoship vault --help lists all eight.
9. Validate a File, or Write Its Schema
# Check an inventory, package definition, or config file against its schema
cargoship validate ./cargoship-config.yaml
# Write the JSON Schema for one of cargoship's own file formats: inventory, package, or config
cargoship schema inventory -o ./inventory.schema.json
# Checksum a file, or one file inside an archive
cargoship sha256sum ./build/cargoship-distro-amd64.tar.zst
10. Record a Run for Audit
# Write a structured record of the run: what ran, against which package, against which fleet,
# and what changed. Written whether the run succeeds or fails, readable only by its owner.
cargoship apply ./build/cargoship-distro-amd64.tar.zst \
--config ./cluster.yaml --confirm \
--receipt-path /var/log/cargoship/receipts/apply-20260928-143000.json
# Strip everything that identifies the fleet, ready to attach to a bug report
cargoship redact-receipt ./receipt.json -o ./share.json
The receipt carries the invocation and the cargoship that ran it, the package’s digests and signature status, the cluster inventory, and the phase-by-phase ledger with per-host change reasons and diffs. redact-receipt renumbers the hosts host-01, host-02, drops the diff bodies – an engine configuration diff carries join tokens – and removes the cluster name, operator and command line, leaving what a maintainer triages from.
A fleet that keeps an audit trail sets the path once in the config file instead, under .distro.receipt_path, where apply, prepare, reset and engine-config-sync all pick it up:
distro:
# {command} and {timestamp} expand per run, so each receipt lands in its own file
receipt_path: /var/log/cargoship/receipts/{command}-{timestamp}.json
Those two placeholders are the only ones accepted; anything else is rejected before the run starts, rather than written out literally. --receipt-path overrides the configured value and takes the same placeholders.
OpenTofu and Terraform
The same phases run from a declarative configuration through the OpenTofu provider, which lives in colonel-byte/terraform-provider-cargoship and consumes this repository as a library: cargoship_cluster converges a cluster – install, join and upgrade are one apply, because the engine version already on each host is what tells them apart – and cargoship_cluster_facts reads a fleet without changing it. It drives cargoship in process, so it still opens every SSH connection itself from the machine OpenTofu runs on.
It grew up here and moved out, so that it could be published under the name the provider registry requires; nothing in cargoship depends on it. See choice-tofu-provider-removal for what moved and what stayed.
Ansible Integration
colonel_byte.cargoship runs the workflows above as ordinary playbook tasks: cargoship_apply, cargoship_prepare, cargoship_reset, cargoship_kube_config, and cargoship_engine_config_sync, plus a cluster role that wraps them. Ansible supplies the inventory and runs one task for the whole fleet; cargoship still opens every SSH connection itself, from the management node the package was staged onto.
For driving zarf itself from a playbook, the colonel_byte.zarf collection lives in colonel-byte/ansible-collection-zarf. It grew up here and moved out; nothing in cargoship depends on it.
- Running Cargoship as an Ansible Module - the modules, their parameters, check mode, and what
changedmeans. - Generating an Inventory from Ansible - the group and host-variable translation, usable with or without the modules.
- The Ansible Container Image -
ghcr.io/colonel-byte/cargoship-ansible, which carries the binary and the collections together.
Configuration and Schemas
Cargoship relies on strongly-typed YAML definitions to govern its operations:
- Cluster Inventories: Specify SSH configurations, credentials, host roles, profiles, and load-balancer addresses.
- Distro Package Definitions: Map out the required binaries, OCI images, and layout settings.
- Distro Runtime Configs: Configure the underlying distribution engine.
Inventories and package definitions are Kubernetes-style documents under cargoship’s own API group:
apiVersion: colonel-byte.dev/v1alpha1
kind: Cluster # or Distro, for a package definition
metadata:
name: fleet-cluster
The corresponding JSON schemas are automatically generated from Go structs into schema/, and cargoship schema writes any of them on demand. When authoring configurations in modern editors, refer to these schemas for real-time validation and autocompletion; cargoship validate checks a file against the right one without an editor.
An inventory authoring guide is available in Setting up an inventory.
The zarf.dev group is deprecated
These documents used to declare apiVersion: zarf.dev/v1alpha1 with kind: ZarfCluster and kind: ZarfDistro. That group belongs to the upstream Zarf project, which defines its own kinds under it, so cargoship moved to a group it owns.
Both spellings are accepted, in any combination, and a document using the old one needs no change today:
| Old | New | |
|---|---|---|
| API group | zarf.dev/v1alpha1 | colonel-byte.dev/v1alpha1 |
| Inventory kind | ZarfCluster | Cluster |
| Package kind | ZarfDistro | Distro |
Everything cargoship writes itself - cargoship inventory from-ansible, the Ansible collection, the OpenTofu provider, the generated example packages - now emits the new spelling. No removal date is set for the old one; when one is, it will be announced in a release before it lands.
Development and Build Workflows
Task automation is built using Mage, which drives builds, tests and code generation with the host Go toolchain.
Mage Automation
Mage handles tasks including local compilation, e2e test execution, schema updates, and documentation generation. Key generated files include:
build/cargoship_*(Release binaries)docs/commands/*(Cobra command references)docs/phases/*(Orchestration phase explanations)docs/schema/*(Schema field reference, from the same struct reflection asschema/*.json)docs/ansible/<collection>/module_*.mdanddocs/ansible/<collection>/role_*.md(Ansible collection reference)docs/index.mdanddocs/security.md(this file and.github/SECURITY.md, with their links rewritten for the book)docs/SUMMARY.md(mdBook layout manifest)schema/*.json(YAML validations)
Continuous Integration (CI) and Releases
GitHub Actions workflows run lint checks, dependency validation, cross-compilation, and end-to-end tests for every pull request. Releases are managed through GoReleaser, automating build signing, verification, and asset publishing.
Inspiration
Cargoship draws major design and engineering inspiration from:
- k0sproject/k0sctl - For elegant SSH-based multi-node orchestration and configuration patterns.
- zarf-dev/zarf - For air-gapped image and file packaging and offline-first design.
Security Policy
Supported versions
Cargoship is pre-1.0 and releases from main on every conventional commit that warrants one, so fixes land in a new minor release rather than being backported. Only the most recent release line receives security fixes; upgrade before reporting an issue against an older tag.
| Version | Supported |
|---|---|
Latest v0.x release | Yes |
| Any earlier release | No |
Reporting a vulnerability
Report privately through GitHub: open a draft advisory. Do not open a public issue, and do not include a working exploit against a third party’s cluster.
You should expect an acknowledgement within 3 business days and an assessment – whether we consider it a vulnerability, and the severity we assign – within 10 business days. Confirmed issues are fixed in the next release, and we aim to publish the advisory within 90 days of the report, sooner when a fix is already out. You will be credited in the advisory unless you ask not to be.
Include the output of cargoship version, the host OS of the management node, the Kubernetes engine involved (K3s or RKE2), the commands that reproduce the behaviour, and what an attacker gains.
Scope
In scope: the cargoship CLI and the packages under the module root; package signing and the verification path (--verify, --key); credential handling in clustercfg, including the vault splice and age encryption; SSH orchestration against target hosts; and the release workflow and the artifacts it publishes.
Out of scope: vulnerabilities in K3s, RKE2, containerd, or any other upstream component Cargoship packages – report those to the project that owns them; anything that requires an attacker to already have root on the management node, which by design holds the cluster’s credentials; resource exhaustion from deliberately malformed or oversized packages supplied by the operator themselves; and misconfiguration of a user’s own cluster.
Verifying a release
Release archives, checksums.txt, and container images are signed with cosign using the key whose public half is cosign.pub in this repository, archives ship with an SBOM, and every tag also publishes ai-bom.json – the AI-usage disclosure manifest described below – attested and signed the same way.
cosign verify-blob --key cosign.pub --bundle checksums.txt.sigstore.json checksums.txt
cosign verify --key cosign.pub ghcr.io/colonel-byte/cargoship:<tag>
See the GoReleaser reference for the full set of published artifacts.
AI-assisted development
Cargoship’s code, docs, and design records are written with AI coding agents in the loop. See AI usage for what’s used and how it’s reviewed. Each release publishes ai-bom.json, a per-release snapshot of that disclosure – verify it the same way as the other release artifacts, above.
Security in CI
Pull requests and main are scanned by CodeQL, dependencies are updated by Dependabot, and supply-chain practices are scored weekly by OpenSSF Scorecard. Results appear in the repository’s Security tab. vendor/ costs points on a couple of Scorecard checks it can’t avoid given the repository’s need to build without network access; choice-osv-vendor-overrides and choice-binary-artifact-opa-wasm explain why.
age Encryption
This guide explains how Cargoship encrypts registry credentials in a cluster configuration with age, and how to read them back out. age is an alternative to Ansible Vault, not a replacement for it: the same commands write both formats, one configuration can hold both, and an apply reads whichever format each value carries.
For the Ansible Vault side of the same commands, see vault-encryption. Everything that guide says about YAML paths, anchors and aliases, dry runs, and the in-place rewrite is true here as well; this guide covers only what differs.
Why age
The draw is the key model rather than the cipher.
Ansible Vault is a shared passphrase. Everyone who has to run an apply has the same secret, which means the secret has to be distributed, and revoking one person’s access means rotating it for everybody.
age is public key encryption. You encrypt a credential to a set of public keys – the recipients – and it can only be read with one of the matching private keys, the identities. A machine that only has to encrypt needs no secret at all: a CI job that commits a configuration holds the public keys and nothing more. Revoking access means re-encrypting to a smaller recipient set, which is one command over the whole file and touches no other key.
Ansible Vault remains the better fit when a single operator or a single shared credential store is the whole story, and it is what an existing configuration already uses. Neither format is going away.
What Cargoship Decrypts
The same four fields per registry, in either format:
| Path | What it holds |
|---|---|
.spec.config.registries[N].auth.user | Registry username |
.spec.config.registries[N].auth.pass | Registry password |
.spec.config.registries[N].auth.token | Registry token |
.spec.config.registries[N].tls.ca | Inline CA certificate |
Cargoship tells the formats apart by the value’s own header. Ansible Vault ciphertext starts with $ANSIBLE_VAULT; age ciphertext is armored, so it starts with -----BEGIN AGE ENCRYPTED FILE----- and ends with -----END AGE ENCRYPTED FILE-----. A value carrying neither is plaintext and is passed through untouched.
Nothing records the format in the schema, and no flag selects it for reading. That is what lets one document mix the two. A document encrypted with age does carry a note of the age recipients it was encrypted to, but that is a record of what was done rather than a field anything reads a value’s format from – see What the File Records.
Keys
Making a Key Pair
Cargoship makes a key pair itself, so the age distribution is not something you have to install first:
cargoship vault keygen --output ~/.age/cargoship.key
It writes the identity – the private key, AGE-SECRET-KEY-1... – to the file with mode 0600, and prints the recipient – the public key, age1... – to stderr. It will not overwrite a file that already exists: replacing an identity makes every value encrypted to the old key unreadable, by anyone, permanently. Move the old file aside if you mean to replace it.
age-keygen -o ~/.age/cargoship.key does the same thing and writes the same file. The format is age’s, not Cargoship’s, which is the point: the key stays usable with age --decrypt and anything else that speaks it.
The public key is also in the file, on a # public key: comment line, so you can recover it later:
cargoship vault keygen --public-key ~/.age/cargoship.key
age-keygen -y ~/.age/cargoship.key is the equivalent, and -y is the short flag here too.
Omitting --output writes the private key to stdout instead, which makes the redirect work:
cargoship vault keygen > ~/.age/cargoship.key
chmod 600 ~/.age/cargoship.key
The chmod matters in that form and not the other: a redirect creates the file with whatever mode your shell gives it. Run bare, with no redirect, the command warns – the key is then on the screen and in that terminal’s scrollback.
The public key is not a secret. Commit it, publish it, put it in a ticket.
Recipients: the Keys That Encrypt
Three flags name recipients, and all three are repeatable:
--age-recipient age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
--age-recipients-file ./recipients.txt
A recipients file holds one public key per line. Blank lines are ignored, and a line beginning with # is a comment, so the file doubles as the list of who can read the configuration:
# alice, platform team
age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
# bob, platform team
age19h0ngeasgxd5vpcfavgavma2m39cmq3a2xlhggs6u0r5rtscx5ms0gw4jh
# the CI runner's deploy key
age1lggyhqrw2nlhcxprm67z43rta597azn8gknawjehu9d9dl0jq3yqqvfafg
The comment has to be on its own line. A # after a key on the same line is read as part of the key, and the file fails to parse with the line number. This is age’s own format, and age --recipients-file reads the same file.
SSH public keys go in the same flags and the same file; see Using an SSH Key.
Every recipient named across every flag is encrypted to, so a value can be readable by several people at once. This is the normal case, not a migration state.
When no recipient flag is given, Cargoship reads CARGOSHIP_AGE_RECIPIENTS, which holds whitespace-separated public keys:
export CARGOSHIP_AGE_RECIPIENTS="age1ql3z... age19h0n..."
A recipients file that parses to no keys at all is an error rather than nothing to do, because carrying on would fall back to writing Ansible Vault ciphertext under a password the operator did not ask to use.
Identities: the Keys That Decrypt
--age-identity-file ~/.age/cargoship.key
Repeatable, and a single file may hold several identities. Every identity Cargoship has been given is tried against a value, so holding keys for two teams in one file works.
When no identity flag is given, Cargoship reads CARGOSHIP_AGE_IDENTITY_FILE, which holds exactly one path. A list would need a separator, and every separator worth choosing is legal in a file name.
Using an SSH Key
An ssh-ed25519 or ssh-rsa key works anywhere an age key does, and goes in the same three flags. There are no --ssh-* flags, because there is no point in making an operator work out which kind of key they hold before choosing a flag:
cargoship vault encrypt-file ./cluster.yaml --age-recipient "$(cat ~/.ssh/id_ed25519.pub)"
cargoship vault encrypt-file ./cluster.yaml --age-recipients-file ./authorized_keys
cargoship vault decrypt-file ./cluster.yaml --age-identity-file ~/.ssh/id_ed25519
This is worth doing because the keys already exist. A team that maintains an authorized_keys file can encrypt a configuration to it as it stands – options in front of a key and comments behind it parse fine – and nobody has to generate, distribute, or lose a new key.
A recipients file may mix the two kinds freely, which is what makes a gradual move onto native age keys possible:
# alice, on her age key already
age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
# bob, still on his SSH key
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIExample bob@example.com
A line that is neither – an ecdsa-sha2-nistp256 key, say, which age does not support – fails the whole file with its line number rather than being skipped. Skipping it would encrypt to fewer people than the file lists, and the one who finds out is the one who cannot decrypt.
Three things differ from a native age key:
- A passphrase-protected private key is prompted for on a terminal, and fails without one. The prompt happens only once a value turns out to be encrypted to that key, and the decrypted key is then reused for the rest of the run. Under CI, or with input piped in, the command fails immediately and says so; it does not read the passphrase out of standard input and does not hang. If that is where the key has to be used, decrypt it first with
ssh-keygen -p. - The public key is not derived from the private one.
cargoship vault keygen --public-keyreads age identity files only.ssh-keygenalready wrote the public key into the.pubfile beside the key, andssh-keygen -y -f ~/.ssh/id_ed25519prints it again. - SSH recipients are not anonymous. See Recipients Are Anonymous below.
Upstream’s own advice is that these key types exist for compatibility with keys you already have, and that a native age key is preferable otherwise. cargoship vault keygen makes one.
Nothing Is Discovered
Cargoship reads key material from the flags, the config file, and those two environment variables. It reads nothing else. It does not look in ~/.config/sops/age/keys.txt, it does not look in $XDG_CONFIG_HOME/age, it does not read an age agent, and it does not pick your SSH key up out of ~/.ssh because it happens to be there.
The one path derived rather than given is the .pub file beside a passphrase-protected SSH key, which older key formats do not carry their public key inside. That path comes from the identity file the operator named on the command line, and is read only when that key needs a passphrase.
This matches ResolveVaultPassword, which has always behaved the same way, and it is deliberate: an apply that succeeds on one operator’s machine because of a file the configuration never mentions is an apply nobody can reason about.
Keys in the Config File
All three flags can be defaulted from the age section of cargoship-config.yaml, so a team can commit its recipient set once instead of restating it on every command:
age:
recipients:
- age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
- age19h0ngeasgxd5vpcfavgavma2m39cmq3a2xlhggs6u0r5rtscx5ms0gw4jh
recipients_files:
- /etc/cargoship/recipients.txt
identity_files:
- /home/operator/.age/cargoship.key
Each key is a list, and each is the equivalent of repeating its flag. A recipient in the config file counts as choosing age exactly as the flag does – see below.
Put only public keys in a committed config file. identity_files holds paths, not keys, so it is safe to commit when the path is a per-machine convention; it is still a per-operator choice, and the environment variable is often the better home for it.
Choosing the Format Cargoship Writes
Reading never involves a choice: the value’s header decides. Writing does, and Cargoship resolves it like this:
-
An age recipient given on the command line or in the config file wins, even when
CARGOSHIP_VAULT_PASSWORDorANSIBLE_VAULT_PASSWORDis set in the environment. Those variables are typically set once in a shell profile on a machine that already uses vault, and an operator who passes--age-recipientis asking for age, not for a complaint about a variable they set months ago. -
An age recipient and an explicit
--vault-password-filetogether is an error:age recipients and a vault password file were both given; pass one or the other, since a value is written in a single formatBoth flags name the key to write with. Picking one would mean writing a secret under a key the operator did not choose, which is worth stopping for rather than guessing well.
-
Otherwise Ansible Vault, exactly as before.
With no key material at all, every vault command fails up front:
no encryption key found: set --vault-password-file or the
CARGOSHIP_VAULT_PASSWORD/ANSIBLE_VAULT_PASSWORD environment variable, or pass
--age-recipient and --age-identity-file
An apply is the exception. A configuration holding no encrypted credential needs no key, so an apply accepts an empty keyring and only fails if it meets a value it cannot read.
The Commands
The cargoship vault group is the same group, and it keeps its name – renaming it would break every operator’s scripts:
cargoship vault encrypt [VALUE]
cargoship vault encrypt-path FILE YAML_PATH [YAML_PATH...]
cargoship vault encrypt-file FILE
cargoship vault decrypt [VALUE]
cargoship vault decrypt-path FILE YAML_PATH [YAML_PATH...]
cargoship vault decrypt-file FILE
cargoship vault rekey FILE
cargoship vault keygen [IDENTITY_FILE]
keygen is the only one of them that takes no key material, since it is the command that makes some.
Encrypting a Value
cargoship vault encrypt my-registry-password --age-recipient age1ql3z...
The output is an armored age block that you paste into the configuration at the right indentation, the same way a $ANSIBLE_VAULT block is pasted. vault encrypt-path does that part for you.
Passing a secret as an argument leaves it in your shell history and in the process list. Prefer stdin, or the hidden prompt you get when stdin is a terminal:
cargoship vault encrypt --age-recipient age1ql3z... < ./registry-token.txt
Decrypting a Value
cargoship vault decrypt --age-identity-file ~/.age/cargoship.key < ./encrypted-token.txt
An armored age value begins with dashes, so passing it as an argument does not work – the shell hands it to Cargoship intact, but the flag parser reads the leading ----- as a flag and fails with bad flag syntax. Use stdin, as above, or put -- after all of the flags so that everything following it is read as an argument:
cargoship vault decrypt --age-identity-file ~/.age/cargoship.key -- "$CIPHERTEXT"
$ANSIBLE_VAULT values do not have this problem, which is why it only comes up now.
When no configured identity can read the value, the error says what the value was encrypted to:
none of the configured age identities can decrypt this value; it is encrypted to
X25519 recipients
That is as specific as it can be. See Recipients Are Anonymous.
Encrypting a Whole Configuration
cargoship vault encrypt-file ./cluster.yaml --age-recipient age1ql3z...
Every registry credential in the file that is plaintext is encrypted to the recipients; anything absent, empty, or encrypted already in either format is skipped. Comments, key order, and the rest of the document survive byte for byte.
Running it twice changes nothing the second time, so it is safe in a script or a pre-commit hook, and running it over a half-encrypted configuration finishes the job.
There is one consequence of that worth stating plainly. encrypt-file never moves a credential between formats, and never re-encrypts one to a new recipient set. Running it with new recipients against a file whose credentials are already encrypted does nothing to the file. It does not do it quietly, though – every credential it left alone is reported, with the reason:
WRN left this credential as it was: it is encrypted to age recipients already, and cargoship
cannot tell whether they are the ones you named, because age ciphertext does not record its
recipients; run 'cargoship vault rekey' with an identity to re-encrypt it to the recipients
you want file=./cluster.yaml path=$.spec.config.registries[0].auth.pass
INF nothing to encrypt: every registry credential is encrypted already, or there are none to encrypt
The same warning appears when the formats differ, naming the one the value is in and the rekey invocation that moves it. Warnings are written to stderr, so --dry-run piped to a file still produces a clean document.
To change which keys a configuration is encrypted to, use rekey.
Decrypting a Whole Configuration
cargoship vault decrypt-file ./cluster.yaml --age-identity-file ~/.age/cargoship.key
Every encrypted registry credential becomes plaintext again, whatever format it was in, and anything that is not encrypted is skipped. If any value fails to decrypt, the command fails and the file is not touched at all.
Give both --age-identity-file and --vault-password-file to decrypt a document holding both formats. That combination is fine here: it is only encrypting that has to choose one format, and only then that giving both is an error.
Mixed Documents
A configuration can hold both formats at once, and not only while migrating:
registries:
- name: harbor.internal
auth:
user: robot
pass: |-
$ANSIBLE_VAULT;1.1;AES256
31333438393936323663373461353533373034663339316335386139363535383531643464613539
...
- name: ghcr.io
auth:
user: ci
pass: |-
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBB...
-----END AGE ENCRYPTED FILE-----
An apply given both a vault password and an identity file reads this document without complaint. A legacy credential nobody wants to touch can stay vaulted indefinitely while everything new is written to age.
Migrating from Ansible Vault
rekey is the migration, and there is no other command for it:
cargoship vault rekey ./cluster.yaml \
--vault-password-file ./vault-pass.txt \
--age-recipient age1ql3z...
Every vaulted registry credential in the file is read with the password and written back to the recipients, in one pass. The plaintext is never written to the file: each value is decrypted and encrypted again in memory, so there is no window in which the configuration on disk is readable, and nothing to clean up if the command fails partway. --dry-run prints the result to stdout and leaves the file alone.
The same rule that has always governed rekey applies: rotate the whole file, not one value at a time. If any encrypted value in the file cannot be read with the key given, the command stops before writing anything and names the path.
--new-vault-password-file and --age-recipient together is an error. Each names the key to rekey onto, and honouring one would mean silently ignoring the other.
To go the other way – age back to Ansible Vault – read with the identity and write with a new password:
cargoship vault rekey ./cluster.yaml \
--age-identity-file ~/.age/cargoship.key \
--new-vault-password-file ./vault-pass.txt
Revoking Access
Removing someone’s key from a recipients file changes nothing on its own. The ciphertext in the configuration was encrypted to them and still is. Re-encrypt it:
cargoship vault rekey ./cluster.yaml \
--age-identity-file ~/.age/cargoship.key \
--age-recipients-file ./recipients.txt
Reading needs an identity that is still in the file; writing uses the new, smaller recipient set. Afterwards the removed key can no longer read the configuration.
Treat the credential itself as compromised anyway if the person had it. Re-encryption removes their access to the file; it does not un-tell them a password they have already read. The registry credential should be rotated at the registry too.
Afterwards, metadata.encryption.age.recipients lists the set you rekeyed to, so the removal is visible in the diff rather than only in the re-randomized ciphertext. That list is a note rather than a check – see What the File Records – but it is the reason a second operator can tell, from the file alone, whether the revocation has happened yet.
If you are not sure whether a configuration has already been rekeyed, running encrypt-file with the recipient set you want is a safe way to ask: it re-encrypts nothing, and it warns if what the file records is not what you named.
Naming the recipient set the file already uses re-encrypts every value to those same recipients:
cargoship vault rekey ./cluster.yaml \
--age-identity-file ~/.age/cargoship.key \
--age-recipients-file ./recipients.txt
age re-randomizes every encryption, so each value comes back as different ciphertext holding the same plaintext for the same recipients – useful when the ciphertext has been somewhere you would rather it had not been.
Cargoship reports this as a rekey rather than as the re-salt it is, because it cannot tell the two apart: the recipients you name may or may not be the ones the values already carry, and the ciphertext does not say. With Ansible Vault, where omitting --new-vault-password-file unambiguously means “the key this file already has”, the same command is reported as a re-salt.
An identity on its own is not enough. rekey --age-identity-file key.txt with no recipients names nothing to write with and fails:
no vault password and no age recipients were provided
Checking Before an Apply
An apply verifies every encrypted value as soon as it has resolved the keys – before it connects to any host, and on a sync, before it starts draining nodes. A missing identity fails the command immediately rather than part-way through a cluster change.
With age this check matters more than it did with a shared password. A value encrypted to the wrong recipient set is indistinguishable from a correct one until something tries to read it, and nothing else in the pipeline will.
To check a configuration by hand without writing plaintext anywhere:
cargoship vault decrypt-file ./cluster.yaml --age-identity-file ~/.age/cargoship.key --dry-run > /dev/null
Or run the real preflight without touching a host:
cargoship apply ./cluster.yaml --age-identity-file ~/.age/cargoship.key --dry-run
Recipients Are Anonymous
An age header written to a native age1... recipient does not say who the value was encrypted to. The X25519 stanza holds an ephemeral share and nothing else – no key identifier, no fingerprint – by design, so that ciphertext does not leak its audience.
SSH recipients are the exception. An ssh-ed25519 or ssh-rsa stanza carries a short 32-bit hash of the public key, so someone holding a copy of the ciphertext and a list of candidate public keys can tell which of them it was encrypted to. If who can read a configuration is itself something you do not want in a committed file, use native age keys.
Either way, two things follow, because Cargoship reads no stanza at all – age keeps that parser internal, so nothing here can see even the recipient hash:
- Cargoship cannot tell you whether a value is already encrypted to the recipients you have in hand. It can only tell you that it is encrypted, and – since it writes down the recipients it used – what the file says it was encrypted to. Those are different claims, and the second one is a note, not a check; see What the File Records. This is why
encrypt-fileskips an encrypted value rather than checking it, why it warns about every value it skipped rather than staying quiet, and why changing recipients isrekey’s job. - An error about a value you cannot read can never name the key you are missing.
none of the configured age identities can decrypt this valueis the whole of what is knowable.
docs/agent/choice-age-encryption.md records the reasoning in full.
What the File Records
Because the ciphertext cannot say who it was encrypted to, the document says it instead. After an age encryption, metadata carries the recipient set the run used:
apiVersion: colonel-byte.dev/v1alpha1
kind: Cluster
metadata:
name: e72
encryption:
age:
recipients:
- age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... alice@laptop
lastModified: "2026-09-16T10:30:00Z"
spec:
config:
registries: ...
An SSH recipient keeps its authorized_keys comment, because that is the part that says whose key it is. Keys appear in the order you named them, and a key named twice – on a flag and again in a recipients file – appears once. lastModified is when Cargoship last rewrote the credentials in the document.
It Is a Note, Not a Guarantee
This is the one place in a Cargoship configuration where the file makes a claim about itself that nothing can verify, and it is worth being precise about what that means.
Cargoship writes this block and never reads it back as key material. Not on encrypt-file, not on rekey, not on an apply. rekey with an identity and no recipients still fails with no age recipients configured, exactly as it did before this block existed – it will not quietly encrypt to whatever the file happens to list.
That restriction is the whole safety argument. If any command encrypted to the keys named in the file, a one-line edit in a pull request would silently redirect every credential in it to the author’s key, and the resulting ciphertext would be indistinguishable from a correct one. So the block is inert: Cargoship writes it, compares against it, and acts on neither.
What that buys you, concretely:
- Reading the file tells you which keys it was encrypted to, which is otherwise unknowable without trying every private key you have.
- A code review can see a recipient set change, in the diff, as a line of YAML rather than as a wall of re-randomized ciphertext.
encrypt-filecan tell you that the recipients you just named are not the ones the file claims – which is a much more useful thing to hear than “skipped, and I cannot tell you why”.
And what it does not buy you: an edited list is a lie the file tells, and nothing in Cargoship will catch it. Treat the block the way you would treat a comment – accurate because the tooling wrote it and the diff is reviewed, not because anything enforces it.
When It Is Written and Removed
encrypt-filewrites it when it encrypted at least one credential to age. A run that encrypted nothing changes nothing, including the timestamp, so running the command twice still leaves the file byte for byte identical.rekeyrewrites it with the recipients it wrote to, and removes it when the target is Ansible Vault – a list of age keys that open nothing in the file is worse than no list at all.decrypt-fileremoves it, unless age ciphertext remains somewhere in the document. A recipient list above a file full of plaintext reads as a file that is still protected.- A
metadatamapping written in flow style –metadata: {name: e72}– has nowhere to put the block. Cargoship encrypts the credentials as usual and warns that it could not record them; writing a nested block into a flow mapping would leave a document that no longer parses, with the credentials already encrypted into it.
Nothing else in Cargoship touches the block. An apply ignores it, and a document without one works exactly as it did before.
When the Recipients Disagree
Running encrypt-file against a file that is already encrypted skips every credential, as it always has. When the file records a different recipient set from the one you named, the skip warnings are joined by one that says so:
WRN the credentials this run left as they were are recorded as encrypted to age recipients
other than the ones you named file=./cluster.yaml
recorded="age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p"
named="age19h0ngeasgxd5vpcfavgavma2m39cmq3a2xlhggs6u0r5rtscx5ms0gw4jh"
hint="run 'cargoship vault rekey' with an age identity and these recipients to re-encrypt it to them"
The comparison is on the keys themselves, so an authorized_keys comment added or removed, and a recipients file reordered, are not drift. The skip warnings beside it still say what they always did – that the value is age ciphertext and Cargoship cannot tell what it opens with. Both are true, and they answer different questions: one is about the ciphertext, this one is about what the document claims.
Older Binaries
The metadata block is new, and cargoship validate rejects fields it does not know about. A configuration carrying the block therefore fails validate on a Cargoship built before it existed. An apply is unaffected – it ignores unknown fields – and so is decryption, since the block has nothing to do with reading a value.
Interoperability with age
Cargoship writes the same armored format age --armor writes, so the tools read each other:
cargoship vault encrypt hunter2 --age-recipient age1ql3z... > ct.txt
age --decrypt --identity ~/.age/cargoship.key ct.txt
printf hunter2 | age --encrypt --armor --recipient age1ql3z... > ct.txt
cargoship vault decrypt --age-identity-file ~/.age/cargoship.key < ct.txt
What Is Not Supported
ssh-agent. A passphrase-protected SSH key is prompted for directly; a key held in an agent is not reachable, because age’s SSH support needs the private key itself rather than a signing oracle.- SSH key types other than
ssh-ed25519andssh-rsa.ecdsa-*andsk-*keys are not supported by age. - age passphrases (scrypt). age can encrypt to a passphrase instead of a public key. Cargoship does not expose that, because it is the thing Ansible Vault already does here – a shared secret – and having two ways to spell it would help nobody.
- Plugins (
age-plugin-yubikeyand friends). Cargoship links the age library rather than shelling out to theagebinary, and plugin support is a property of the binary.
Notes
- A failed command never touches the file. The rewrite goes to a temporary file alongside the original and is renamed over it, so an interruption cannot leave a half-written configuration.
- The file’s mode is preserved across a rewrite.
- Armored age ciphertext is longer than the plaintext it holds by a fixed overhead of a couple of hundred bytes, so a short password becomes a block several lines long. This is normal, and why every command writes it as a YAML block scalar rather than a quoted string.
Running the Collection from the Container
The cargoship-ansible image is a management node in a container: ansible-core on an AlmaLinux 10 base, plus the signed cargoship RPM, which carries the binary and the colonel_byte.cargoship collection. It exists so that a fleet can be converged from a playbook without staging Ansible, the collection and the binary onto a VM by hand. This guide covers what has to be mounted into it, why the package mount is read-only, and the two things that break a run before it reaches the fleet: the UID the container runs as, and host key checking.
For the modules themselves – their parameters and what changed means – see ansible-module, and for the inventory they translate, ansible-inv. Nothing about either changes inside the container.
podman pull ghcr.io/colonel-byte/cargoship-ansible:v0.37.0
The image has no ENTRYPOINT, because a run legitimately reaches for ansible-playbook, ansible-inventory, ansible-galaxy or cargoship itself; name the command you want. The collection is installed to /usr/share/ansible/collections, which is already on ansible-core’s default collections path, so ansible-playbook resolves colonel_byte.cargoship.cargoship_apply with no configuration. HOME is /home/nonroot, the working directory is /workspace, and the process runs as UID and GID 65532.
The image carries cargoship and the collection, and nothing for talking to the cluster afterwards. kubernetes.core is present because the base carries the full ansible distribution, but the Python client its modules import is not, so a k8s task fails at import time; there is no helm or kubectl either. A play that converges the cluster and then applies manifests to it needs an image of your own, built FROM this one with those added.
Every Path in a Playbook Is a Path Inside the Container
Cargoship runs in the container and connects to the fleet from there, so package, inventory_path, vault_password_file and ansible_ssh_private_key_file all name paths the container can see – not paths on the machine that started it. Mount each one at the same path it has on the host wherever you can. It costs nothing and it means the inventory reads the same whether the run happens in the container or on a management node that installed the package.
| What | Where | Mode |
|---|---|---|
| The distro package | wherever the playbook names it, e.g. /srv/staging | read-only |
| Playbooks, roles, the Ansible inventory | /workspace | read-only |
| The SSH private key | wherever ansible_ssh_private_key_file names it | read-only, 0600 |
| The vault password file | wherever vault_password_file names it | read-only, 0400 |
known_hosts | /home/nonroot/.ssh/known_hosts | writable, see below |
| Where a kubeconfig is written | /home/nonroot/.kube/config, or wherever kubeconfig names it | writable |
| The cargoship cache | /home/nonroot/.cargoship-cache | writable, optional |
An apply leaves update_kubeconfig on by default, which merges the cluster’s admin credentials into KUBECONFIG when it is set and ~/.kube/config otherwise – inside the container, a file under /home/nonroot that goes away with the container. Mount a kubeconfig, name one with the kubeconfig parameter, or pass update_kubeconfig: false and fetch it deliberately with cargoship_kube_config; what you should not do is leave the default and wonder where the credentials went.
Everything under /home/nonroot is already created and owned by 65532 in the image, so a run with none of the optional mounts works – it just starts cold each time and keeps nothing. The cache is the one worth adding first, and it has a section of its own below.
Reaching a Cluster on the Same Host
The same rule catches addresses, not just paths, and a local k3d cluster is where it bites first. k3d kubeconfig get <name> writes server: https://0.0.0.0:41609, naming the port k3d published on the host; 0.0.0.0 inside the container is the container’s own network namespace, where nothing is listening. A task given that kubeconfig fails on a connection refused that reads like a cluster that is down, and the cluster is fine.
Two addresses do reach it, and the serving certificate is what limits it to two. k3d starts k3s with --tls-san 0.0.0.0 and --tls-san k3d-zarf-serverlb, on top of k3s’s own 127.0.0.1 and localhost, so a name the certificate does not carry – host.docker.internal, or the bridge gateway address – fails verification even once it routes.
Joining the cluster’s own network is the arrangement to prefer. Rewrite the server to the load balancer’s name inside that network:
mkdir -p build/tmp
k3d kubeconfig get zarf | sed -E 's|https://0.0.0.0:[0-9]{5}|https://k3d-zarf-serverlb:6443|' > build/tmp/zarf-config.yaml
chmod 0644 build/tmp/zarf-config.yaml
Then attach the container to that network and mount what it wrote:
docker run --rm \
--network k3d-zarf \
-v "$PWD/build/tmp/zarf-config.yaml":/home/nonroot/.kube/config:ro,z \
-e KUBECONFIG=/home/nonroot/.kube/config \
-v "$PWD":/workspace:ro,z \
ghcr.io/colonel-byte/cargoship-ansible:v0.37.0 \
ansible-playbook init.yaml
Rewriting k3d’s own output rather than hand-writing a kubeconfig keeps the embedded CA and client certificate intact. docker is deliberate here: the k3d network is docker’s, and rootless podman cannot attach to it. The load balancer’s name resolves inside that network for as long as the cluster exists, so this survives a k3d cluster stop && k3d cluster start that republishes the API on a different host port.
Reaching the published port instead means host networking, since that port is bound on the host:
k3d kubeconfig get zarf | sed 's|https://0.0.0.0:|https://127.0.0.1:|' > kubeconfig.yaml
chmod 0644 kubeconfig.yaml
docker run --rm \
--network host \
-v "$PWD/kubeconfig.yaml":/home/nonroot/.kube/config:ro,z \
-e KUBECONFIG=/home/nonroot/.kube/config \
ghcr.io/colonel-byte/cargoship-ansible:v0.37.0 \
zarf tools kubectl get nodes
It is the shorter way to a one-off check, and it costs the container’s network namespace, which a run that also opens SSH connections to a fleet should not give up lightly. The port is whatever docker port k3d-zarf-serverlb 6443 reports, and it is reassigned every time the cluster is recreated. zarf tools kubectl is the smoke test to reach for because the image has no kubectl of its own.
Either way the kubeconfig is mounted read-only – zarf reads it and writes nothing back – and it has to be readable by UID 65532, which is what the chmod 0644 on the copy is for. A mounted directory has to be traversable by that UID too, so a playbook tree under a 0700 directory arrives as a /workspace the container cannot open, and ansible-playbook reports a playbook it could not find rather than a permission error.
None of this applies to a real fleet. A cluster cargoship installed is reached over the network like any other host, and cargoship_kube_config – or an apply’s update_kubeconfig, above – is what puts its credentials where a later task can find them.
The Package Mount Is Read-Only Because Nothing Writes to It
A distro package is the one input large enough that copying it into an image, or into the container’s writable layer, is a real cost – it carries every image and file the cluster installs. It is also read-only by nature: cargoship apply reads the package and writes to the fleet, never back into the artifact. Mounting it :ro costs nothing and turns a whole class of mistake, a run that mutates the staging directory every other run depends on, into an error at the moment it is attempted.
podman run --rm \
--userns=keep-id:uid=65532,gid=65532 \
-v /srv/staging:/srv/staging:ro,z \
-v "$PWD":/workspace:ro,z \
-v ~/.ssh/fleet_ed25519:/home/nonroot/.ssh/fleet_ed25519:ro,z \
-v ~/.ssh/known_hosts:/home/nonroot/.ssh/known_hosts:z \
-v ~/.cargoship-cache:/home/nonroot/.cargoship-cache:z \
ghcr.io/colonel-byte/cargoship-ansible:v0.37.0 \
ansible-playbook -i inventory.yaml converge.yaml
On a host with SELinux enforcing – Fedora, RHEL, and anything derived from them – a volume needs a label option or the container cannot read it. Use z, the shared label, for anything the host or another container also uses, which is all of the above. Z relabels the content privately to one container, and applying it to a shared staging directory takes that directory away from every other reader on the host, including the next container. Docker takes the same options; it is the label, not the runtime, that decides this.
Mounting the Cache Read-Write
The cache is where cargoship keeps what it fetches – OCI artifacts it pulls, and the release assets the example generation reads – under ~/.cargoship-cache, which is /home/nonroot/.cargoship-cache in the container. It is the only mount in this guide that exists purely to make the next run faster, and the only one that is worthless read-only: a cache that cannot be written to is a cache that is empty on every run, and a read-only mount turns what would be a slow run into a failed write.
podman run --rm \
--userns=keep-id:uid=65532,gid=65532 \
-v ~/.cargoship-cache:/home/nonroot/.cargoship-cache:rw,z \
-v /srv/staging:/srv/staging:ro,z \
-v "$PWD":/workspace:ro,z \
ghcr.io/colonel-byte/cargoship-ansible:v0.37.0 \
ansible-playbook -i inventory.yaml converge.yaml
rw is the default for a bind mount and is written out here only to say that it is meant: the package above it is ro on purpose, and the pair reads as a decision rather than an omission. Create the directory on the host first – mkdir -p ~/.cargoship-cache – because a bind mount whose source does not exist is created by the runtime as an empty directory owned by root, which the container cannot then write to.
The same UID rule as everywhere else decides whether the write succeeds, and there are three ways to satisfy it. --userns=keep-id:uid=65532,gid=65532 maps your host UID onto the one the container runs as, which is the right answer when the cache lives in your home directory and you want to look at it afterwards. chown -R 65532:65532 ~/.cargoship-cache is the right answer under Docker or rootful Podman, where there is no mapping to arrange. Podman’s :U option – -v ~/.cargoship-cache:/home/nonroot/.cargoship-cache:rw,z,U – recursively chowns the source to the container’s user for you, which is convenient and also permanent, so do not point it at a directory anything else on the host owns.
A named volume avoids the question entirely, and is the better arrangement on a dedicated staging host where nobody needs to read the cache from outside:
podman volume create cargoship-cache
podman run --rm \
-v cargoship-cache:/home/nonroot/.cargoship-cache \
-v /srv/staging:/srv/staging:ro,z \
-v "$PWD":/workspace:ro,z \
ghcr.io/colonel-byte/cargoship-ansible:v0.37.0 \
ansible-playbook -i inventory.yaml converge.yaml
A new named volume is populated from the image path it is mounted over, ownership included, so it arrives owned by 65532 with no mapping, no chown and no SELinux label option to choose. It survives --rm, and podman volume rm cargoship-cache is how you discard it deliberately.
To put the cache somewhere other than the default path, set it in the environment as DISTRO_ZARF_CACHE, or as zarf_cache in a cargoship configuration file – the modules take no parameter for it, so a playbook cannot pass it per task. The --zarf-cache flag does the same thing for a cargoship command run directly in the container – it is named for zarf’s cache, which is what it is, and it takes cargoship’s path. The default printed in docs/commands is the one this repository’s own cargoship-config.yaml sets, not the built-in ~/.cargoship-cache a container resolves, since viper searches the working directory for a configuration file and a checkout has one.
The UID Is What Actually Breaks First
The image runs as 65532, and a private key mounted from your home directory is owned by you and readable only by you. Under rootless Podman your UID maps to root inside the container by default, so the key arrives owned by a user that is not the one reading it, and the run fails on a key it can see and cannot open. --userns=keep-id:uid=65532,gid=65532 maps your host UID to 65532 inside, which makes every mount above land with an owner that matches the process.
Under Docker, or rootful Podman, there is no mapping to arrange: the file has to be readable by UID 65532 on the host. Give the staging directory and the key to that UID (chown -R 65532:65532 /srv/staging), which is the usual arrangement for a dedicated staging host. Running the image with --user "$(id -u):$(id -g)" instead looks simpler and is not: /home/nonroot and its subdirectories are owned by 65532, so the cache, known_hosts and Ansible’s own temporary directory all become unwritable, and the failures arrive late and read like something else.
When using SSH multiplexing (ControlMaster) with Ansible inside the container, configure ControlPath (or mount a tmpfs volume) where nonroot has write permissions. If an inventory points to a host home directory (e.g. /home/<user>/.ssh/sockets), mount a writable tmpfs with permissive mode (--tmpfs /home/<user>/.ssh/sockets:rw,mode=0777) or point ControlPath to /home/nonroot/.ssh/sockets. Also avoid using :z SELinux relabeling flags on the host ~/.ssh directory when mounting SSH keys, as relabeling host SSH directories can disrupt host sshd services.
Host Keys
Cargoship verifies host keys against ~/.ssh/known_hosts – /home/nonroot/.ssh/known_hosts in the container – and appends a key it has not seen before, which is trust on first use. A fresh container has no such file, so with nothing mounted every host in the fleet is trusted on the first connection of every run, and the file recording that decision is thrown away with the container. That is weaker than it looks: it is not a first use, it is a first use each time.
Two ways out, in order of preference. Mount a known_hosts file writable, as above, so the first run records what it saw and every later run verifies against it. Or point SSH_KNOWN_HOSTS at a file you seeded beforehand; a read-only mount is fine here, but only because every key is already in it – cargoship still has to be able to append when one is not, and a read-only file fails that write.
Setting SSH_KNOWN_HOSTS=/dev/null disables host key verification entirely. It is useful for a lab and is not a workaround for the paragraph above: a management node that accepts any host key accepts a machine standing in for a controller.
SSH Agents Work, With the Socket Mounted
Cargoship reads SSH_AUTH_SOCK and offers the agent’s keys, so an agent on the host serves a run in the container once its socket is mounted and the variable points at it.
podman run --rm \
--userns=keep-id:uid=65532,gid=65532 \
-v "$SSH_AUTH_SOCK":/run/ssh-agent.sock:z \
-e SSH_AUTH_SOCK=/run/ssh-agent.sock \
-v /srv/staging:/srv/staging:ro,z \
-v "$PWD":/workspace:ro,z \
ghcr.io/colonel-byte/cargoship-ansible:v0.37.0 \
ansible-playbook -i inventory.yaml converge.yaml
An agent is the better arrangement when the key is protected by a passphrase, since nothing has to hold the passphrase inside the container, and it leaves no key file to mount read-only and forget about. It is worse when the run is unattended, because an agent is a session that has to be unlocked by somebody; a key file with a dedicated fleet key is the right answer there.
Encrypting the Generated Inventory on Disk
When inventory_path is set to keep the generated ZarfCluster document across runs, that file normally holds decrypted registry passwords in plaintext with 0600 permissions. To prevent plaintext credentials from remaining on the host staging filesystem, pass age recipients to the role or CLI:
- name: Install cluster
ansible.builtin.include_role:
name: colonel_byte.cargoship.cluster
vars:
cargoship_action: apply
cargoship_package: /srv/staging/rke2.tar.zst
cargoship_inventory_path: /srv/staging/cluster-inventory.yaml
cargoship_age_recipients:
- age1wjqegc62gpyvp4yfdqfk4vclfgdh3awlv03rgthcje398a860p7qpglp6w
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO9OkpTcjwWLH2fo8SUTVjb2NYEc6w996Cwr0wL2cRDx operator@example.com
When cargoship_age_recipients (or --age-recipient on cargoship inventory from-ansible) is provided, cargoship encrypts every registry credential (user, pass, token, tls.ca) into age ciphertext blocks before writing the document to disk.
To decrypt the credentials at apply time, supply the private key via cargoship_age_identity_files:
cargoship_age_identity_files:
- /home/nonroot/.ssh/id_ed25519
- /srv/staging/keys/age.key
Mount the identity file into the container (e.g. -v ~/.ssh/id_ed25519:/home/nonroot/.ssh/id_ed25519:ro), ensure it has 0600 permissions owned by UID 65532, or set age.identity_files in cargoship-config.yaml. Cargoship reads the private key and decrypts the credentials in memory when executing phases.
Air-Gapped Runs Need No Network of Their Own
Everything the convergence installs comes out of the package, so the container needs a route to the fleet and to nothing else. It does not need a registry, a proxy, or a route off the management network, and the image pull is the only step in this guide that touches the internet – do it on the machine that has a route, save the image with podman save, and load it where it runs.
What Not to Mount
Do not mount the container runtime’s socket, a whole home directory, or /etc from the host. The image needs four kinds of thing – a package, a playbook tree, credentials for the fleet, and somewhere to write – and each of them is better named individually than swept in with a parent directory. A resolved Ansible inventory already carries credentials for hosts that are not this cluster, which is why the collection’s action plugin forwards only the variables cargoship reads; mounting your whole home directory undoes that care at the filesystem level.
Do not bake credentials into an image of your own with COPY. An image layer keeps what it is given even after a later layer removes it, and this image is published to a registry that the key’s owner does not control.
Generating an Inventory from Ansible
Cargoship can build its cluster inventory from an Ansible inventory you already have, deriving each host’s role from the Ansible groups it belongs to. This guide covers that translation: the group mapping, the host variables cargoship reads, and the rules the translation follows. The same translation runs inside a playbook, where cargoship is the Ansible module; the modules, the collection, and the cluster role are covered in Running Cargoship as an Ansible Module.
Ansible supplies the inventory and nothing else. It does not connect to the fleet, gather facts on it, or run a task per host. Cargoship opens every SSH connection itself, from the node it runs on – a management node outside the cluster, which is the node the package and images were staged onto. See choice-ansible-module for why the work is split that way.
What the translation reads
Cargoship does not parse an Ansible inventory file. Resolving an inventory means group membership, group_vars, host_vars, dynamic inventory plugins, and the precedence rules over all of them; Ansible does that and hands over the merged result. What cargoship reads is a JSON document holding that result:
{
"groups": {
"all": ["kc01", "kc02", "kw01", "db01"],
"controller": ["kc01", "kc02"],
"worker": ["kw01"],
"databases": ["db01"]
},
"hostvars": {
"kc01": {"ansible_host": "10.1.2.3", "cargoship_hostname": "distro-kc01"},
"kc02": {"ansible_host": "10.1.2.4", "cargoship_hostname": "distro-kc02"},
"kw01": {"ansible_host": "10.1.2.5", "cargoship_hostname": "distro-kw01"},
"db01": {"ansible_host": "10.1.2.9"}
},
"cluster": {
"name": "bubbles",
"loadbalancer": "bubbles-kc.test.com"
}
}
groups and hostvars are Ansible’s own, verbatim. cluster holds the settings that belong to the cluster rather than to any host, which an Ansible inventory has no way to carry: the cluster name, the load balancer address, and optionally profiles, registries, and values. Those are the same fields .spec.config takes in a hand-written inventory – see the inventory guide.
Translate it:
cargoship inventory from-ansible ./resolved.json -o ./inventory.yaml
cargoship validate ./inventory.yaml
The command reads stdin when given no file, and writes to stdout when given no --output, so it composes. --name and --loadbalancer override what the document says, which is how one projection of a fleet becomes more than one cluster without being rewritten.
Roles come from groups
By default the Ansible group named controller supplies the control-plane nodes, and the group named worker supplies the rest. Those are cargoship’s own role names, so the default guesses at no Ansible convention.
Set roleGroups to map cargoship’s two roles onto the group names your inventory actually uses:
{
"roleGroups": {
"controller": ["k8s_masters"],
"worker": ["k8s_workers", "k8s_infra"]
}
}
A role may take more than one group. Only controller and worker exist; naming any other role is an error.
Three rules govern what comes out:
- A host in none of the mapped groups is left out. The
db01host in the example above is not part of the cluster and does not appear in the generated inventory. A play’s inventory legitimately carries hosts that have nothing to do with Kubernetes. - A host in groups mapped to two different roles is an error, naming the host and both groups. There is no precedence rule, because there is no defensible one: picking a role silently would install a cluster nobody asked for.
- Controllers are written first, and the first controller becomes the cluster leader. Within a role, hosts come out in the order the mapping names the groups, and within a group, in the order the inventory lists the hosts. If it matters which node bootstraps the cluster, put it first in the first controller group.
If roleGroups names a group the inventory does not define, that is an error – a misspelled group name there has no other symptom. The default mapping is not held to that, so a cluster of nothing but control-plane nodes needs no worker group.
Host variables
Connection details come from the Ansible variables you already set. Everything cargoship needs beyond those comes from variables under a cargoship_ prefix.
| Inventory field | Variable | Default |
|---|---|---|
ssh.address | ansible_host | the inventory hostname |
ssh.user | ansible_user | root |
ssh.port | ansible_port | 22 |
ssh.keyPath | ansible_ssh_private_key_file | unset |
ssh.bastion | cargoship_bastion | unset |
hostname | cargoship_hostname | the inventory hostname |
profile | cargoship_profile | the host’s role |
privateAddress | cargoship_private_address | unset |
privateInterface | cargoship_private_interface | unset |
engine.labels | cargoship_node_labels | unset |
engine.taints | cargoship_node_taints | unset |
host | cargoship_host | unset |
environment | cargoship_environment | unset |
files | cargoship_files | unset |
ansible_host and cargoship_hostname are different facts and both are kept: the first is where cargoship connects, the second is what the node calls itself. When neither is set, the name the inventory knows the host by serves as both.
The structured variables – cargoship_host, cargoship_files, cargoship_bastion, cargoship_node_labels, cargoship_node_taints, cargoship_environment – take exactly the shape the inventory schema documents for the field they configure, so there is one vocabulary to learn rather than two:
cargoship_profile: control
cargoship_node_labels:
adrp.xyz/purpose-control: "true"
cargoship_host:
ports:
- port: "6443"
protocol: tcp
firewall:
rules:
- name: allow-metrics
action: allow
source: 10.0.0.0/8
port: "9100"
protocol: tcp
cargoship_bastion:
address: 10.0.0.1
user: jump
port: 22
A bastion is declared as its own variable rather than read out of ansible_ssh_common_args. Cargoship does not parse SSH argument strings, and an inventory that reaches its fleet through a jump host via ansible_ssh_common_args has to say so again here.
Misspelled variables are errors
A variable under the cargoship_ prefix that cargoship does not read is an error, and so is an unknown key inside one of the structured variables. Ansible has no notion of a variable belonging to anyone, so a misspelled cargoship_profil would otherwise sit in hostvars looking set and doing nothing, and a misspelled key would be dropped on the way in. A misspelled variable and an unset one are indistinguishable at install time, which is too late to find out.
Variables under the ansible_ prefix that are not in the table are ignored. There are hundreds of them, they belong to Ansible’s own connection plugins, and they are not cargoship’s to police.
Checking the result
The generated document is validated against the inventory schema before it is written, so a translation that produced something invalid fails here rather than ten minutes into an apply. Check it yourself as well, and read it: it is an ordinary cluster inventory, and it is the exact document cargoship installs from.
cargoship inventory from-ansible ./resolved.json -o ./inventory.yaml
cargoship validate ./inventory.yaml
cargoship apply ./package.tar.zst --config ./inventory.yaml --confirm
The generated file carries connection details and is written with mode 0600.
A playbook does not run these commands itself. The modules write and validate the same document, and the path they write it to is the inventory_path parameter described in Running Cargoship as an Ansible Module.
Running Cargoship as an Ansible Module
Cargoship runs inside a playbook as an Ansible module. The cargoship binary is the module: it is installed as a set of symlinks named cargoship_<action>, and a symlink’s name selects the action. There is no separate module package and nothing to install on the managed nodes.
The inventory a module takes is the one described in Generating an Inventory from Ansible – Ansible’s resolved groups and hostvars, plus a cluster block – and the group mapping and host variables documented there apply here unchanged. This guide covers the modules themselves: how the collection is installed, how a task is written, and what its result means. The parameter reference for each module and for the cluster role is generated from the collection and lives in the collection reference.
Ansible supplies the inventory and nothing else. It does not connect to the fleet, gather facts on it, or run a task per host. Every task runs on one management node outside the cluster – the node the package and images were staged onto – and cargoship opens every SSH connection itself from there. See choice-ansible-module for why the work is split that way.
Five modules ship, one per fleet action: cargoship_apply, cargoship_prepare, cargoship_engine_config_sync, cargoship_reset, and cargoship_kube_config. Package creation is not among them – it builds an artifact from a definition in a repository, which belongs in a pipeline rather than in a convergence run.
Working inventories and a playbook for each of the five are in ansible/examples, along with a script that translates an inventory without running a play.
- name: Converge engine configuration
cargoship_engine_config_sync:
package: /srv/staging/rke2-1.31.tar.zst
inventory_path: /srv/staging/generated-inventory.yaml
inventory:
groups: "{{ groups }}"
hostvars: "{{ hostvars }}"
cluster:
name: bubbles
loadbalancer: bubbles-kc.test.com
vault_password_file: /srv/staging/vault-pass
delegate_to: localhost
no_log: true
The task runs on the management node – connection: local when the Ansible controller is that node, delegate_to when they are separate. Cargoship connects to the fleet itself, from there.
Installing the collection
The modules live in the colonel_byte.cargoship collection. The RPM and deb packages install it to /usr/share/ansible/collections, which is on ansible-core’s default collections path, so on a management node that installed cargoship from a package there is nothing further to do.
Installing from a tarball or a checkout instead, install the collection, then point its module files at the binary. Each release carries a built colonel_byte-cargoship-<version>.tar.gz and a cosign bundle beside it; the collection is not published to Galaxy, because the node that installs it has no route to it.
cosign verify-blob --key cosign.pub \
--bundle colonel_byte-cargoship-0.37.0.tar.gz.sigstore.json \
colonel_byte-cargoship-0.37.0.tar.gz
ansible-galaxy collection install colonel_byte-cargoship-0.37.0.tar.gz
./hack/ansible-link-modules.sh ~/.ansible/collections/ansible_collections/colonel_byte/cargoship /usr/local/bin/cargoship
From a checkout, build the tarball first with hack/ansible-build-collection.sh ./bin/ansible, which is the same script the release pipeline runs.
That last step is not optional and has no equivalent in Galaxy’s own packaging. Ansible resolves a task name to a module file, so each action needs a file named after it; the files are symlinks to the one cargoship binary, which is what the packages create and what the script creates by hand. A collection installed without them fails with the module not being found.
The collection carries one action plugin per module. It runs inside ansible-playbook, on the interpreter Ansible is already running on, and does one thing: copy groups and hostvars into the module’s inventory parameter so you do not have to. Nothing is installed on the fleet, and running cargoship by hand still needs no Python.
It also accepts a parameters dict and merges it into the rest, which is how the role passes a caller-supplied set of parameters without templating the whole of a task’s arguments – something Ansible warns about, and rightly. A parameter given both directly and inside parameters is an error naming it.
The plugin forwards the four ansible_* connection variables listed in Generating an Inventory from Ansible and every cargoship_* variable, and drops the rest. A host’s resolved variables routinely carry credentials for things that are not this cluster, and a module’s parameters are written to a file on disk.
When defining profiles inside cargoship_cluster in YAML, note type constraints: concurrency (e.g. "1", "50%") and port definitions (ports[].port, e.g. "6443") are unmarshaled as strings by the underlying schema. Quote numeric strings in YAML to prevent JSON numeric unmarshaling failures.
Parameters
inventory takes the document described in Generating an Inventory from Ansible, groups and hostvars included. Everything else maps onto a flag of the command the module runs.
The parameter reference lives with the modules rather than here, so that it cannot drift from what the binary accepts: cargoship_apply, cargoship_prepare, cargoship_engine_config_sync, cargoship_reset, and cargoship_kube_config. Each page lists every parameter the module takes, its type, and the flag it renders. Modules is the index, and covers why the surfaces differ from one another.
Of the five, cargoship_engine_config_sync is the most Ansible-shaped thing cargoship does: it converges configuration across a fleet that is already installed.
A parameter the module does not know is an error naming it. A parameter left unset is left unset: the module passes no flag for it, so whatever your cargoship configuration file sets still applies. That is why label_nodes: false and omitting label_nodes are different.
Signature verification
public_key and verify are the package signature parameters, and they are the key-based ones only. Keyless verification identifies a signer against a Fulcio root and a transparency log, which needs network access a management node inside an airlock does not have. Verify a keyless-signed package with cargoship package verify before the play, where the flags and their exclusions are all available.
Key material
Key material is always named by a path, never given by value. Ansible writes a module’s parameters into a file on the managed node, and a password passed by value would be a password written to a file you did not choose the mode of. Set no_log: true on the task regardless: a binary module has no per-parameter no_log, so the task-level setting is what suppresses the arguments and the result.
The cluster role
The role is the shorter way to write the same task. It picks the module from cargoship_action, assembles the inventory parameter, and sets run_once, delegate_to, and no_log for you.
- name: Install the cluster
ansible.builtin.include_role:
name: colonel_byte.cargoship.cluster
vars:
cargoship_action: apply
cargoship_package: /srv/staging/rke2-1.31.tar.zst
cargoship_cluster:
name: bubbles
loadbalancer: bubbles-kc.test.com
cargoship_args:
vault_password_file: /srv/staging/vault-pass
timeout: 45m
Every variable the role takes, with its default, is on role_cluster. Anything the role does not name a variable for goes in cargoship_args, which is passed to the chosen module verbatim.
Every task in the role carries run_once: true. Cargoship converges the whole fleet in one run, so a play over the fleet’s own inventory would otherwise run a full convergence once per host. It defaults cargoship_no_log: true to protect decrypted credentials and fleet inventories from terminal and CI logs, which is why cargoship_show_result exists: the debug task prints cargoship’s sanitized report without exposing secrets.
Check mode and changed
--check runs cargoship’s dry run. Phases opt into it one at a time – a phase that has not said how it behaves under a dry run is reported and not run – so a check-mode task connects, detects, gathers facts, and validates the hosts, and reports the rest as phases it would have run.
Under check mode, changed means what Ansible means by it there: a real run would change something. It is true when a phase that knows how to tell had work outstanding – a fleet whose engine configuration has drifted – and false when every such phase found nothing to do.
cargoship_kube_config is the exception. cargoship kube-config has no dry run, so under --check the module reports skipped: true and runs nothing at all. That is what Ansible does with a module that has declared it does not support check mode; a binary module has nowhere to declare it, so it says so in its result instead.
What changed is worth
changed is built from what the phases themselves report, and a phase has to opt in. cargoship.changedSignal says how much of the run is covered:
| Signal | Meaning |
|---|---|
complete | Every phase the run reached said whether it changed anything. |
partial | Some did not. They are named in cargoship.changedUndeclared. |
unknown | No phase reported at all, so changed is a convention rather than an observation, and is reported true. |
Today the engine configuration sync phases report and the rest do not, so an ordinary run reads partial. That is deliberate: a phase that has said nothing cannot make changed true, and it is not treated as having changed nothing either. KubeConfig and LabelNodes are the two in the undeclared list that can genuinely change something – the local kubeconfig and the node role labels – so if a handler of yours depends on either, gate on the result rather than on changed.
cargoship.phasesRan and cargoship.phasesPlanned name the phases the run executed and, under check mode, the ones it reported instead of running. Ansible sees one result for the whole fleet, so these stand in for the per-host detail a task-per-host module would give you.
The result
{
"changed": true,
"msg": "synchronised engine configuration across the fleet",
"cargoship": {
"module": "engine_config_sync",
"inventoryPath": "/srv/staging/generated-inventory.yaml",
"inventoryKept": true,
"checkMode": false,
"command": ["cargoship", "engine-config-sync", "..."],
"phasesRan": ["Connect", "Detect OS", "Sync Registry Config Controller"],
"changedSignal": "partial",
"changedUndeclared": ["Connect", "Detect OS"]
}
}
command is the command line the module ran, so a failure can be reproduced by hand on the management node. inventoryPath is the generated document. A run that fails leaves it on disk whether or not you named the path, because it is the first thing to look at when a failure reads like the wrong cluster; a run that succeeds removes it unless the path was yours.
Cargoship’s own logging goes to stderr, which Ansible captures and shows on failure. Per-host detail lives there, not in the result: Ansible sees one task for the whole fleet, not one result per host.
Live progress
A fleet-wide apply is one Ansible task that runs for several minutes, so the modules report where they are while they run. The action plugin prints one line per phase transition:
[cargoship] Phase 01/12: Connect to hosts [running]
[cargoship] Phase 01/12: Connect to hosts [done]
[cargoship] Phase 02/12: Detect host operating systems [running]
The counter is the phase’s position in the list for that module, so docs/phases/apply.md and its siblings read as the same sequence. A phase that fails is printed [failed], ahead of the failure Ansible itself reports.
None of this needs configuring. The action plugin creates a temporary status file, points the module at it, polls it while the module runs, and removes it afterwards whether the run succeeded or not.
Driving the binary directly gets the same progress through the same mechanism: set CARGOSHIP_STATUS_FILE to a path and cargoship rewrites that file as each phase starts and finishes, holding one JSON object – phase, index, total, status (running, completed, or failed), timestamp, and error on a failure. The file is replaced atomically on every transition, so a reader either sees the previous phase or the current one, never a half-written object. It is removed when the run finishes. Leave the variable unset and cargoship writes nothing.
File Override Guide
This guide explains how to redirect the files Cargoship downloads when creating a package, so a build can run against an internal mirror or against assets staged on disk ahead of time.
It is the file-download counterpart to the registry override guide, which covers the same idea for container images.
What a File Override Does
A distro definition declares two kinds of external artifact. Container images are pulled from OCI registries. Files and OS packages are downloaded over HTTP from whatever URLs .spec.config.files and .spec.config.os.files name – https://rpm.rancher.io/..., https://github.com/..., and so on.
--registry-override has always covered the first kind. A file override covers the second: it rewrites the front of a file’s source URL at package-create time, so the bytes come from a mirror instead of the upstream host. On a build host that cannot reach the public internet, this is the difference between building a package and hand-editing every URL in the distro definition first.
This only affects where create downloads from. What ends up in the package is byte-for-byte what the definition declared, because the declared checksum is still enforced – see Integrity below.
Setting an Override
Overrides are specified as source=target pairs, either on the command line or in the config file. source is a URL prefix. target is either a replacement URL prefix or a local directory.
Flag
cargoship create ./distro-defs --file-override https://rpm.rancher.io=https://mirror.example.com/rpm-rancher
The flag is repeatable, so several hosts can be redirected in one create:
cargoship create ./distro-defs \
--file-override https://rpm.rancher.io=https://mirror.example.com/rpm-rancher \
--file-override https://github.com=https://mirror.example.com/github-releases
A comma-separated value works too, which is occasionally easier to pass through a wrapper script:
cargoship create ./distro-defs --file-override https://rpm.rancher.io=https://mirror.example.com/rpm,https://github.com=https://mirror.example.com/gh
Config File
The same mapping can be set under .distro.create.file_override in your config file, so it doesn’t need to be repeated on every invocation:
distro:
create:
file_override:
https://rpm.rancher.io: https://mirror.example.com/rpm-rancher
https://github.com: https://mirror.example.com/github-releases
https://pkgs.k8s.io: /srv/staged/pkgs-k8s
Flag values take precedence over the config file. They replace it rather than merging with it: passing any --file-override on the command line means the config file’s mappings are not used for that run, so a command line that overrides one host has to name every host it still wants redirected. This is the same behavior as --registry-override.
Building From a Local Directory
A target that is not an http:// or https:// URL names a directory of files already on disk, which lets a build run with no network at all and without standing up an internal HTTP server:
cargoship create ./distro-defs --file-override https://rpm.rancher.io=/srv/staged/rpm-rancher
The path under the source prefix is appended to the directory. A file declared as:
source: https://rpm.rancher.io/public/centos/9/x86_64/k3s-selinux.rpm
is then read from /srv/staged/rpm-rancher/public/centos/9/x86_64/k3s-selinux.rpm. Mirroring the upstream layout with wget --mirror, or with whatever your artifact store exports, is usually the least effort way to populate the directory.
A relative target is resolved against the working directory when the flag is parsed, not against the distro definition, so it means what it looks like it means from the shell you typed it in.
Prefix Matching
A source can be as broad as a host (https://github.com) or as narrow as a path prefix (https://github.com/rancher). When more than one override could match a file, the longest matching source wins:
cargoship create ./distro-defs \
--file-override https://github.com=https://mirror.example.com/gh \
--file-override https://github.com/rancher=https://mirror.example.com/rancher
Here, https://github.com/rancher/rke2/releases/... resolves against the more specific https://github.com/rancher override, while https://github.com/k3s-io/k3s/releases/... falls back to the broader one.
Matching is done on the parsed URL rather than on the raw string, and a path prefix has to end at a path separator. Both of those matter in practice:
- An override for
https://github.comdoes not matchhttps://github.com.example.invalid/..., which a plain string prefix would have captured. - An override for
https://github.com/rancherdoes not matchhttps://github.com/rancher-dev/....
The scheme, host, and port all have to match exactly. An override for https://files.example.com does not apply to http://files.example.com or to https://files.example.com:8443.
A file whose source is already a local path is never overridden. There is nothing to redirect.
Integrity
Rewriting where a file comes from does not change what the file has to be.
Every overridden file is checked against the shasum its distro definition declares, exactly as it would be without the override. If a mirror serves a modified, corrupted, truncated, or simply out-of-date artifact, the build fails with a checksum mismatch naming the file, and nothing is written into the package. A file that fails for any reason – a failed download, a failed extraction, a missing local file – fails the whole create rather than being skipped.
A file that declares no shasum cannot be overridden. Cargoship refuses the build and names the file. Without a declared checksum there is nothing to compare a mirror’s bytes against, and “the file came from wherever this flag pointed” is not a property worth shipping in an offline package. If you hit this, add the upstream checksum to the distro definition; that is the fix, and it improves the definition whether or not you use an override.
When a file declares an extractPath, the override applies to the archive that is downloaded, and the shasum continues to be checked against the extracted file rather than the archive – unchanged from how extraction already worked.
Auditing a Built Package
A package built with file overrides records two things in its distro.yaml, and the difference between them is the point.
.build.fileOverrides is the configuration the build was given, alongside .build.registryOverrides:
build:
fileOverrides:
https://rpm.rancher.io: https://mirror.example.com/rpm-rancher
https://github.com: /srv/staged/github
.build.fileSources is what that configuration actually did – one entry per file an override redirected:
build:
fileSources:
- path: files/0/k3s
declared: https://rpm.rancher.io/public/k3s
resolved: https://mirror.example.com/rpm-rancher/public/k3s
override: https://rpm.rancher.io
shasum: 0a15685b072d0afcc218eceea5a93216a6accb48ff9e122bbc53c643c87b4a21
The configuration alone cannot answer the questions an audit actually asks. An override can be configured and match nothing – the https://github.com entry above did not, since no file source started with it. And when several prefixes overlap, only the resolved entry tells you which one won for a given file. Each fileSources entry is self-contained: this file, from this place, verified against this digest.
Two properties worth knowing:
- A package built with no overrides has neither key, so it is byte-for-byte what it would have been before this feature existed.
- Entries are written in the order the files were staged, which is the order they appear in the definition, so
--reproduciblestill produces identical output.
resolved names a path on the build host when the override pointed at a local directory, as in /srv/staged/github above. That discloses that much of the build host’s layout to anyone who reads the package. It is recorded deliberately – a provenance record that omits where the bytes came from is not one – but it is worth knowing before you stage assets under a path you would rather not publish.
None of this is what makes the package trustworthy. The enforced checksums are. This is the record of how it was built.
Host Firewall Configuration Guide
Cargoship configures the firewall on every node that runs one. It picks the firewall per node, from the node’s operating system and from what is running on it, so a single inventory can target a mix of Enterprise Linux hosts running firewalld, Debian or Ubuntu hosts running ufw, and hosts that manage nftables directly. Firewall management is opt-in: it only runs when cargoship apply is given the --firewall flag.
Every node cargoship manages gets three things: the private address of every other node in the cluster is trusted, the engine’s pod and service CIDR blocks are trusted, and the ports listed in .host.ports are opened. Anything beyond that comes from the rules in .host.firewall.rules.
Backends
| Backend | Used when | Notes |
|---|---|---|
firewalld | the firewalld service is running | Cluster trust is applied with ipsets in the trusted zone; ports become a distro-exposed-ports service on the public zone |
ufw | ufw is installed and ufw status reports active | Cluster trust and rules are applied with ufw commands; ports become a cargoship-ports application profile in /etc/ufw/applications.d/cargoship |
nftables | nft is installed and either the nftables service is running or the host has an /etc/nftables.conf or /etc/sysconfig/nftables.conf | Everything cargoship writes lives in one table, inet cargoship, rendered to /etc/cargoship/nftables.nft and loaded in a single transaction |
Cargoship never runs ufw enable. A node whose ufw is inactive is left alone, because bringing up a default-deny firewall from a remote phase would cut the SSH connection cargoship is running over. Enable ufw yourself, with a rule that keeps SSH reachable, before pointing cargoship at the node.
A node whose only nftables content comes from kube-proxy or the CNI is deliberately not a match. Every node in a running cluster has a non-empty ruleset, so matching on that would claim hosts whose operator never configured a firewall at all.
Choosing a backend
The node’s operating system decides first. Each OS module names the firewall front end its distribution ships: firewalld on Enterprise Linux and SUSE, ufw on Debian and Ubuntu. Distributions that ship no front end – Alpine, Arch, CoreOS, Flatcar, Slackware – name none.
| The node’s OS ships | And that front end is | Cargoship configures |
|---|---|---|
| firewalld or ufw | running | that front end |
| firewalld or ufw | installed, but stopped or inactive | nothing on the node |
| firewalld or ufw | not installed | whichever backend matches, in the order above |
| no front end | – | whichever backend matches, in the order above |
The second row is the deliberate part. An operator who installed a front end and left it down has made a decision about the node’s firewall posture, and cargoship does not take that decision away: it neither starts the service nor reaches past the stopped front end to write rules into the nftables underneath it, where the front end would overwrite them the moment it came up. Cargoship logs the node it skipped, and the apply continues.
This affects a stock Ubuntu node, which ships ufw installed and inactive. Run ufw enable on those nodes, with a rule that keeps SSH reachable, before cargoship apply --firewall.
The fallback match runs in the table order above, and nftables is last on purpose: firewalld and ufw are both front ends onto nftables, so a host running either would match the nftables backend as well. The nftables backend is for the hosts that never had a front end – Debian and Arch nodes configured by hand, and the minimal or immutable images such as CoreOS and Flatcar that ship no firewall front end at all.
Ports
The .host.ports list is the simple case, and it works the same on every backend. It opens a port, or an inclusive port range, to any source:
spec:
hosts:
- hostname: distro-kc01
role: controller
host:
ports:
- port: 6443
protocol: tcp
- port: 30000-32767
protocol: tcp
Rules
The .host.firewall.rules list is the backend-neutral rule model. Each rule needs an action; every other field is a match, and an omitted match means “any”.
| Field | Meaning |
|---|---|
name | Names the rule. It must be unique within a host, and cargoship uses it to name the files and rule comments it writes on the node. Cargoship derives one from the match fields when it is omitted |
action | allow, deny, or reject |
direction | in (the default), out, or forward |
source | The address or CIDR traffic comes from |
destination | The address or CIDR traffic goes to |
ingress | Where forward traffic enters. Forward rules only |
egress | Where forward traffic leaves. Forward rules only |
port | A port, or an inclusive low-high range. A port match also needs a protocol |
protocol | tcp, udp, sctp, or dccp |
spec:
hosts:
- hostname: distro-kc01
role: controller
host:
firewall:
rules:
- name: allow-metrics
action: allow
source: 10.0.0.0/8
port: "9100"
protocol: tcp
- name: ingress-http
action: allow
direction: forward
ingress: public
egress: trusted
port: "80"
protocol: tcp
Rules on a profile
Rules can also be set on a profile, under .spec.config.profiles.<name>.host.firewall.rules. Every host that selects the profile gets them. This is the usual place for a rule that belongs to a role rather than to one machine, e.g. opening the API server port on every control-plane node.
spec:
config:
profiles:
control:
host:
ports:
- port: 6443
protocol: tcp
firewall:
rules:
- name: allow-metrics
action: allow
source: 10.0.0.0/8
port: "9100"
protocol: tcp
hosts:
- hostname: distro-kc01
role: controller
profile: control
host:
firewall:
rules:
- name: allow-backup
action: allow
source: 10.0.9.4
port: "2049"
protocol: tcp
The host above ends up with both rules. Firewall rules union with the profile’s rather than replacing them, so a host can add one rule of its own without giving up the baseline its profile sets. Rules are matched by name, and a host rule wins over a profile rule of the same name – that is how a host overrides one rule from its profile, or turns a profile allow into a deny for itself.
Rules with no name are matched by their fields instead, so a host that repeats a profile rule verbatim still only gets it once. Name your rules if you intend to override them.
The rest of .host still replaces rather than unions: a host that lists any ports of its own inherits none of its profile’s, and the same holds for the legacy policy map.
How rules land on each backend
The backends express the same rule differently, and two fields mean something slightly different on each.
| Rule | firewalld | ufw | nftables |
|---|---|---|---|
direction: in or out with an address match | A rich rule on the public zone | ufw allow in ... / ufw allow out ... | A rule in the input or output chain, matching on ip saddr or ip6 saddr |
direction: in with only a port match | firewall-cmd --add-port on the public zone. Only action: allow is expressible this way | ufw allow in from any to any port ... | A tcp dport or udp dport rule in the input chain |
direction: forward | A policy file in /etc/firewalld/policies, where ingress and egress name zones | ufw route allow ..., where ingress and egress name interfaces | A rule in the forward chain, where ingress and egress name interfaces matched with iifname and oifname |
Because ingress and egress name zones on firewalld and interfaces on the other two, a forward rule is the one part of the model that is not portable across a mixed-OS cluster. Set forward rules on a profile that only matches hosts of one OS family.
Idempotency
Every backend can be applied repeatedly without accumulating rules. On firewalld, cargoship writes the same ipset, service, and policy files each run and lets firewall-cmd skip what is already present. On ufw, cargoship records the rules it applied in /var/lib/cargoship/ufw.rules; on the next run, any rule in that file that the current inventory no longer asks for is deleted from ufw before the current rules are applied. Rules cargoship owns are tagged with a cargoship: comment, visible in ufw status, so they can be told apart from rules added by hand. Rules added by hand are never touched.
On nftables, idempotency needs no state file: the rendered ruleset is cargoship’s whole desired state, and it is loaded with an add table, delete table, and redefinition of inet cargoship in one transaction, so a rule the current inventory dropped is gone once the transaction commits. The add before the delete is what keeps the script working on a node cargoship has not configured before.
nftables and the cluster’s own rules
Kube-proxy in nftables mode, and every CNI, keep their service and network policy rules in the same subsystem cargoship writes to. Cargoship never flushes the ruleset and never reads or writes a table other than its own, because a global flush would break cluster networking until those components resynced.
Two consequences are worth knowing. Every base chain cargoship writes has an accept policy, the same reason cargoship never runs ufw enable: a default-drop policy applied from a remote phase would cut the connection cargoship is running over. And an accept in cargoship’s chain ends only that chain, not the netfilter hook, so a rule in an operator’s own table can still drop traffic cargoship allowed. Unlike firewalld’s trusted zone, cargoship’s trust is not the last word on a packet.
Cargoship’s rules survive a reboot through an include "/etc/cargoship/nftables.nft" line added to whichever boot-time file the distro reads, /etc/nftables.conf or /etc/sysconfig/nftables.conf. That file is appended to, once, and never rewritten. A host with neither file gets a warning rather than an error: the rules are loaded for this boot, but nothing will reload them at the next one.
Legacy firewalld policies
The older .host.policy map still works and is still firewalld-only. It maps a policy name to a firewalld policy written verbatim to /etc/firewalld/policies/<name>.xml:
spec:
hosts:
- hostname: distro-kc01
role: controller
host:
policy:
k8s-forward:
target: ACCEPT
ingress:
name: public
egress:
name: trusted
ports:
- port: "80"
protocol: tcp
Prefer a direction: forward rule under .host.firewall.rules for new configuration; it does the same thing on firewalld and also works on ufw and nftables hosts.
Adding a backend
A backend is a firewall.Backend implementation in pkg/firewall: a Name, a Detect that reports whether the backend manages a given node, an Installed that reports whether the firewall is present on the node at all, and an Apply that makes the node match a firewall.Plan. Register it in the backends list in pkg/firewall/firewall.go, in match order, and the apply phase picks it up with no further wiring.
A backend that is a distribution’s own front end also needs the OS modules for that distribution to return its Name from PreferredFirewall, in types/os/linux. That method is declared on each leaf OS module rather than on a shared base such as EnterpriseLinux, because those modules embed both their base and configurer.Linux at the same depth, which leaves a method promoted from either one ambiguous.
Importing Archives Guide
This guide explains how to convert an OCI Image Layout - a Skopeo oci-archive file, or an unpacked layout directory - into a native Cargoship package (.tar.zst) with cargoship import.
The Air-Gap Workflow
In an air-gapped workflow, packages are often published to an OCI registry on an internet-connected staging network, then copied to removable media using container tools like Skopeo.
Because Cargoship packages are multi-platform artifacts whose OCI manifest lists record the platform OS as "multi" (inherited from the OCI packaging specification), pass --override-os multi when pulling with Skopeo so its platform filter matches:
# Export the package from an OCI registry as an oci-archive
skopeo copy --override-os multi \
docker://registry.example.com/cargoship/my-package:1.0.0 \
oci-archive:my-package.tar:1.0.0
Once transferred across the airlock into the disconnected environment, cargoship import converts the archive into a native Cargoship package archive that cargoship apply can install.
Basic Usage
Run cargoship import on the archive:
cargoship import my-package.tar
The command:
- Unpacks the archive, if it is not already a directory, and reads its
index.json. - Selects the manifest matching the target architecture, descending into a nested index for a multi-arch package.
- Extracts and verifies all package layers (binaries, engine configurations, container images, and checksums).
- Writes the native compressed archive (
cargoship-<name>-<arch>-<version>.tar.zst) into the current directory.
To write the native package to a specific directory, pass -o, --output:
cargoship import my-package.tar --output /srv/cargoship/packages/
Importing Unpacked Directories
If the archive has already been extracted on disk as an OCI layout directory (containing oci-layout, index.json, and blobs/), pass the directory path directly:
cargoship import ./extracted-oci-dir/ -o ./build/
Other Tools That Write an OCI Layout
Any tool that writes an OCI Image Layout can feed cargoship import. Two common ones:
# oras: copy a package out of a registry into a layout directory, or into a layout tar
oras cp --to-oci-layout registry.example.com/cargoship/my-package:1.0.0 ./my-package:1.0.0
oras backup --output my-package.tar registry.example.com/cargoship/my-package:1.0.0
# crane: --format=oci writes a layout directory (its default, tarball, does not work - see below)
crane pull --format=oci registry.example.com/cargoship/my-package:1.0.0 ./my-package
Why There Is No docker-archive Support
cargoship import reads an OCI Image Layout only. A Docker image archive, the format docker save and crane pull --format=tarball write, cannot carry a Cargoship package, for two independent reasons:
- No tool can produce one. A Cargoship package is an OCI artifact with a custom config media type (
application/vnd.cargoship.config.v1+json). Skopeo refuses to write it as adocker-archive, failing withunsupported image-specific operation on artifact with type "application/vnd.cargoship.config.v1+json";orashas nodocker-archiveoutput at all; and the Docker daemon cannot pull an artifact with a non-image config, sodocker savenever sees one. - The format cannot carry what the import needs. A
docker-archive’smanifest.jsonrecords onlyConfig,RepoTags, andLayers. There is no field for OCI annotations, soorg.opencontainers.image.title- which is how a package layer declares whether it isdistro.yaml,checksums.txt, or an image blob - is dropped. Even a hand-builtdocker-archivewould arrive with its layers unidentifiable.
Use oci-archive or an OCI layout directory instead. Every tool above can write one.
Verifying Signatures
If the package was signed before export, pass --key and --verify to validate its Cosign signature as part of the import:
cargoship import my-package.tar --verify=always --key ./cosign.pub
Multi-Architecture Packages
A distro package normally targets one CPU architecture. A package can instead target several, so that one artifact installs a cluster whose control-plane and worker hosts do not all run the same architecture. This guide covers declaring the architectures, selecting files per architecture, and how such a package is built, published, pulled, and applied.
Declaring Architectures
metadata.architectures lists every architecture the package targets:
apiVersion: colonel-byte.dev/v1alpha1
kind: Distro
metadata:
name: rancher-k3s-multi
version: 1.36.4-k3s1
architectures:
- amd64
- arm64
The supported values are amd64, arm64, and riscv64. Listing the same architecture twice is an error, as is listing one cargoship does not support.
metadata.architecture, the scalar field, stays valid for a package targeting a single architecture and means exactly the same thing as a one-entry list. Set one or the other, not both: when both are set the list wins. A definition that sets neither takes the architecture of the machine running cargoship create.
Selecting Files Per Architecture
Most upstream projects publish one binary per architecture, at a different URL and with a different checksum. Write one file entry per architecture, each landing on the same target, and let selector.arch decide which one a host receives:
files:
- source: https://github.com/k3s-io/k3s/releases/download/v1.36.4%2Bk3s1/k3s
target: /usr/local/bin/k3s
executable: true
shasum: 835873f37245fc615f547a2fe2af9402a347875f13fa64a1f136de644955ea3f
selector:
package: binary
arch:
- amd64
- source: https://github.com/k3s-io/k3s/releases/download/v1.36.4%2Bk3s1/k3s-arm64
target: /usr/local/bin/k3s
executable: true
shasum: c920706346d5ad4e5cd3c7bf1bb09ce71ebe07fec829e513e40f1caf98aed8bb
selector:
package: binary
arch:
- arm64
An entry with no arch selector goes to every host, the same way an entry with no profile selector goes to every role. That is what the architecture-independent files – unit files, scripts, noarch RPMs – want.
selector.arch may only name architectures the definition declares. A file selecting x86_64 in a package targeting amd64 fails at create time rather than silently never uploading. example/k3s-multi, example/rke2-multi-cni-canal and example/rke2-multi-cni-cilium are complete definitions written this way, generated from the same templates as the single-architecture examples, and the example/upstream-cilium, example/upstream-canal and example/upstream-flannel flavors are generated the same way, installing Kubernetes from upstream packages for both architectures plus the CNI each one bakes in.
Building
cargoship create builds every architecture the definition declares into one package. Images are pulled for each of them, so a two-architecture package with images takes roughly twice as long to build and is roughly twice the size.
The archive is named cargoship-<name>-<arch>-<version>.tar.zst, and a package covering more than one architecture takes multi in the architecture position, since no single architecture describes what it carries:
cargoship-rancher-k3s-multi-multi-1.36.4-k3s1.tar.zst
Narrowing With --architecture
--architecture (-a) narrows a definition to one of the architectures it declares, which is how you build a single-architecture package from a multi-architecture definition:
cargoship create example/k3s-multi/v1_36/v1.36.4-k3s1 -a arm64
It narrows; it does not override. Asking for an architecture the definition does not declare is an error, because the file entries for that architecture were never written. For the same reason it cannot re-target a definition that names a single architecture: -a arm64 against an architecture: amd64 definition fails rather than producing an amd64 package labelled arm64. A definition that names no architecture at all is the one case where the flag chooses freely, since there is nothing to contradict.
Narrowing does not invalidate the entries for the architectures left behind. A definition declaring amd64 and arm64 narrowed to amd64 still carries its arm64 file entries, and those entries are still checked against the full declared list.
Publishing
A package is one set of blobs whatever the architecture count, so it publishes as a single manifest. The image index tagged at the package version is what makes it resolvable per architecture: it lists that one manifest digest once for every architecture the package covers.
--tag names the tag instead. On publish it changes the OCI reference alone, leaving the package’s own metadata.version as built, which is what a promotion between registries wants: the same package published under staging and then under a release tag. On create it goes further and overrides metadata.version itself, so the package is built as that version and the index it publishes to is tagged accordingly. Both work per architecture the same way the version does, so an amd64 and an arm64 package given the same tag resolve under one index.
cargoship publish cargoship-rancher-k3s-multi-multi-1.36.4-k3s1.tar.zst oci://registry.example.com/distros
Ordinary platform resolution therefore works from either architecture, and every consumer lands on the same blobs. Publishing to a tag that already holds an index leaves the entries for architectures the package does not cover alone, so publishing an amd64 package and then an arm64 one of the same version leaves both resolvable under the one tag.
Because example packages and templates evolve as new features and settings are introduced, packages published under a general release tag may be updated across Cargoship releases. If an environment requires an immutable, consistent image, either build the definition locally using cargoship create or pull the package using its precise tag containing the short commit SHA (e.g. <version>-<job>-<short git commit>).
Pulling
cargoship pull resolves the architecture of the machine it runs on, or the one given with -a, against the index:
cargoship pull oci://registry.example.com/distros/rancher-k3s-multi:1.36.4-k3s1 -o .
A multi-architecture package resolves from any architecture it covers, and the pulled archive is identical whichever one resolved it.
When nothing in the index matches and the package holds a single manifest, that manifest is pulled anyway: it is the only thing a matching platform could have resolved to, and an arm64 package is worth having on an amd64 workstation to inspect, sign, or push elsewhere. Pulling a package is not running it. An index holding several distinct manifests and none for the requested architecture is still an error, since choosing between those is what platform matching is for.
Applying
Each host in the inventory receives the files its own architecture selects, so a mixed-architecture cluster installs from the one package: an arm64 worker gets the arm64 binary, an amd64 controller gets the amd64 one, and both get the entries with no arch selector. Images are likewise exported per host architecture, so each host loads a tarball holding the images built for it.
Package Values
A cargoship package ships with a set of values: Helm-style configuration, addressed by dotted paths, that a cluster can override without rebuilding the package. Values reach the engine configuration in two ways – structural mappings, and Go templates that read .Values – and they are also available to the files a package uploads and to the actions that build it.
This guide covers authoring values in a package, overriding them from a cluster inventory or the command line, and choosing between mappings and templates.
Declaring values in a package
Values live under spec.values of a distro.yaml:
spec:
values:
files:
- values.yaml
- overrides/prod.yaml
schema: values.schema.json
files are merged in order, key by key, with a later file winning over an earlier one. Each entry is a path relative to the package definition, an absolute path, or a URL. Cargoship copies them into the package as values/<index>-<basename>, so two files that share a base name do not collide.
schema is a JSON Schema document the merged values must satisfy. Cargoship checks the values against it while building the package and refuses to build when they do not match, so a typo is caught by whoever builds the package rather than by whoever installs it. The schema must not use $ref. Set "additionalProperties": false on the root object: without it, a misspelled top-level key in a cluster inventory is silently ignored instead of rejected.
A values file is ordinary YAML:
cilium:
encryption:
enabled: false
hubble:
ui:
enabled: false
ipv6: false
Overriding values
Values are resolved from three sources, each winning over the one before it:
- The values files the package ships with.
spec.config.valuesin the cluster inventory.- One or more
--values <file>flags on the command line.
The inventory carries the values a cluster always wants:
kind: Cluster
spec:
config:
values:
cilium:
encryption:
enabled: true
The flag carries the ones that change between runs:
cargoship apply ./package.tar.zst --config inventory.yaml --values ./encryption-off.yaml
--values may be given more than once, with a later file winning over an earlier one, and it is accepted by cargoship apply, cargoship prepare, and cargoship engine-config-sync. There is no -f shorthand.
Merging is per key, not per document. An override that sets cilium.encryption.enabled leaves every other key under cilium as the package defined it.
The merged values are checked against the package’s schema again at install time, so an inventory that sets a key the package does not accept fails before anything is written to a host.
That check happens when the package is installed, which is late to learn that a key is misspelled. cargoship schema inventory --package <package> composes the package’s values schema into the inventory schema, so an editor completes and checks spec.config.values while the inventory is being written:
cargoship schema inventory --package ./package.tar.zst -o ./inventory.schema.json
--package accepts a package source directory as well as a built one, so a package author can point an editor at a definition that has not been built yet. The composed schema describes the override subtree on its own, while cargoship validates the overrides merged with the values the package ships, so required is dropped from it – a key the package’s own values.yaml supplies is not required of the operator. It catches an unknown key, a wrong type, and a name outside an enum or pattern; it does not claim to reproduce install-time validation. See the inventory guide for wiring it into an editor.
cargoship validate runs that same composed schema without an editor, which is the form that fits CI:
cargoship validate ./inventory.yaml --package ./package.tar.zst
Mappings
A mapping projects one value onto one place in the engine configuration:
spec:
values:
mappings:
- source: .cilium.encryption.enabled
target: .manifest.rke2-cilium.encryption.enabled
source is read out of the resolved values. target is written relative to spec.config.engine. Both paths must begin with a dot. A source the values do not define is left alone, so the package’s own engine configuration stands as the default.
Mappings are structural: the value keeps its YAML type, and nothing in the package has to be written as a template. They are the right tool when a value lands on exactly one key.
A mapping that cannot be applied – a path without a leading dot, or a target underneath a chart entry that is a YAML string rather than a map – is reported as a warning when the package is built, and again when it is installed. It does not fail the build. The feature is new, and a hard failure on a package that installs correctly today is the wrong first move.
Templates
A template reads .Values directly, anywhere in the engine configuration:
spec:
config:
engine:
manifest:
rke2-cilium:
encryption:
enabled: "{{ .Values.cilium.encryption.enabled }}"
A scalar that is entirely one template is re-typed after rendering, the same way Helm’s --set typing works, so the example above lands in the generated HelmChartConfig as a YAML boolean rather than the string "true". A scalar that mixes text and templating stays a string.
Rendering happens before the mappings are applied, and before the engine configuration is validated. Package-authored text is therefore what gets executed: a value supplied by a cluster inventory that happens to contain {{ is written through as literal text, never evaluated.
Conditionals
Templating can do the one thing mappings cannot: decide whether a block of configuration exists at all. Rendering walks the parsed structure one scalar at a time and never re-parses a document, so a conditional only works inside a chart entry written as a YAML block scalar:
spec:
config:
engine:
manifest:
rke2-cilium: |-
kubeProxyReplacement: true
encryption:
enabled: {{ .Values.cilium.encryption.enabled }}
{{- if .Values.cilium.hubble.ui.enabled }}
hubble:
enabled: true
ui:
enabled: true
{{- else }}
hubble:
enabled: false
{{- end }}
Both shapes are in the examples: example/rke2-cilium-vsphere spends its values through mappings, and example/rke2-multi-cni-cilium spends the same values through a templated block scalar.
Which to reach for
Use a mapping when a value has one destination and its type should survive untouched. Use a template when one value drives several keys, when the value needs reshaping on the way, or when the package has to emit or omit a whole block. A package may use both.
A chart entry carries more than its values
Everything above writes a chart entry under spec.config.engine.manifest as the chart’s Helm values and nothing else. An entry may instead be a wrapper, which names the values under a values key and says two more things about the chart:
spec:
config:
engine:
manifest:
rke2-cilium:
enabled: true
images:
- docker.io/rancher/mirrored-cilium-cilium:v1.20.2
- docker.io/rancher/mirrored-cilium-hubble-ui:v0.13.6
values: |-
kubeProxyReplacement: true
| Key | What it does |
|---|---|
values | The chart’s Helm values, exactly as the bare form carries them - a YAML string or a mapping, templated and mapped the same way |
images | The offline images this chart needs, carried in the package beside spec.config.imageConfig.images |
enabled | Whether those images reach a host. Defaults to true. This is not an install switch - the engine’s disable key, described below, is what leaves a chart out of the cluster |
Why images belong to a chart
imageConfig.images is one flat list, and nothing about it records which chart an image is for. So a cluster that turns Hubble’s web interface off is still sent mirrored-cilium-hubble-ui and -ui-backend, and a cluster running no vSphere CSI is still sent its ten driver images - exported to a tarball, pushed over SSH to every node, and left on disk. On an air-gapped link that is the expensive kind of waste.
Naming an image under the chart that needs it is what lets cargoship skip it. A chart with enabled: false has none of its images exported or uploaded, and because the upload phase removes image tarballs it did not write this run, disabling a chart on a later apply takes its images back off the hosts.
The package still carries every image. cargoship create pulls all of them and a published package holds all of their blobs, whatever any chart’s enabled says - a package whose contents depended on the cluster it was going to be installed on would be a different package per cluster. The saving is in what crosses the wire to the hosts, not in the size of the tarball an operator carries across the air gap.
A chart that is not enabled also gets no HelmChartConfig, for the same reason a disabled one does not: there is nothing to configure when the images are not on the node.
Deciding it from an inventory
enabled is an ordinary place for a value to land, so a cluster decides it:
spec:
values:
mappings:
- source: .addons.vsphere.csi.enabled
target: .manifest.rancher-vsphere-csi.enabled
Mappings are applied before any phase runs, so by the time the upload phase picks images the answer is in hand.
A chart is only switchable if the package says so. enabled defaults to true, and so does an entry that omits it – including every entry written in the bare form. A cluster reaches spec.config.engine only through a mapping or a template the package itself declares; a cluster’s own engine section carries node labels and taints and nothing else. So a chart whose entry has no enabled key, and no mapping onto one, cannot be turned off by an inventory or by --values at all. Exposing the switch is the package author’s decision, and not exposing it is how a package says a chart is not optional.
Note that enabled is about the chart’s images, not about whether the engine installs it – the disable key below is that, and the two are not wired together. See todo-chart-disable-image-gating for what that currently costs.
Both shapes are accepted, and one is going away
An entry is read as a wrapper when it has a values key, and as the chart’s values themselves when it does not. That is the whole rule, and it is deliberately narrow: a chart’s own values may well have a top-level enabled or images key - cilium has enablePolicy, and rancher-vsphere-csi really does take an images list - so neither of those can tell the two shapes apart. The cost is that a chart whose values have a top-level values key cannot use the wrapper, and has to stay in the bare form.
Two consequences worth knowing:
- A typo in a wrapper key is silent. Because a bare entry is any mapping,
enabledd: falseis indistinguishable from a Helm value by that name. It validates, and the chart’s images ship. Nothing warns. - The bare form is what
v1beta1drops. It is supported for as long asv1alpha1is, and a package written today needs no change; a package written now for a chart that will grow images is better off in the wrapper.
Disabling bundled addons
RKE2 and K3s both install a set of charts of their own – ingress, a load balancer, metrics, DNS – and both are told to leave one out through a single disable key in config.yaml. Every example package exposes that key as a value, so which addons a cluster runs is a cluster’s decision rather than a packaging one:
# values.yaml, in the package
addons:
disabled:
- rke2-ingress-nginx
# distro.yaml, in the package
spec:
values:
mappings:
- source: .addons.disabled
target: .config.disable
An inventory names the charts it wants left out:
spec:
config:
values:
addons:
disabled:
- rke2-ingress-nginx
- rke2-traefik
- rke2-traefik-crd
The list is replaced, not merged. Values merge per key, and a list is one value, so an inventory that sets addons.disabled decides the whole list. Repeat the entries the package disables by default – rke2-ingress-nginx above – or they come back.
This works because a mapping copies the value as it stands, lists included. Templating cannot produce a list: rendering re-types a fully templated scalar into a boolean, a number, or a string, never into a sequence. A value that has to reach the engine as a list has to arrive through a mapping.
The names are the chart names the engine itself uses, and which ones a build packages is a property of that build rather than of the engine: RKE2 1.37 carries rke2-gateway-api-crd and rke2-security-responder, RKE2 1.34 carries neither, and K3s calls its charts by bare names (coredns, servicelb, traefik, local-storage, metrics-server, runtimes). Rather than keep a list here that drifts from what is packaged, every example’s values.schema.json carries its own version’s charts as suggestions under addons.disabled, which an editor offers as completions. That list is generated from the engine’s source – see mage and thirdparty-src – so it is the vocabulary that build advertises. Read the schema next to the distro.yaml you are installing.
The suggestions are not the whole vocabulary. The engine’s --disable help text is where the list comes from, and it is not exhaustive: RKE2 bundles rke2-runtimeclasses and never names it there. A schema that rejected everything outside the list would reject a chart the engine would have honored, so addons.disabled accepts any chart name and the list only drives completion. An unknown name reaches config.yaml unchanged: cargoship warns that the build packages no such component and leaves the entry in place, since dropping it would quietly install the very chart the cluster asked to be without. A typo is visible in that warning rather than at validation time.
Two other things follow from disabling a chart:
- No
HelmChartConfigis written for it. A package may configure a chart underspec.config.engine.manifestand still have it disabled by an inventory. Cargoship skips the manifest for a disabled chart rather than configuring something that will not be installed, and a file left over from an earlier run is removed the next time the node is synced. A chart whose entry setsenabled: falseis skipped the same way, for the same reason - its images are not on the node either. - Agents are unaffected.
disableis a server-only key. It is dropped from an agent’sconfig.yamlalong with every other controller-only key, and only controllers carry chart manifests in the first place.
Pod Security Admission levels
Every example package writes a Pod Security admission configuration, and the level it enforces is a value rather than a packaging decision:
# values.yaml, in the package
podSecurity:
defaults:
enforce: restricted
audit: restricted
warn: restricted
# distro.yaml, in the package
spec:
values:
mappings:
- source: .podSecurity.defaults.enforce
target: .podSecurity.plugins[0].configuration.defaults.enforce
- source: .podSecurity.defaults.audit
target: .podSecurity.plugins[0].configuration.defaults.audit
- source: .podSecurity.defaults.warn
target: .podSecurity.plugins[0].configuration.defaults.warn
Each key takes one of privileged, baseline, or restricted, which the schema enforces, and all three default to restricted. A cluster that has to run a workload the restricted profile rejects can lower one of them without rebuilding:
spec:
config:
values:
podSecurity:
defaults:
enforce: baseline
The three are separate on purpose. enforce is the only one that rejects a pod; audit and warn record it and tell the client about it. Lowering enforce to baseline while leaving audit and warn at restricted runs the workload and still reports every pod that would not have passed, which is the shape to reach for while a workload is being brought up to the profile rather than exempted from it.
Only the levels are values. The -version keys, the exemptions, and the plugin itself stay with the package: an exemption list is a property of what the package installs – kube-system, rook-ceph, and zarf are exempted because the charts in the package need it – and a cluster that edited it would be deciding for components it did not choose. Anything beyond the three levels is a package edit.
What renders
Templating is deliberately not applied everywhere. Three surfaces render:
-
spec.config.engine– always, without an opt-in. This is the subtree that becomes the engine’s configuration and itsHelmChartConfigmanifests. -
spec.config.filesandspec.config.os.files– per file, when the entry setstemplate: true. It defaults to false, because most files are binaries or archives that a template pass would corrupt.spec: config: files: - source: audit-policy.yaml target: /etc/rancher/rke2/audit-policy.yaml template: trueThe file is rendered once, in place in the extracted package, before any phase uploads it.
-
spec.actions.onCreate– per action, when the action setstemplate: true. This renders at build time, on the machine building the package, and covers the action’scmdand itswaitblock.
spec.config.os.sysctl, spec.config.os.environment, spec.config.os.fapolicyd, and spec.config.os.selinux do not render.
A template that names a value the package does not define is an error, not an empty string, on every surface.
Rendered manifests on a running cluster
The HelmChartConfig manifests a template produces are written to the engine’s manifest directory, which the Rancher engines re-read on their own. Cargoship marks them as not requiring a restart: cargoship engine-config-sync compares them on disk, writes the drift in place, and logs the paths it wrote. No node is drained for a values change that lands only in manifests.
Available functions
Templates are rendered with the sprig-compatible function set from sprout, which is what a Helm chart author already expects: toYaml, upper, b64enc, default, quote, the arithmetic and list helpers, and the rest.
Four functions are removed: env, expandEnv, expandenv, and getHostByName. A package author writes the templates and an operator runs them, so leaving these in place would let a package read the operator’s environment into chart values, or resolve names against the operator’s DNS. Helm removes the same four for the same reason. They are removed rather than stubbed, so a package that uses one fails to parse instead of quietly rendering an empty string.
Five functions Helm defines and sprig itself lacks are added back: toToml, fromToml, toYamlPretty, fromYamlArray, and fromJsonArray. As in Helm, these report a parse failure in band – in the returned value – rather than failing the render.
Create-only functions
Four more functions are available to onCreate actions, and only there:
| Function | Purpose |
|---|---|
fileExists <path> | Whether a path exists on the machine building the package. |
cachedFileExists <relative path> | Whether a file exists under cargoship’s cache directory. |
cachedFilePath <relative path> | The absolute path of a file in cargoship’s cache directory. |
downloadToCache <url> <relative path> [sha256] | Downloads a URL into the cache and returns the local path. |
Pulling a missing artifact into the cache is the whole point of a create action, so these belong there. They are not available to values, manifests, files, or anything that runs at install time: a template that renders chart values should not be able to probe the deploy host or issue HTTP requests. A template that names one outside an action fails at parse time.
Each of these returns its error rather than a zero value, so a cache directory that cannot be read stops the build instead of reading as “the file is not there” and sending the package down the wrong branch.
Because the create-time function set is cargoship’s own, cargoship – not the vendored Zarf action runner – renders onCreate actions. Zarf still executes them, with its shells, retries, timeouts, and output capture unchanged, and .Constants and .Variables still resolve. Actions are rendered one at a time, in order, so a variable one action sets is readable by the next.
spec:
actions:
onCreate:
before:
- description: pull the chart
template: true
cmd: |
{{- if not (cachedFileExists "charts/cilium.tgz") }}
curl -sSLo {{ cachedFilePath "charts/cilium.tgz" }} {{ .Values.cilium.chartURL }}
{{- end }}
One consequence of resolving values before the build runs: an onCreate action cannot produce a values file that the same build then reads. Values are resolved, schema-checked, and packaged before the first action runs.
Verifying a package
Build the package and read the rendered output back:
cargoship create ./example/rke2-multi-cni-cilium/v1_36/v1.36.4-rke2r1
cargoship apply <package> --config inventory.yaml --dry-run
Values are resolved, the schema is checked, and every surface is rendered before the first phase runs, so a bad value or a template that names a key the package does not define fails ahead of any change to a host. A dry run stops there.
Per-Profile Update Concurrency
This guide explains why a cluster admin may want to set a different concurrency limit per node profile, and how to configure it.
The problem with one global concurrency limit
-c/--concurrency and -w/--work-concurrency cap how many hosts, or how many workers, Cargoship touches at once, but that limit applies flat across the whole cluster. A cluster is rarely uniform, though. It often mixes roles with very different blast radii if something goes wrong mid-update:
- Infra nodes may run stateful services, ingress, or storage where taking more than one down at a time risks an outage or data unavailability.
- General worker nodes may be numerous, stateless, and behind a load balancer, where updating a quarter of them at once is safe and finishes far faster than one at a time.
A single global limit forces a tradeoff: set it low enough to protect the sensitive infra nodes, and worker upgrades crawl; set it high enough for fast worker upgrades, and infra nodes risk being updated too aggressively.
Setting concurrency per profile
Every profile under .spec.config.profiles accepts its own concurrency, which overrides the CLI’s -c/-w value for hosts using that profile only. It accepts either a fixed count or a percentage of the hosts that share the profile:
spec:
config:
profiles:
infra:
concurrency: "1"
worker:
concurrency: "25%"
With this configuration:
- Hosts with
profile: infraare always updated one at a time, regardless of the-wflag value. - Hosts with
profile: workerare updated 25% at a time, rounded up, with a minimum batch size of 1 so a small worker pool never stalls at “0 at a time.” - A profile that sets no
concurrency(or a host with no profile) falls back to the-wflag value, same as before this feature existed.
A percentage is resolved against the number of hosts in that profile, not the whole cluster, so 25% on a 4-host worker profile means 1 host at a time, and 25% on a 40-host worker profile means 10 at a time.
This applies everywhere Cargoship batches worker operations: initializing, upgrading, draining, deleting, and uninstalling the engine.
Choosing values
- Use a fixed low count (
"1","2") for profiles running singleton or quorum-sensitive services, where losing more than N nodes at once breaks availability. - Use a percentage (
"25%","50%") for horizontally scaled, stateless profiles where speed matters more than caution, and the remaining capacity can absorb the load while a batch updates. - Use
"100%"(or leaveconcurrencyunset with-w 0) for profiles where all hosts can safely update simultaneously, e.g. a dev/test cluster.
Registry Override Guide
This guide explains how to redirect the images Cargoship pulls when creating a package, and how to authenticate to the resulting registry at apply time using an Ansible Vault-encrypted credential.
For the same idea applied to the files and OS packages a distro definition downloads over HTTP, see the file override guide.
What a Registry Override Does
By default, cargoship create pulls images from the registries referenced in your source manifests (e.g. docker.io, ghcr.io). A registry override rewrites the registry portion of an image reference at package-create time, so images are instead pulled from a mirror – for example an internal pull-through cache or an air-gapped registry.
This only affects where create pulls from. It doesn’t touch the cluster the package is later applied to; for authenticating the cluster’s container engine to a registry, see Authenticating to the Registry below.
Setting an Override
Overrides are specified as source=override pairs, either on the command line or in the config file.
Flag
cargoship create --registry-override docker.io=mirror.example.com
The flag is repeatable, so multiple registries can be overridden in one create:
cargoship create \
--registry-override docker.io=mirror.example.com \
--registry-override ghcr.io=mirror.example.com/ghcr
Config File
The same mapping can be set under .distro.create.registry_override in your config file, so it doesn’t need to be repeated on every invocation:
distro:
create:
registry_override:
docker.io: mirror.example.com
ghcr.io: mirror.example.com/ghcr
Flag values take precedence over the config file when both are set.
Prefix Matching
A source can be as broad as a registry (docker.io) or as narrow as a specific repository prefix (docker.io/library). When more than one override could match an image, the longest matching source wins:
cargoship create \
--registry-override docker.io=mirror.example.com \
--registry-override docker.io/library=mirror.example.com/library-cache
Here, docker.io/library/nginx resolves against the more specific docker.io/library override, while docker.io/someorg/app falls back to the broader docker.io override.
Authenticating to the Registry
Once images are being pulled from your mirror, the cluster’s container engine also needs credentials for it at apply time. This is configured separately, in the cluster inventory file’s .spec.config.registries (see the inventory guide):
spec:
config:
registries:
- name: docker.io
proxy:
url: https://mirror.example.com:5000
auth:
user: myuser
pass: hunter2
name is the registry the images were originally referenced from, and proxy.url is the mirror the engine pulls from instead. Cargoship writes both into /etc/rancher/<engine>/registries.yaml: the pair becomes a mirrors entry, and the credentials become a configs entry keyed by the mirror’s host and port. proxy.url may carry a scheme and a path. A URL given without a scheme is completed to https://, and the host is taken from it for the configs key, which is what the engine matches on.
Cargoship writes registries.yaml as root with mode 0640, so the engine’s group can read the mirror list while the credentials in it stay off limits to everyone else. Cargoship does not copy user and pass into registries.yaml as they are given. It encodes the pair into a single credential – base64 of user:pass – and writes that as the engine’s auth directive, so the password does not sit on every node as plain text. Base64 is an encoding, not encryption, so treat the resulting file as a secret regardless.
Use token on its own when the registry issues one directly. It is a different credential from a password, not another way to spell one, so Cargoship writes it as the engine’s identity_token directive rather than as basic auth. A token given alongside user and pass wins:
auth:
token: abc123
proxy is optional. Leave it out to authenticate a registry Cargoship pulls from directly, without redirecting anything – a private registry that hosts your own images, say:
- name: nexus.example.com
auth:
user: myuser
pass: hunter2
That writes a configs entry keyed by nexus.example.com and no mirrors entry. An entry that carries neither a proxy.url, credentials, nor tls configures nothing, and apply rejects it by name rather than writing an empty entry. So does a proxy block with no url in it, and a registry listed twice, which would otherwise mean the second entry quietly replacing the first.
The rest of the entry is checked the same way, when the inventory file is read and before apply connects to anything. name is the registry part of an image reference, so it carries no scheme and no repository path: docker.io, not https://docker.io and not docker.io/library. (* is the exception – engines read it as every registry.) proxy.url has to name a host and use http or https, and any credentials in it belong in auth instead. A proxy.rewrite pattern has to compile as a regular expression and rewrite to something, since the engine compiles it on the node, where failing means a failed pull rather than a rejected document. user and pass are a pair; half of one authenticates nothing and reaches you as a 401 from the registry rather than as the missing half it is.
Several registries may share one mirror – a single Nexus proxying docker.io, ghcr.io, and quay.io under different paths is the usual shape – and cargoship writes one configs entry for the host they have in common. Those entries have to agree: if two registries resolve to the same host but ask for different credentials or different TLS settings, only one of them could survive into the file, so apply reports the conflict by name instead of picking one.
Encrypted credentials are the exception. Both age and Ansible Vault salt each encryption, so encrypting one password twice produces two different blocks, and two entries carrying the same credential would read as a mismatch for as long as they are still ciphertext – which is the whole time this check runs, since it runs before anything is decrypted. So when every credential field both entries set is encrypted, and neither sets token, cargoship takes them as agreeing rather than reporting a conflict that exists only in the ciphertext. Everything else is compared as written: plaintext credentials that differ still conflict, differing TLS settings still conflict, and so does a token on either side, encrypted or not. A token is a whole credential by itself, so two of them cannot be assumed to be one credential encrypted twice.
Writing pass: hunter2 in plaintext works, but puts a real credential in the inventory file. Cargoship also accepts an encrypted value in user, pass, token, or tls.ca, in either of two formats: an Ansible Vault block starting with $ANSIBLE_VAULT, or an age block starting with -----BEGIN AGE ENCRYPTED FILE-----. The header tells them apart, so nothing has to be told which format a file uses, and a file part-way through a migration between them applies the same as one that is not. Either is decrypted automatically when the package is applied. The rest of this page uses Ansible Vault; see the age guide for the age format and its keys.
Registry TLS
A registry serving a certificate the host does not already trust needs a tls block. It takes paths to files already on the host:
spec:
config:
registries:
- name: docker.io
proxy:
url: https://mirror.example.com:5000
tls:
caFile: /etc/pki/ca-trust/source/anchors/mirror-ca.pem
certFile: /etc/ssl/certs/mirror-client.pem
keyFile: /etc/ssl/private/mirror-client-key.pem
caFile verifies the registry’s certificate; certFile and keyFile are the client certificate the registry authenticates the host with, and go together. Setting insecureSkipVerify: true turns certificate verification off for that registry – it makes the connection trivially interceptable, so prefer distributing the CA and verifying against it.
Distributing that CA to every host beforehand is often the annoying part, so ca takes the PEM-encoded certificate inline instead:
tls:
ca: |
-----BEGIN CERTIFICATE-----
MIIDdzCCAl+gAwIBAgIEbGVhZjANBgkqhkiG9w0BAQsFADBaMQswCQYDVQQGEwJV
...
-----END CERTIFICATE-----
Cargoship writes it to /etc/cargoship/tls/<host>.crt on each host – <host> being the same host the configs entry is keyed by, with anything outside A-Za-z0-9._- replaced by an underscore – and sets caFile to that path for you. The certificate travels with the cluster configuration, so a rotated CA reaches the nodes the same way a changed mirror does: apply sees the file drift, then drains, rewrites, and restarts each node in turn.
Set ca or caFile, not both. apply rejects an entry that sets both, and one whose ca is not a PEM certificate, naming the registry. A certificate is public, so encrypting one buys nothing, but a ca given as an Ansible Vault value is accepted and decrypted like any other – a document vaulted as a whole should not have to be picked apart. The certificate is checked once it is decrypted, which is the first point at which there is a certificate to look at.
Cargoship owns /etc/cargoship/tls outright: every file in it was written for a registry entry. Drop the ca from an entry, or drop the entry itself, and the next apply removes the certificate that was written for it along with the reference to it, rather than leaving it on every node for however long the cluster lives.
Keeping the Nodes in Step
apply compares what it would write against what is already on each node, and syncs only the nodes that differ. Both halves of a file count: contents, and the mode it is written with. A registries.yaml that someone has since widened to 0644 is rewritten the same as one with the wrong mirror in it, since the mode is what keeps the credentials in it off limits.
Syncing a node means draining it, rewriting the files, restarting the engine, and uncordoning it once it is ready again – one node at a time. The log line before each drain names the files that drifted and why, so a cluster-wide rewrite is legible while it happens.
That matters on upgrade. Cargoship 0.21 changed how registries.yaml is written: keys under mirrors and configs are quoted, an endpoint given without a scheme is completed to https://, a user and pass pair is encoded into a single auth credential, and the file is written 0640 instead of 0600. None of that changes what the engine does, but all of it changes the file, so the first apply after upgrading rolls through every node in the cluster once. Plan for it the way you would plan for any rolling restart.
Encrypting the Credential
Use cargoship vault-encrypt to produce the encrypted value. It reads the plaintext from an argument, from stdin, or – if stdin is a terminal – prompts for it with hidden input so the secret is never echoed or left in shell history:
cargoship vault-encrypt --vault-password-file ./vault-pass.txt
Value to encrypt:
$ANSIBLE_VAULT;1.1;AES256
62306432326630316632646366363136303161316635343463306637643137646336363634363338
...
Paste the resulting block into the inventory file:
spec:
config:
registries:
- name: mirror.example.com
auth:
user: myuser
pass: |
$ANSIBLE_VAULT;1.1;AES256
62306432326630316632646366363136303161316635343463306637643137646336363634363338
...
Applying with the Vault Password
cargoship apply needs the same vault password to decrypt the credential at run time, supplied the same way as encryption – via --vault-password-file, or the CARGOSHIP_VAULT_PASSWORD environment variable if the flag is omitted:
cargoship apply --vault-password-file ./vault-pass.txt cluster.tar.zst
If a registry has a vault-encrypted credential and neither the flag nor the environment variable resolves a password, apply fails with an error naming the registry and the field – auth.user, auth.pass, auth.token, or tls.ca – rather than silently treating the ciphertext as a literal username or password. So does a password that does not match the document, or a tls.ca that decrypts to something that is not a certificate. All of it is checked as soon as the password is resolved, before the first host is connected to, so a wrong password costs you an error message rather than a half-configured cluster.
Removing a Host from the Cluster
This guide explains what happens when you delete a host from your Cargoship config, why an apply stops rather than reconciling the removal, and how to actually remove the machine.
What an apply does with a removed host
Nothing, and that is deliberate. Every phase after the fact-gathering ones reads .spec.hosts and acts on what it finds there, so a host you delete from the config becomes invisible to all of them. The machine keeps running the engine and stays joined to the cluster while the apply that was supposed to remove it reports success.
An apply catches this instead of walking past it. Right after it gathers facts, and before it changes anything, it lists the nodes joined to the cluster and compares them to the hosts in the config. If the cluster holds a node no host accounts for, the apply stops:
the cluster holds nodes the config does not: add the host back to the config,
run `cargoship install reset` against it, or pass --allow-unmanaged-nodes to
apply anyway: worker3 (worker)
Each node is named with its role, because a leftover controller is the more dangerous case. The check is read-only, so --dry-run runs it too, which is where you want to find out about a removal you did not intend.
A node is matched to a host by any identifier they share: the node’s name or any address it reports, against the host’s hostname, private address, or connection address. That is deliberately generous, since a false positive would stop an apply on a working cluster – it means a node whose name you overrode with node-name in the engine config still matches its host through its address. If the cluster cannot be reached at all, the check warns and the apply continues, because failing to reach it is not evidence that a host was removed. The node list is read with kubectl on a controller over the existing SSH connection, so the check works on a cluster whose firewall admits nothing but SSH from outside.
Why apply refuses instead of reconciling
Removing a node properly is two jobs, and an apply can only do one of them.
Draining a node and deleting its Node object needs the node’s name, which the API server will happily give us. Uninstalling the engine – stopping the service, removing the packages, deleting the data directory and certificates – needs an SSH connection to the machine, and the address, user and key for that lived in the hosts entry you just deleted. An apply has no memory of previous runs, so it has no way to reach a machine the config no longer describes.
Doing only the first half would be worse than doing nothing, because it looks like the removal worked. The machine keeps its certificates and keeps trying to rejoin, and if it was a controller it stays an etcd member: the cluster looks smaller while quorum is still counted against a member nobody can reach.
So an apply reports the gap and leaves the cluster alone.
Removing a host in OpenTofu
In OpenTofu, mark the host as state = "absent" in your cargoship_cluster resource:
resource "cargoship_cluster" "prod" {
# ...
hosts = {
controller1 = {
address = "10.0.0.11"
role = "controller"
key_path = "~/.ssh/id_ed25519"
profile = "control"
}
# worker3 is marked for removal:
worker3 = {
address = "10.0.0.31"
role = "worker"
key_path = "~/.ssh/id_ed25519"
state = "absent"
}
}
}
One tofu apply does the whole removal:
- The node is drained and deleted from the cluster, through a controller that is staying. The run refuses rather than guessing if every running controller is itself marked absent – removing all of them is a teardown, and
tofu destroyis what does that. - Cargoship connects to
10.0.0.31over SSH, stops the engine and uninstalls it. - Every other host is left alone, and the apply then converges them as usual.
- The host’s
removedattribute becomestrue.
Leave the entry where it is. removed is what tells later applies the work is done, so they neither repeat the teardown nor try to reach a machine that may since have been decommissioned. Delete the entry whenever it suits; nothing depends on it going, and deleting it changes nothing on any host.
If the removal fails, the apply stops there and says so. The entry stays in state with removed still false, which is deliberate: the machine is still running and still joined, and that entry holds the only address and key path that can reach it. Fix the cause and apply again. The rest of the convergence does not run, because converging on top of a half-removed cluster would report success over a node that is still in it.
What a destroy does with the markers
Nothing, beyond reading them. tofu destroy tears down everything the cluster still holds: a host marked absent whose removal failed is included, because the machine is still running and still joined, and one whose removal finished is left out, because reaching for a decommissioned machine would fail. The markers are not something to clean up before destroying.
If the machines are OpenTofu resources too and are being destroyed in the same run, consider retain_on_destroy = true – draining and uninstalling a node that is about to be deleted is minutes spent on a result nobody sees. See the provider’s Terragrunt guide.
Removing the machine by hand (CLI)
With the CLI, use targeted reset with --target-hosts:
$ cargoship reset --config ./cargoship-config.yaml --distro rke2 --target-hosts worker3 --confirm
Cargoship automatically selects a surviving running controller as the leader to delete worker3 from the cluster, then uninstalls the engine only on worker3, leaving your surviving controllers and workers untouched. After the targeted reset finishes, delete the host block from cargoship-config.yaml.
Removing a controller is the same command with more rules attached, and a run that targets two of them is refused rather than left to etcd. See Removing a controller.
Removing a controller
A worker leaves the cluster by being drained, deleted and uninstalled. A controller does that and also gives up an etcd member, so the order matters and the run is more constrained.
Cargoship stops the engine on the target before deleting its node. That is what makes the deletion stick: k3s and RKE2 remove the etcd member when the Node object is deleted, but only once the engine behind it has stopped, and an engine that is still running re-registers the node seconds later. The one exception is a removal that leaves a single controller, where the member being removed has to still be up to form the majority of two that commits its own removal – there the engine stays running through the deletion and is uninstalled immediately after.
One controller per run. A run that keeps the cluster and targets two running controllers is refused. Each removal is a raft reconfiguration that needs a majority of the membership it starts from; the second one in a run would start from three members with one already stopped, and taking the next one down leaves one of two, which hangs with the cluster unavailable rather than failing cleanly. Removing them in separate runs is safe, because the cluster is whole again at the start of each.
Removing the first controller
This section is about k3s and rke2. On upstream there is nothing to do: kubeadm is given controlPlaneEndpoint: <load_balancer> at init and every host – controllers included – joins through apiServerEndpoint: <load_balancer>, so no host holds the first controller’s name. The removed controller lingers in the API server’s certSANs until the next apply re-renders them, which changes nothing.
On k3s and RKE2 there is nothing special about the first controller at runtime either – it is the first in the host list, re-elected on every run, and it holds no state the others do not. Three things point at it by name, and only one of them needs attention:
| Points at | Removing the first controller |
|---|---|
| The kubeconfig Cargoship writes | Unaffected. Its server is https://<load_balancer>:6443, never a node address, so admin access survives any single controller |
| Workers | Unaffected. They join through https://<load_balancer>:9345 for the same reason |
The other controllers’ config.yaml | server: names the first controller, written when they joined. A running controller never reads it – it is the join address, used at startup – so the cluster keeps working, and the next restart is what would fail |
So the step that makes it safe is re-rendering config.yaml on the controllers that stayed, which cargoship apply does: it re-elects the first controller from the remaining hosts and writes the new server: to every one of them. The engines are not restarted for it, which is the point – the file is correct again before anything reads it.
In OpenTofu this is already one operation. tofu apply removes the host and then converges what is left, in that order, in the same run.
With the CLI, the targeted reset and the apply are two commands, and the apply is not optional:
$ cargoship reset --config ./cargoship-config.yaml --distro rke2 --target-hosts controller1 --confirm
$ cargoship apply ./package.tar.zst --config ./cargoship-config.yaml --confirm
Between them the surviving controllers hold a join address that no longer resolves to a machine. They keep serving, and a reboot in that window is what turns it into an outage.
Two things Cargoship does not do for you:
- The load balancer.
load_balanceris a name you own – a VIP, a DNS record, an HAProxy backend list. If it still routes to the controller you removed, a share of every API call fails. Update it in whatever manages it. - A quorum check against live etcd. The refusals above count controllers from the configuration and from whether their engine answers, not from etcd’s own view of the membership. Removing a controller from a cluster that is already one member down is a removal from a membership that cannot commit it, and nothing here will say so. Confirm the cluster is healthy first.
When the extra nodes are deliberate
A cluster can hold nodes Cargoship never joined, and refusing on those would block every apply on a working setup. Pass --allow-unmanaged-nodes to downgrade the refusal to a warning:
$ cargoship apply ./package.tar.zst --config ./cargoship-config.yaml --allow-unmanaged-nodes --confirm
It is also available as distro.allow_unmanaged_nodes in the Cargoship config file, and as DISTRO_DISTRO_ALLOW_UNMANAGED_NODES in the environment. Cargoship still leaves those nodes alone either way; the flag only chooses whether their presence is an error or a log line.
Inventory Configuration Guide
This guide explains how to author a cluster inventory file for bootstrapping or upgrading clusters with Cargoship.
Basic Structure
A basic inventory file defines the cluster’s metadata, global configuration, profiles, and individual target hosts. Start with the following template:
---
# yaml-language-server: $schema=https://raw.githubusercontent.com/colonel-byte/cargoship/refs/heads/main/schema/cluster-v1alpha1-schema.json
kind: Cluster
metadata:
name: bubbles
The .metadata.name field sets the cluster name. Cargoship uses this to configure the context name in the resulting kubeconfig (e.g., bubbles).
Editor Support
The yaml-language-server comment on the first line is what makes an editor complete and check the file as you write it. The URL above reads the schema from GitHub’s default branch, which is convenient and wrong in two situations: an air-gapped machine cannot fetch it at all, and yaml-language-server says nothing when a fetch fails, so the file silently stops being checked; and a machine that can fetch it is checking against unreleased main rather than the release it is running.
cargoship schema writes the schema out of the binary, so it is the schema that build validates against and it needs no network:
cargoship schema inventory -o ./cluster-v1alpha1-schema.json
Then point the inventory at the local copy:
---
# yaml-language-server: $schema=./cluster-v1alpha1-schema.json
kind: Cluster
metadata:
name: bubbles
Package Values
The .spec.config.values section overrides the values the distro package was built with. It is the one part of an inventory whose vocabulary belongs to the package rather than to cargoship, so the schema above leaves it untyped – and a hosted schema never could do better:
spec:
config:
values:
addons:
disabled:
- rke2-ingress-nginx
Pass --package to compose the package’s own values schema into the inventory schema, and the block completes and checks like the rest of the file:
cargoship schema inventory --package ./package.tar.zst -o ./inventory.schema.json
--package takes anything cargoship apply takes – a tarball, an oci:// or https:// reference – or a package source directory, so it works before the package is built. The composed schema describes the overrides on their own, while cargoship validates them merged with the values the package ships: it catches an unknown key, a wrong type, or a name outside an enum, and does not reproduce install-time validation. See the package values guide for what a package may expose and how the merge works.
Checking an Inventory
An editor is not the only place this check belongs. Cargoship unmarshals an inventory without strict key checking, so a misspelled key is dropped rather than reported: a file that says loadbalancr: parses cleanly and installs a cluster with no load balancer address. cargoship validate runs the same schema over a file with no editor and no network:
cargoship validate ./inventory.yaml
It reports every problem rather than the first, so one pass is enough to fix a file:
inventory.yaml does not match the inventory schema:
spec.config: Additional property loadbalancr is not allowed
spec.config: loadbalancer is required
spec.hosts.0.role: must be one of the following: "controller", "worker"
The schema is chosen from the document’s own kind, so a directory of inventories and package definitions can be checked in one run. A cargoship config file declares no kind and has to be named:
cargoship validate ./inventories/*.yaml
cargoship validate --kind config ./cargoship-config.yaml
--package works here too, and checks .spec.config.values against the package’s own schema with the same caveat as above:
cargoship validate ./inventory.yaml --package ./package.tar.zst
This is a check you run, not one cargoship runs for you. apply does not validate the inventory against the schema before installing; it reads the file the way it always has. Put cargoship validate in CI, or in front of an install, where a typo is cheap to fix.
Global Configuration
The .spec.config section configures cluster-wide settings, such as load balancer endpoints:
spec:
config:
loadbalancer: bubbles-kc.test.com
The loadbalancer address is required. Cargoship adds this to the TLS Subject Alternative Names (SANs) for the Kubernetes API server. This can be a Round-Robin DNS record pointing to your control-plane nodes, or an external load balancer like an AWS NLB.
Node Profiles
The optional .spec.config.profiles section defines reusable groups of node configurations (e.g., control, infra, or worker). Profiles simplify management by applying standard node labels, taints, or host firewall rules across matching hosts.
spec:
config:
profiles:
control:
host:
ports:
- port: 6443
protocol: tcp
firewall:
rules:
- name: allow-metrics
action: allow
source: 10.0.0.0/8
port: "9100"
protocol: tcp
engine:
labels:
adrp.xyz/purpose-control: "true"
taints:
- CriticalOnly=True:NoExecute
Target Hosts
The .spec.hosts array is the main body of the inventory. It defines each host node, its connection details, role, and profile:
spec:
hosts:
- hostname: distro-kc01
ssh:
address: 10.1.2.3
user: root
port: 22
keyPath: ~/.ssh/id_ed25519
profile: control
role: controller
Host Firewall
The optional .host.firewall section, on a host or on a profile, declares firewall rules cargoship applies to the node. Cargoship detects whether the node runs firewalld, ufw, or nftables directly, and renders the same rules onto any of them. A host’s own rules union with its profile’s, so the host below gets allow-metrics from the control profile as well as its own allow-backup. See the firewall guide for the rule model and the per-backend details.
spec:
hosts:
- hostname: distro-kc01
role: controller
profile: control
host:
firewall:
rules:
- name: allow-backup
action: allow
source: 10.0.9.4
port: "2049"
protocol: tcp
Signing Packages
This guide explains how Cargoship signs and verifies packages. It covers key pair generation, the three points where you can sign, and verification.
A Cosign key pair is the only signing method. Keyless (Fulcio/OIDC) signing and verification, and cloud KMS key references, were removed: all of them need to reach a cloud endpoint, and the management node Cargoship runs from is air-gapped. If you need keyless back, say so on issue #848.
What Gets Signed
A Cargoship package is a .tar.zst archive. It contains a distro.yaml manifest, a checksums.txt file, and the image and file layers.
Cargoship signs distro.yaml only. The signing step sets build.signed: true in the manifest. It then signs the manifest bytes with Cosign and writes the signature beside the manifest as distro.bundle.sig, a Sigstore bundle.
One signature covers the whole package. distro.yaml records metadata.aggregateChecksum, the checksum of checksums.txt. checksums.txt lists a checksum for every layer in the package. Signing does not modify the checksums.
Signing is optional. Unsigned packages build, publish, pull, and apply normally. Verification then has nothing to check.
Generating a Cosign Key Pair
Cargoship does not generate keys. It uses keys in Cosign’s format. Use the cosign CLI:
cosign generate-key-pair
The command prompts for a password. It writes two files into the current directory:
cosign.key– the encrypted private key. Keep this file secret. Treat it like any other signing credential.cosign.pub– the public key. Give this file to everyone who verifies your packages.
To skip the prompt in CI, set the password in the environment first:
COSIGN_PASSWORD='...' cosign generate-key-pair
Restrict the private key file to the signing user:
chmod 600 cosign.key
Keeping the Key Off Disk
--signing-key accepts a local file path, or env://NAME to read the PEM out of an environment variable. The environment form keeps the private key off the build host’s filesystem, which is what a CI pipeline wants.
export COSIGN_PRIVATE_KEY="$(cat cosign.key)"
export COSIGN_PASSWORD=...
cargoship sign ./my-package.tar.zst --signing-key env://COSIGN_PRIVATE_KEY
--key takes the same two forms when verifying, so a public key can come from env://COSIGN_PUBLIC_KEY as well as from a file.
Cloud KMS references – awskms://, gcpkms://, azurekms://, hashivault://, openbao:// – are not supported. Each resolves a key by reaching a cloud endpoint, which an air-gapped management node cannot do. Passing one fails naming the scheme rather than reporting a missing file.
Signing a Package
You can sign a package at three points. Each point accepts the same --signing-key and --signing-key-pass flags.
At Create Time
cargoship create ./distro-defs --signing-key ./cosign.key
Cargoship prompts for the key password when you omit --signing-key-pass. Use --confirm to skip the prompt in a non-interactive environment. The password must then come from --signing-key-pass, the config file, or the environment.
cargoship create ./distro-defs --signing-key ./cosign.key --signing-key-pass "$COSIGN_PASSWORD" --confirm
At Publish Time
publish signs an unsigned package as it pushes to a registry. It also re-signs a signed package with a different key.
cargoship publish ./build/cargoship-rancher-rke2-amd64-1.0.0.tar.zst oci://ghcr.io/my-org \
--signing-key ./cosign.key --confirm
Use this form for a two-stage pipeline. One job builds the package. A second job holds the key material and signs.
As a Separate Step
cargoship sign signs an existing package without a rebuild. The source is a local tarball or an OCI reference.
cargoship sign cargoship-rancher-rke2-amd64-1.0.0.tar.zst --signing-key ./cosign.key
--output sets the destination of the signed package. Cargoship applies a default when you omit the flag. For a local source, the default is the source directory. For an OCI source, the default is the repository part of the source reference. The output can also be an OCI URL. Cargoship then signs and publishes in one step.
cargoship sign oci://ghcr.io/my-org/my-package:1.0.0 --signing-key ./cosign.key --output ./signed/
cargoship sign ./my-package.tar.zst --signing-key ./cosign.key --output oci://ghcr.io/my-org/signed-packages
Re-signing
Cargoship refuses to sign a package that already has a signature. It reports package is already signed, use --overwrite to re-sign. Use --overwrite to replace the existing signature.
cargoship sign ./my-package.tar.zst --signing-key ./new-cosign.key --overwrite
Add --verify=always --key ./old-cosign.pub to check the existing signature against the old key first. Use this form for key rotation. It proves the package is the one you expect before the new signature replaces the old one.
Verifying a Package
Every command that loads a package accepts --verify: sign, publish, pull, apply, prepare, and engine-config-sync. The flag takes one of three modes.
| Mode | Behavior |
|---|---|
never | Cargoship skips signature verification. |
if-possible | Default. Cargoship verifies when the package has a signature and you supply verification material. It tolerates a package with nothing to verify against. Every other failure is fatal, including a tampered signature or a wrong key. |
always | Cargoship requires a successful verification. It fails when the package is unsigned. It also fails when you supply no verification material. |
Key-Based Verification
Pass the public key with --key or -k.
cargoship pull oci://ghcr.io/my-org/my-package:1.0.0 --verify=always --key ./cosign.pub
Verification on the Install Path
apply, prepare, and engine-config-sync verify the package before they touch any host. They accept the same flags as pull, and they read the same config keys.
cargoship apply ./cargoship-distro-amd64.tar.zst --config ./cluster.yaml --confirm \
--verify=always --key /etc/cargoship/cosign.pub
Set verify: always and public_key in the config file to apply this policy to every install on the host. See Configuration File and Environment Variables.
Verification runs while Cargoship loads the package. A failure stops the command before it configures a node.
Configuration File and Environment Variables
Set signing and verification options in cargoship-config.yaml to avoid repeating them on every command. The distro.verify and distro.public_key keys apply to every command that loads a package, including apply.
distro:
# verification defaults, applied to any command that loads a package
verify: always
public_key: /etc/cargoship/cosign.pub
publish:
signing_key: /etc/cargoship/cosign.key
signing_key_password: hunter2
Every key also reads from the environment. Use the DISTRO_ prefix and the full config path, with each . replaced by _.
export DISTRO_DISTRO_PUBLISH_SIGNING_KEY=./cosign.key
export DISTRO_DISTRO_PUBLISH_SIGNING_KEY_PASSWORD="$COSIGN_PASSWORD"
export DISTRO_DISTRO_PUBLIC_KEY=./cosign.pub
Note the doubled prefix. DISTRO_ is the environment prefix. The second DISTRO is the distro section of the config file.
Command-line flags take precedence over the config file and the environment.
signing_key_password in the config file puts a plaintext secret on disk. Use the environment variable instead, and env:// for the key itself, so neither the key nor its password is written to the build host.
Recommended CI Pattern
Use this pattern for a pipeline that builds and signs with a stored key.
env:
COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}
COSIGN_PRIVATE_KEY: ${{ secrets.COSIGN_PRIVATE_KEY }}
# ...
- run: |
cargoship create ./distro-defs -o ./build \
--signing-key env://COSIGN_PRIVATE_KEY \
--signing-key-pass "$COSIGN_PASSWORD" \
--confirm
cargoship publish ./build/cargoship-*.tar.zst oci://ghcr.io/my-org \
--verify=always --key ./cosign.pub
--verify=always on the publish step matters. The step fails instead of pushing a package whose signature does not validate against the public key you distribute.
Vault Encryption
This guide explains how Cargoship encrypts registry credentials in a cluster configuration with Ansible Vault, and how to read them back out. It covers the four fields Cargoship decrypts, the cargoship vault commands, and the password-rotation workflow.
Ansible Vault is one of two formats Cargoship reads. The other is age, which encrypts to a set of public keys instead of a shared password – see age-encryption. The same commands write both, one configuration can hold both, and vault rekey moves a configuration from one to the other. Everything below describes the Ansible Vault side.
What Cargoship Decrypts
A cluster configuration holds registry credentials in plain sight. Encrypting them lets you commit the file.
Cargoship decrypts exactly four fields per registry, and only these four:
| Path | What it holds |
|---|---|
.spec.config.registries[N].auth.user | Registry username |
.spec.config.registries[N].auth.pass | Registry password |
.spec.config.registries[N].auth.token | Registry token |
.spec.config.registries[N].tls.ca | Inline CA certificate |
Decryption happens during an apply, when Cargoship writes the engine’s registry configuration. A field that does not carry the $ANSIBLE_VAULT header is passed through untouched, so a configuration can mix encrypted and plaintext values freely.
A CA certificate is public and does not need encrypting. Cargoship accepts an encrypted one anyway, so that a document can be vaulted as a whole without the parts that did not have to be causing a failure.
Encrypting any other field produces a document Cargoship can load but never unwraps: the ciphertext reaches the host verbatim, as if that were the literal value. vault encrypt-path warns when you ask for such a path, but does not refuse - see Encrypting a Value Already in a File.
Supplying the Password
Every vault command, and every command that applies a configuration, resolves the password the same way and stops at the first that is set:
--vault-password-file, a path to a file holding the password.- The
CARGOSHIP_VAULT_PASSWORDenvironment variable. - The
ANSIBLE_VAULT_PASSWORDenvironment variable.
A trailing newline in a password file is stripped, so a file written by echo works as expected.
Cargoship never prompts for the vault password. If none of the three is set, the command fails rather than treating the password as empty.
Restrict the password file to the user that runs Cargoship:
chmod 600 vault-pass.txt
The Commands
cargoship vault encrypt [VALUE]
cargoship vault encrypt-path FILE YAML_PATH [YAML_PATH...]
cargoship vault encrypt-file FILE
cargoship vault decrypt [VALUE]
cargoship vault decrypt-path FILE YAML_PATH [YAML_PATH...]
cargoship vault decrypt-file FILE
cargoship vault rekey FILE
The encrypt and decrypt pair work on a value and print to stdout. The -path pair work on named values inside a configuration file and rewrite the file in place. The -file pair work on every registry credential in a configuration at once, and are what you want most of the time. rekey moves a whole configuration from one vault password to another, or re-salts it under the one it has – see Rotating the Vault Password.
YAML_PATH is a path into the document, not a path on the filesystem: .spec.config.registries[0].auth.pass names a value inside FILE, the way jq or yq would address it. The leading $ that go-yaml uses is optional, so $.spec..., .spec..., and spec... all name the same value. Quote it in your shell – it contains [ and ].
The -path commands take as many of them as you like after FILE. The file is written once, after every value has been rewritten, so a path that is missing or in the wrong state fails the run without leaving the file half-done. Naming the same value twice – including as two different spellings of one path – is an error rather than a second pass over it.
Encrypting a Value
vault encrypt takes the value as an argument, on stdin, or at a hidden prompt when stdin is a terminal:
cargoship vault encrypt my-registry-password --vault-password-file ./vault-pass.txt
printf my-registry-password | cargoship vault encrypt --vault-password-file ./vault-pass.txt
cargoship vault encrypt --vault-password-file ./vault-pass.txt < ./registry-token.txt
Passing a secret as an argument leaves it in your shell history and in the process list. Prefer stdin or the prompt on a shared machine.
The output is a multi-line $ANSIBLE_VAULT block that you then paste into the configuration yourself, at the right indentation. vault encrypt-path does that part for you.
Encrypting a Value Already in a File
vault encrypt-path encrypts the plaintext the file already holds at a YAML path, and writes it back as a block scalar:
cargoship vault encrypt-path ./cluster.yaml '.spec.config.registries[0].auth.pass' --vault-password-file ./vault-pass.txt
Name several paths to encrypt them in one pass, which is the way to vault values encrypt-file does not cover – anything outside a registry’s credentials, such as an x-tra block of your own:
cargoship vault encrypt-path ./cluster.yaml '.x-tra.test.user' '.x-tra.test.pass' --vault-password-file ./vault-pass.txt
Given this configuration:
registries:
- name: harbor # our mirror
auth:
user: admin
pass: hunter2 # rotate me
the command rewrites only that one value:
registries:
- name: harbor # our mirror
auth:
user: admin
pass: |- # rotate me
$ANSIBLE_VAULT;1.1;AES256
31333438393936323663373461353533373034663339316335386139363535383531643464613539
6235646237326332663931643964633165323934646363390a353136626562643632326566353765
3961
Everything else is preserved byte for byte: comments, key order, quoting elsewhere in the document, and the trailing comment on the key’s own line. The file stays yours rather than becoming a re-rendering of itself.
Two flags change what it does:
--dry-runprints the resulting document to stdout and leaves the file untouched.--forceencrypts a value that is$ANSIBLE_VAULTalready, wrapping it a second time. Without it, an already-encrypted value is an error, so that re-running the command cannot bury the plaintext under a layer nothing unwraps.
If a YAML_PATH is not one of the four fields in What Cargoship Decrypts, the command warns and proceeds:
WRN cargoship does not decrypt this field at apply time, so the value will reach the host as ciphertext
Decrypting a Value
vault decrypt is the inverse of vault encrypt. It takes the ciphertext as an argument or on stdin:
cargoship vault decrypt --vault-password-file ./vault-pass.txt < ./encrypted-token.txt
The plaintext is written to stdout as it is, with a trailing newline added only when it does not already end in one. A multi-line value such as a PEM certificate can therefore be redirected straight into a file:
cargoship vault decrypt --vault-password-file ./vault-pass.txt < ./encrypted-ca.txt > ./ca.pem
Unlike vault encrypt, there is no hidden prompt. Ciphertext is neither secret nor a single line, so a terminal with nothing piped into it is reported as a mistake.
Decrypting a Value Already in a File
vault decrypt-path puts the plaintext back into the document, in place of the ciphertext:
cargoship vault decrypt-path ./cluster.yaml '.spec.config.registries[0].auth.pass' --vault-password-file ./vault-pass.txt
It takes several paths too, and like encrypt-path writes the file once at the end:
cargoship vault decrypt-path ./cluster.yaml '.x-tra.test.user' '.x-tra.test.pass' --vault-password-file ./vault-pass.txt
Encrypting a value and decrypting it again gives back the file that went in, byte for byte.
The plaintext is written in whichever scalar style holds it faithfully: plain where YAML allows it, a literal block for something multi-line such as a PEM certificate, and a quoted string for anything the other two would change on the way back in – a value with a leading space, a trailing space, a #, or one that would otherwise read back as a number or a boolean.
The command warns that it has just removed the protection the file relied on:
WRN the file now holds this value in plaintext
--dry-run prints the resulting document to stdout and leaves the file untouched, which is the way to check that a value decrypts without writing the plaintext to disk at all:
cargoship vault decrypt-path ./cluster.yaml '.spec.config.registries[0].auth.pass' --vault-password-file ./vault-pass.txt --dry-run
There is no --force. A value that is not $ANSIBLE_VAULT has nothing to decrypt, and rewriting it would be a no-op that still rewrote the file, so that is an error.
Encrypting a Whole Configuration
Naming each path gets tedious once a configuration has more than one registry, and it is easy to miss one. encrypt-file walks every registry and encrypts the four fields in the table above:
cargoship vault encrypt-file ./cluster.yaml --vault-password-file ./vault-pass.txt
It reports each path it rewrote and a count at the end. Everything else in the document is left exactly as it was, the same way encrypt-path leaves it.
A field that is absent, empty, or encrypted already is skipped rather than treated as an error. Two things follow from that, and both are the point of the command:
- Running it over a configuration that was vaulted by hand, or half-vaulted and then abandoned, finishes the job without disturbing what is already there.
- Running it twice changes nothing the second time, so it is safe in a script or a pre-commit hook.
This is what makes the ordinary credential rotation easy. Paste the new registry password into the file in the clear, leave the vaulted username alone, and run encrypt-file: it picks up the one value you changed and does not re-wrap the rest. Because Ansible Vault salts every encryption, re-wrapping them would change every one of those lines in the diff for no reason.
A credential encrypted with age is skipped too, since this command never moves a value between the two formats. That skip is reported rather than passed over in silence:
WRN left this credential as it was: it is age-encrypted, and encrypt-file does not move a
credential between formats; run 'cargoship vault rekey' with an age identity and the new
vault password to move it onto Ansible Vault file=./cluster.yaml path=$.spec.config.registries[0].auth.pass
--dry-run prints the resulting document to stdout and leaves the file alone; warnings go to stderr, so the document it prints stays clean. --force re-encrypts values that are ciphertext already, wrapping them a second time – which is almost never what you want, since an apply unwraps only one layer.
Decrypting a Whole Configuration
cargoship vault decrypt-file ./cluster.yaml --vault-password-file ./vault-pass.txt
The inverse: every vaulted registry credential in the file becomes plaintext again, and anything that is not encrypted is skipped. An encrypt-file followed by a decrypt-file under the same password gives back the original document byte for byte.
If any value fails to decrypt, the command fails and the file is not touched at all – it does not leave you with half a configuration in the clear.
Shared Credentials (Anchors and Aliases)
A configuration that pulls from two names on the same host usually writes the credential once and points the second registry at it:
registries:
- name: docker.io
auth: &auth
user: robot
pass: hunter2
- name: test.io
auth: *auth
All of the vault commands follow anchors and aliases. The value lives under the anchor, so that is what gets rewritten, and the alias is left as it is:
- name: docker.io
auth: &auth
user: |-
$ANSIBLE_VAULT;1.1;AES256
...
- name: test.io
auth: *auth
Both registries read the same ciphertext at apply time, which is what sharing the value meant in the first place. encrypt-file reports it once – count=2 for the shared user and pass above, not four – because there is one value behind each pair of paths. Naming the aliased registry yourself works too: encrypt-path ... '.spec.config.registries[1].auth.pass' rewrites the anchored value, since that is the value that path resolves to. Naming both registries in one encrypt-path run is the one thing to avoid; the second path is the value the first has already encrypted, so it fails as already encrypted.
An alias whose anchor is missing, or comes later in the file, is an error naming the alias. Nothing in this package reports a value it cannot reach as a value that was not there.
Merge keys work the same way. A registry that merges a shared block reads whatever that block holds, encrypted or not, and the keys it writes beside the merge are its own values, encrypted separately:
- name: lab.test.local
auth:
user: lab-robot
pass: correct-horse
tls:
<<: *internal-tls
insecureSkipVerify: true
A merge can add keys to a block but never override one the anchor already sets – YAML forbids the same key twice in a mapping, and the configuration fails to load before cargoship sees it – so put in a shared block only what every user of it wants, and write what differs beside the merge.
test/e2e/noncluster/testdata/inventory-anchors.yaml is a complete inventory built this way, kept as a worked example and exercised by the end-to-end suite.
Rotating the Vault Password
cargoship vault rekey ./cluster.yaml --vault-password-file ./old-pass.txt --new-vault-password-file ./new-pass.txt
Every vaulted registry credential in the file moves to the new password in one command. The plaintext is never written to the file: each value is decrypted and encrypted again in memory, so there is no window in which the configuration on disk is readable, and nothing to remember to clean up if the command fails or the terminal goes away mid-rotation. --dry-run prints the result to stdout and leaves the file alone.
--new-vault-password-file is optional. Omit it and every value is re-wrapped under the password the file already carries:
cargoship vault rekey ./cluster.yaml --vault-password-file ./vault-pass.txt
That is a re-salt rather than a rotation, and it is logged as one. Ansible Vault salts each encryption, so every credential comes back as different ciphertext holding the same plaintext under the same password – useful when the ciphertext has been somewhere you would rather it had not been, or when a value was vaulted long enough ago that you want it rewritten, and it never puts the plaintext on disk. Naming a file that holds the old password does the same thing.
Unlike --vault-password-file, --new-vault-password-file has no environment fallback. The environment holds the password the file is vaulted under now, so falling back to it would report a rotation onto a password nobody asked to move to.
A value that is plaintext is skipped, and so is one that is empty or absent – rekeying does not encrypt anything that was not encrypted before. To pick up a newly added credential, run encrypt-file with the new password afterwards.
Rotate the whole file, not one value at a time. An apply decrypts all of a registry’s fields with a single password, so a configuration whose values were vaulted under two different passwords is one that no password can read. rekey refuses to create or perpetuate that state: if any encrypted value in the file cannot be read with the old password, it stops before writing anything and names the path. encrypt-file refuses the same way. But rotating path by path with decrypt-path and encrypt-path will walk you into it, since those commands only ever look at the one value you named.
Ansible Vault salts each encryption, so the ciphertext differs on every run even when the password and the plaintext are unchanged. A rekey produces a diff on every value it touches, and running it twice with the same pair of different passwords is not idempotent – the second run fails, because the file is no longer readable with the old password. A re-salt can be run as often as you like, since the password it reads with is the password it writes back.
Checking Before an Apply
An apply verifies every vaulted value as soon as it has resolved the password – before it connects to any host, and on a sync, before it starts draining nodes. A missing or wrong password fails the command immediately rather than part-way through a cluster change.
To check a configuration by hand without writing plaintext anywhere, use decrypt-file --dry-run and throw the output away. It decrypts every vaulted credential in the file and exits non-zero if any of them fails:
cargoship vault decrypt-file ./cluster.yaml --vault-password-file ./vault-pass.txt --dry-run > /dev/null
Interoperability with ansible-vault
Cargoship writes the Ansible Vault 1.1 AES256 format, so ansible-vault reads what Cargoship writes and Cargoship reads what ansible-vault writes:
cargoship vault encrypt hunter2 --vault-password-file ./vault-pass.txt > ct.txt
ansible-vault decrypt --vault-password-file ./vault-pass.txt --output - < ct.txt
One difference matters. ansible-vault encrypt_string tags its output with !vault:
pass: !vault |
$ANSIBLE_VAULT;1.1;AES256
...
Cargoship writes a plain block scalar with no tag, and its -path and -file commands reject a tagged value rather than strand the tag on a value that no longer matches it. Drop the !vault tag when moving a value across; the ciphertext underneath needs no change.
Notes
cargoship vault-encryptis the old spelling ofcargoship vault encrypt. It still works and still prints a deprecation notice. Use thevaultgroup instead.- A failed command never touches the file. The rewrite goes to a temporary file alongside the original and is renamed over it, so an interruption cannot leave a half-written configuration.
- The file’s mode is preserved across a rewrite.
cargoship
cargoship
CLI for cargoship installs
cargoship COMMAND [flags]
Options
-h, --help help for cargoship
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship apply - Apply a config file to bootstrap and upgrade a cluster
- cargoship create - Creates a Cargoship Package from a given directory or the current directory
- cargoship engine-config-sync - Sync engine config (registries, audit, and pod security) to a cluster, draining and restarting the engine service on any node whose config has drifted
- cargoship import - Converts an OCI Image Layout archive into a native Cargoship package
- cargoship inventory - Generates a cluster inventory from another source of truth
- cargoship kube-config - Get the admin kube-config for a control-plane node
- cargoship prepare - Prepares the nodes, including restarting the node if new kernel modules are enabled
- cargoship publish - Publish the Cargoship Package to an OCI registry
- cargoship pull - Pulls a Cargoship package from a remote registry and saves it to the local filesystem
- cargoship redact-receipt - Rewrites a run receipt so it can be attached to an issue
- cargoship reset - Reset a cluster, stopping, uninstalling, and removing all data for an engine
- cargoship schema - Writes out a JSON Schema for one of cargoship’s own file formats
- cargoship sha256sum - Generates a SHA256SUM for the given file
- cargoship sign - Signs an existing Cargoship distro package
- cargoship validate - Checks a cluster inventory, package definition, or config file against its schema
- cargoship vault - Encrypts and decrypts cluster configuration values with Ansible Vault or age
- cargoship verify - Verifies the signature on a Cargoship distro package
- cargoship version - Shows the version of the running binary
apply
cargoship apply
Apply a config file to bootstrap and upgrade a cluster
Synopsis
Bootstraps a cluster from a package and a cluster configuration, or upgrades one that is already running, by stepping through the apply phases against every host the configuration names. One command does both: the engine version already on each host is what tells an upgrade apart from an install, and a downgrade is refused rather than attempted, before any phase has written to a host. Pass –allow-downgrade when the move backwards is deliberate.
Cargoship opens every SSH connection itself, from the machine it runs on, and nothing is installed on a target host beyond what a phase uploads.
An apply never removes a node. A host deleted from the configuration leaves its node in the cluster and stops the run rather than having the difference reconciled, so that no machine is drained or uninstalled by a configuration edit alone. Pass –allow-unmanaged-nodes when the extra nodes were joined deliberately and cargoship should leave them alone.
This changes every host it is pointed at, so it needs –confirm. Pass –dry-run instead to connect to every host and run the preflight checks for real, reporting what the run would change without changing it.
cargoship apply [Distro Package] [flags]
Examples
# Bootstrap or upgrade a cluster from a package and a config file
$ cargoship apply ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm
# Decrypt vault-encrypted registry credentials with a password file
$ cargoship apply ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm --vault-password-file ./vault-pass.txt
# Upgrade workers 25% at a time instead of all at once
$ cargoship apply ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm --work-concurrency 25%
# Update /etc/hosts, the firewall, and fapolicyd on every node as part of the apply
$ cargoship apply ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm -H -F -f
# Add the node-role label to each node, and leave the local kubeconfig untouched
$ cargoship apply ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm --label-nodes --update-kubeconfig=false
Options
--age-identity-file stringArray Path to an age identity file holding the private keys that decrypt registry credentials, or to an SSH private key such as ~/.ssh/id_ed25519. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.identity_files in the cargoship config file, or as a single path in CARGOSHIP_AGE_IDENTITY_FILE.
--age-recipient stringArray A public key to encrypt registry credentials to: either an age recipient such as age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p, or an SSH public key such as 'ssh-ed25519 AAAAC3Nza...'. Repeatable; also settable as age.recipients in the cargoship config file, or space-separated in CARGOSHIP_AGE_RECIPIENTS. Giving any recipient makes cargoship write age ciphertext instead of Ansible Vault.
--age-recipients-file stringArray Path to a file holding public keys, one per line, which may mix age recipients and SSH public keys; an authorized_keys file works as it is. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.recipients_files in the cargoship config file.
--allow-downgrade Continue when a host already runs an engine version newer than the one the package carries. A downgrade is refused by default, because an engine does not support being moved backwards and the data directory it leaves behind was written by the newer version. Set this when the move backwards is deliberate and the hosts are expected to survive it.
--allow-unmanaged-nodes Continue when the cluster holds a node that no host in the config accounts for. An apply never removes a node, so by default one left behind by a host deleted from the config stops the run. Set this when the extra nodes were joined deliberately and cargoship should leave them alone.
-a, --architecture string Architecture for OCI images and Zarf packages
-c, --concurrency int Maximum number of hosts to configure in parallel, set to 0 for unlimited. (default 30)
--config string Config file used to bootstrap a cluster.
--confirm Proceed with the run. Without it, a command that would change a host reports what it needs and stops.
--dry-run Report what would be done without changing any host. Connects to every host and runs the preflight checks for real, then lists the phases it did not run. Does not need --confirm.
-F, --firewall Whether to update every host node's firewall configuration.
-h, --help help for apply
-H, --hosts Whether to update every host node's /etc/hosts file.
-k, --key string Path to public key file for validating signed packages
--kubeconfig string Path of the kubeconfig file to merge the admin creds for this cluster into. The file is created when it does not exist, and an existing one keeps every other cluster it holds. Defaults to the standard location: KUBECONFIG when set, otherwise ~/.kube/config.
--label-nodes Whether to check and add the node-role.kubernetes.io/PROFILE label on cluster nodes. The phase reads the cluster's admin credentials off a controller itself, so this does not depend on --update-kubeconfig.
--receipt-path string Write a structured JSON record of the run to PATH: what ran, against which package, against which fleet, and what changed. The file is written whether the run succeeds or fails, and is created readable only by its owner. PATH may carry the placeholders {command} and {timestamp}, which expand to the command that ran and when it started, so a PATH set once in '.distro.receipt_path' gives every run its own file. It names every host in the cluster configuration, so run 'cargoship redact-receipt' over it before attaching it to a ticket.
--timeout string Set the timeout for how long functions will last. (default "60m")
--tmpdir string Specify the temporary directory to use for intermediate files (default "/tmp")
--update-kubeconfig Whether to write the admin creds for this cluster to a kubeconfig file at all. (default true)
--values stringArray Path to a YAML values file overriding the values the package ships with. May be given more than once, with a later file winning over an earlier one, and all of them winning over the values in the cluster config file.
--vault-password-file string Path to a file containing the Ansible Vault password used to decrypt vault-encrypted registry credentials. Falls back to the CARGOSHIP_VAULT_PASSWORD, then ANSIBLE_VAULT_PASSWORD, environment variable.
--verify verifyMode Verify the Cargoship package signature. (default if-possible)
-w, --work-concurrency string Maximum number of workers that will be installed or updated in parallel, as a fixed count or a percentage (e.g. "25%"), set to 0 for unlimited. (default "0")
--zarf-cache string Specify the location of the Zarf cache directory (default "$HOME/.cache/cargoship")
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
create
cargoship create
Creates a Cargoship Package from a given directory or the current directory
Synopsis
Builds an offline distro package from a definition directory – a distro.yaml and whatever sits beside it – producing one compressed archive that carries everything a cluster needs: the engine’s packages and binaries, the OCI images, the configuration templates, and the checksums over all of them.
The archive is fat on purpose. One package holds every architecture it targets, and the host’s own architecture selects what gets uploaded at apply time, so a mixed fleet installs from a single file.
Everything the definition declares is fetched while this runs, which is the one step that needs a network. –registry-override and –file-override redirect those fetches to an internal mirror or to files staged on disk ahead of time, which is how a package is built where there is no route to the upstream hosts. A declared shasum is enforced either way.
–signing-key signs the package as it is built. –reproducible pins the recorded build time so that identical inputs produce a byte-identical archive.
cargoship create [Dir] [flags]
Examples
# Build a package from the definition in the current directory
$ cargoship create .
# Build from another directory, writing the package to ./build/
$ cargoship create ./distro-defs -o ./build/
# Pull images through an internal mirror instead of their upstream registry
$ cargoship create ./distro-defs --registry-override docker.io=mirror.example.com
# Download files through an internal mirror instead of their upstream host
$ cargoship create ./distro-defs --file-override https://rpm.rancher.io=https://mirror.example.com/rpm-rancher
# Build from files staged on disk ahead of time, with no network at all
$ cargoship create ./distro-defs --file-override https://rpm.rancher.io=/srv/staged/rpm-rancher
# Sign the package as it is built, without prompting for the key password
$ cargoship create ./distro-defs --signing-key ./private-key.pem --confirm
# Build a byte-identical package on every run
$ cargoship create ./distro-defs --reproducible
Options
-a, --architecture string Architecture for OCI images and Zarf packages
-c, --confirm Confirms package publish without prompting. Skips prompt for the signing key password
--file-override strings Redirect the file downloads a distro definition declares, as SOURCE=TARGET pairs where SOURCE is a URL prefix and TARGET is either a replacement URL prefix or a local directory of pre-staged files. Repeatable, and comma-separated values are accepted. When several sources match, the longest wins. The declared shasum is still enforced, and a file with no shasum is refused rather than fetched unverified from a mirror.
-h, --help help for create
--insecure-skip-tls-verify Skip checking server's certificate for validity. This flag should only be used if you have a specific reason and accept the reduced security posture.
--oci-concurrency int Number of concurrent layer operations when pulling or pushing images or packages to/from OCI registries. (default 6)
-o, --output string Specify the output (either a directory or an oci:// URL) for the created Cargoship distro package. (default ".")
--plain-http Allow OCI registry connections over HTTP instead of HTTPS. This flag should only be used if you have a specific reason and accept the reduced security posture.
--registry-override strings Specify a mapping of domains to override on package create when pulling images (e.g. --registry-override docker.io=dockerio-reg.enterprise.intranet)
--reproducible Pin the recorded package build time to a fixed value instead of the current time, so identical inputs produce a byte-identical package.
--signing-key string Private key for signing packages. Accepts either a local file path or env://ENV_VAR to read the key from an environment variable. Cloud KMS references are not supported.
--signing-key-pass string Password to the private key used for signing packages
--tag string The tag or version to override the package metadata version with.
--tmpdir string Specify the temporary directory to use for intermediate files (default "/tmp")
--zarf-cache string Specify the location of the Zarf cache directory (default "$HOME/.cache/cargoship")
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
engine-config-sync
cargoship engine-config-sync
Sync engine config (registries, audit, and pod security) to a cluster, draining and restarting the engine service on any node whose config has drifted
Synopsis
Writes the engine configuration a package and a cluster configuration describe – registry mirrors and credentials, audit policy, and pod security – to every host the configuration names, and restarts the engine only where what is on disk no longer matches. A node whose configuration already agrees is left running untouched.
This exists because those three settings are the ones that change without the engine version changing, and a full apply is a heavier way to deliver them. Where a restart is needed the node is drained first, and the nodes are worked through at the rate –work-concurrency allows, so a cluster stays serving while its configuration moves.
Registry credentials encrypted with Ansible Vault or age are decrypted here, and the keys for every one of them are checked before the first host is connected to.
This changes every host whose configuration has drifted, so it needs –confirm. Pass –dry-run to report which nodes would be restarted without restarting one.
cargoship engine-config-sync [Distro Package] [flags]
Examples
# Sync registry, audit, and pod security config to every node that has drifted
$ cargoship engine-config-sync ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm
# Decrypt vault-encrypted registry credentials with a password file
$ cargoship engine-config-sync ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm --vault-password-file ./vault-pass.txt
# Restart at most 25% of the workers at a time
$ cargoship engine-config-sync ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm --work-concurrency 25%
# Sync without touching the local kubeconfig
$ cargoship engine-config-sync ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm --update-kubeconfig=false
Options
--age-identity-file stringArray Path to an age identity file holding the private keys that decrypt registry credentials, or to an SSH private key such as ~/.ssh/id_ed25519. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.identity_files in the cargoship config file, or as a single path in CARGOSHIP_AGE_IDENTITY_FILE.
--age-recipient stringArray A public key to encrypt registry credentials to: either an age recipient such as age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p, or an SSH public key such as 'ssh-ed25519 AAAAC3Nza...'. Repeatable; also settable as age.recipients in the cargoship config file, or space-separated in CARGOSHIP_AGE_RECIPIENTS. Giving any recipient makes cargoship write age ciphertext instead of Ansible Vault.
--age-recipients-file stringArray Path to a file holding public keys, one per line, which may mix age recipients and SSH public keys; an authorized_keys file works as it is. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.recipients_files in the cargoship config file.
--allow-downgrade Continue when a host already runs an engine version newer than the one the package carries. A downgrade is refused by default, because an engine does not support being moved backwards and the data directory it leaves behind was written by the newer version. Set this when the move backwards is deliberate and the hosts are expected to survive it.
-a, --architecture string Architecture for OCI images and Zarf packages
-c, --concurrency int Maximum number of hosts to configure in parallel, set to 0 for unlimited. (default 30)
--config string Config file used to bootstrap a cluster.
--confirm Proceed with the run. Without it, a command that would change a host reports what it needs and stops.
--dry-run Report what would be done without changing any host. Connects to every host and runs the preflight checks for real, then lists the phases it did not run. Does not need --confirm.
-h, --help help for engine-config-sync
-k, --key string Path to public key file for validating signed packages
--kubeconfig string Path of the kubeconfig file to merge the admin creds for this cluster into. The file is created when it does not exist, and an existing one keeps every other cluster it holds. Defaults to the standard location: KUBECONFIG when set, otherwise ~/.kube/config.
--label-nodes Whether to check and add the node-role.kubernetes.io/PROFILE label on cluster nodes. The phase reads the cluster's admin credentials off a controller itself, so this does not depend on --update-kubeconfig.
--receipt-path string Write a structured JSON record of the run to PATH: what ran, against which package, against which fleet, and what changed. The file is written whether the run succeeds or fails, and is created readable only by its owner. PATH may carry the placeholders {command} and {timestamp}, which expand to the command that ran and when it started, so a PATH set once in '.distro.receipt_path' gives every run its own file. It names every host in the cluster configuration, so run 'cargoship redact-receipt' over it before attaching it to a ticket.
--timeout string Set the timeout for how long functions will last. (default "60m")
--tmpdir string Specify the temporary directory to use for intermediate files (default "/tmp")
--update-kubeconfig Whether to write the admin creds for this cluster to a kubeconfig file at all. (default true)
--values stringArray Path to a YAML values file overriding the values the package ships with. May be given more than once, with a later file winning over an earlier one, and all of them winning over the values in the cluster config file.
--vault-password-file string Path to a file containing the Ansible Vault password used to decrypt vault-encrypted registry credentials. Falls back to the CARGOSHIP_VAULT_PASSWORD, then ANSIBLE_VAULT_PASSWORD, environment variable.
--verify verifyMode Verify the Cargoship package signature. (default if-possible)
-w, --work-concurrency string Maximum number of workers that will be installed or updated in parallel, as a fixed count or a percentage (e.g. "25%"), set to 0 for unlimited. (default "0")
--zarf-cache string Specify the location of the Zarf cache directory (default "$HOME/.cache/cargoship")
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
import
cargoship import
Converts an OCI Image Layout archive into a native Cargoship package
Synopsis
Converts an OCI Image Layout archive (such as one written by skopeo copy … oci-archive:) or an OCI layout directory into a native Cargoship package archive, extracting all package layers and verifying package integrity.
cargoship import [ARCHIVE] [flags]
Examples
# Import a Skopeo oci-archive into the current directory
$ cargoship import ./my-package.tar
# Import an oci-archive into a specific output directory
$ cargoship import ./my-package.tar --output ./build/
# Import an extracted OCI directory layout
$ cargoship import ./oci-layout-dir -o ./build/
Options
-a, --architecture string Architecture for OCI images and Zarf packages
-h, --help help for import
-k, --key string Path to public key file for validating signed packages
-o, --output string Specify the output (either a directory or an oci:// URL) for the created Cargoship distro package. (default ".")
--tmpdir string Specify the temporary directory to use for intermediate files (default "/tmp")
--verify verifyMode Verify the Cargoship package signature. (default if-possible)
--zarf-cache string Specify the location of the Zarf cache directory (default "$HOME/.cache/cargoship")
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
inventory
cargoship inventory
Generates a cluster inventory from another source of truth
Synopsis
Generates the cluster inventory cargoship installs from, out of a description of the fleet that already exists somewhere else. These commands install nothing and touch no hosts; they write a file for ‘cargoship validate’ to check and ‘cargoship apply’ to read.
Options
--age-identity-file stringArray Path to an age identity file holding the private keys that decrypt registry credentials, or to an SSH private key such as ~/.ssh/id_ed25519. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.identity_files in the cargoship config file, or as a single path in CARGOSHIP_AGE_IDENTITY_FILE.
--age-recipient stringArray A public key to encrypt registry credentials to: either an age recipient such as age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p, or an SSH public key such as 'ssh-ed25519 AAAAC3Nza...'. Repeatable; also settable as age.recipients in the cargoship config file, or space-separated in CARGOSHIP_AGE_RECIPIENTS. Giving any recipient makes cargoship write age ciphertext instead of Ansible Vault.
--age-recipients-file stringArray Path to a file holding public keys, one per line, which may mix age recipients and SSH public keys; an authorized_keys file works as it is. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.recipients_files in the cargoship config file.
-h, --help help for inventory
--vault-password-file string Path to a file containing the Ansible Vault password. Falls back to the CARGOSHIP_VAULT_PASSWORD, then ANSIBLE_VAULT_PASSWORD, environment variable.
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
- cargoship inventory from-ansible - Translates an Ansible inventory into a cluster inventory
from-ansible
cargoship inventory from-ansible
Translates an Ansible inventory into a cluster inventory
Synopsis
Translates an inventory Ansible has already resolved into a ZarfCluster document, deriving each host’s role from the Ansible groups it belongs to.
This does not read an Ansible inventory file. Ansible resolves the inventory – group membership, group_vars, host_vars, dynamic inventory plugins, and the precedence rules over all of them – and this reads the resolved result: a JSON document holding ‘groups’, ‘hostvars’, an optional ‘roleGroups’ mapping, and the cluster-wide settings an Ansible inventory has no way to carry. The cargoship Ansible collection produces that document; write it by hand to reproduce a translation outside a playbook run.
By default the Ansible group named ‘controller’ supplies the control-plane nodes and the group named ‘worker’ supplies the rest. Set ‘roleGroups’ to map cargoship’s roles onto the group names the inventory actually uses. A host in none of the mapped groups is left out, so a play’s inventory may carry hosts that are not part of the cluster; a host in groups mapped to two different roles is an error.
Host order is not cosmetic. Controllers are written first, and the first controller in the document becomes the cluster leader.
Each host’s connection details come from its Ansible variables – ansible_host, ansible_user, ansible_port, ansible_ssh_private_key_file – and everything cargoship needs beyond those comes from variables under a ‘cargoship_’ prefix. A ‘cargoship_’ variable cargoship does not read is an error rather than a value ignored, because a misspelled variable and an unset one are indistinguishable at install time.
The generated document is checked against the inventory schema before it is written. The output goes to stdout, or to –output. Ansible does not connect to the fleet: cargoship opens every SSH connection itself, from the node it runs on.
cargoship inventory from-ansible [FILE] [flags]
Examples
# Translate a resolved Ansible inventory, and check the result against the schema
$ cargoship inventory from-ansible ./resolved.json -o ./inventory.yaml
$ cargoship validate ./inventory.yaml
# Read from stdin and write to stdout, so it composes with whatever produced the projection
$ cargoship inventory from-ansible < ./resolved.json | tee ./inventory.yaml
# Translate one projection of a fleet for a second cluster without rewriting it
$ cargoship inventory from-ansible ./resolved.json --name staging --loadbalancer staging-kc.test.com
Options
-h, --help help for from-ansible
--loadbalancer string Control-plane address, overriding the one in the input. Cargoship adds it to the API server's TLS subject alternative names.
--name string Cluster name, overriding the one in the input. It becomes metadata.name, and the context name in the kubeconfig.
-o, --output string Path to write the generated inventory to. Omit it to write to stdout.
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
--vault-password-file string Path to a file containing the Ansible Vault password. Falls back to the CARGOSHIP_VAULT_PASSWORD, then ANSIBLE_VAULT_PASSWORD, environment variable.
SEE ALSO
- cargoship inventory - Generates a cluster inventory from another source of truth
kube-config
cargoship kube-config
Get the admin kube-config for a control-plane node
Synopsis
Fetches the engine’s admin credentials from the first control-plane node the cluster configuration names and merges them into a kubeconfig file. The file is created when it does not exist, and an existing one keeps every other cluster it already holds.
The server address is rewritten to the control-plane address the configuration declares, rather than the node the credentials came from, so the context keeps working when that node does not. The context is named after the cluster’s metadata.name.
This changes no host – it connects, reads, and writes a file locally – so it needs no –confirm. The file written to is –kubeconfig, which defaults to KUBECONFIG when that is set and to the standard location otherwise.
cargoship kube-config [flags]
Examples
# Fetch the admin kubeconfig from an RKE2 control-plane node
$ cargoship kube-config --config ./cargoship-config.yaml --distro rke2
# Fetch it from a K3s cluster
$ cargoship kube-config --config ./cargoship-config.yaml --distro k3s
# Use the distro set in the resolved cargoship config
$ cargoship kube-config --config ./cargoship-config.yaml
Options
--config string Config file used to bootstrap a cluster.
-D, --distro string What type of distro we will get the admin config from. Valid options are: 'rke2', 'k3s'.
-h, --help help for kube-config
--kubeconfig string Path of the kubeconfig file to merge the admin creds for this cluster into. The file is created when it does not exist, and an existing one keeps every other cluster it holds. Defaults to the standard location: KUBECONFIG when set, otherwise ~/.kube/config.
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
prepare
cargoship prepare
Prepares the nodes, including restarting the node if new kernel modules are enabled
Synopsis
Brings every host the cluster configuration names up to the prerequisites the package declares, without installing the engine: environment variables and sysctl settings, container-selinux on hosts with SELinux enabled, the fapolicyd rules the distro supplies, and the kernel modules the package asks for. A host that gained a module is rebooted, because the module has to be loaded before an apply can use it.
Running this is optional. An apply runs the same preparation phases itself, and this command exists so that the disruptive half can be done on its own schedule – ahead of a maintenance window rather than inside it.
–hosts and –firewall each opt into rewriting a part of the host that cargoship otherwise leaves alone. The fapolicyd rules are not one of them: they are written wherever a host runs fapolicyd and the package carries rules for it.
This changes every host it is pointed at, so it needs –confirm. Pass –dry-run to report what it would change instead.
cargoship prepare [Distro Package] [flags]
Examples
# Prepare every node in the config for an install
$ cargoship prepare ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm
# Update /etc/hosts, the firewall, and fapolicyd on every node
$ cargoship prepare ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm -H -F -f
# Prepare at most five hosts at a time
$ cargoship prepare ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm --concurrency 5
# Prepare workers 25% at a time
$ cargoship prepare ./build/cargoship-distro-amd64.tar.zst --config ./cargoship-config.yaml --confirm --work-concurrency 25%
Options
-a, --architecture string Architecture for OCI images and Zarf packages
-c, --concurrency int Maximum number of hosts to configure in parallel, set to 0 for unlimited. (default 30)
--config string Config file used to bootstrap a cluster.
--confirm Proceed with the run. Without it, a command that would change a host reports what it needs and stops.
--dry-run Report what would be done without changing any host. Connects to every host and runs the preflight checks for real, then lists the phases it did not run. Does not need --confirm.
-F, --firewall Whether to update every host node's firewall configuration.
-h, --help help for prepare
-H, --hosts Whether to update every host node's /etc/hosts file.
-k, --key string Path to public key file for validating signed packages
--receipt-path string Write a structured JSON record of the run to PATH: what ran, against which package, against which fleet, and what changed. The file is written whether the run succeeds or fails, and is created readable only by its owner. PATH may carry the placeholders {command} and {timestamp}, which expand to the command that ran and when it started, so a PATH set once in '.distro.receipt_path' gives every run its own file. It names every host in the cluster configuration, so run 'cargoship redact-receipt' over it before attaching it to a ticket.
--timeout string Set the timeout for how long functions will last. (default "60m")
--tmpdir string Specify the temporary directory to use for intermediate files (default "/tmp")
--values stringArray Path to a YAML values file overriding the values the package ships with. May be given more than once, with a later file winning over an earlier one, and all of them winning over the values in the cluster config file.
--verify verifyMode Verify the Cargoship package signature. (default if-possible)
--zarf-cache string Specify the location of the Zarf cache directory (default "$HOME/.cache/cargoship")
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
publish
cargoship publish
Publish the Cargoship Package to an OCI registry
Synopsis
Pushes a built package to an OCI registry, where ‘cargoship apply’ and ‘cargoship pull’ can read it by reference instead of by path. The package is uploaded as OCI artifacts, so any registry that stores them will hold it, and –oci-concurrency sets how many layers move at once.
The signature a package already carries is published with it. Passing –signing-key re-signs the package under a different key on the way out, which is how one built and signed in a development environment is re-signed for a production registry.
cargoship publish [Package] [REPOSITORY] [flags]
Examples
# Publish a package to an OCI registry
$ cargoship publish ./build/cargoship-rancher-rke2-amd64-1.0.0.tar.zst oci://ghcr.io/my-org
# Publish and re-sign the package with a different key
$ cargoship publish ./build/cargoship-rancher-rke2-amd64-1.0.0.tar.zst oci://ghcr.io/my-org --signing-key ./private-key.pem --confirm
# Retry failed layer uploads over a slow link
$ cargoship publish ./build/cargoship-rancher-rke2-amd64-1.0.0.tar.zst oci://ghcr.io/my-org --retries 3 --oci-concurrency 3
# Refuse to publish unless the package carries a signature this key validates
$ cargoship publish ./build/cargoship-rancher-rke2-amd64-1.0.0.tar.zst oci://ghcr.io/my-org --verify=always --key ./public-key.pem
Options
-a, --architecture string Architecture for OCI images and Zarf packages
-c, --confirm Confirms package publish without prompting. Skips prompt for the signing key password
-h, --help help for publish
--insecure-skip-tls-verify Skip checking server's certificate for validity. This flag should only be used if you have a specific reason and accept the reduced security posture.
-k, --key string Path to public key file for validating signed packages
--oci-concurrency int Number of concurrent layer operations when pulling or pushing images or packages to/from OCI registries. (default 6)
--plain-http Allow OCI registry connections over HTTP instead of HTTPS. This flag should only be used if you have a specific reason and accept the reduced security posture.
--retries int Number of retries to perform for Cargoship's operations, such as package publishes.
--signing-key string Private key for signing or re-signing packages with a new key. Accepts either a local file path or env://ENV_VAR to read the key from an environment variable. Cloud KMS references are not supported.
--signing-key-pass string Password to the private key used for publishing packages
--tag string The tag to be used in the OCI reference for the package in the registry
--tmpdir string Specify the temporary directory to use for intermediate files (default "/tmp")
--verify verifyMode Verify the Cargoship package signature. (default if-possible)
--zarf-cache string Specify the location of the Zarf cache directory (default "$HOME/.cache/cargoship")
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
pull
cargoship pull
Pulls a Cargoship package from a remote registry and saves it to the local filesystem
Synopsis
Downloads a package from an OCI registry or an https:// URL and writes it to the local filesystem as a single archive, so that it can be carried to a machine with no route to the registry it came from. An apply reads the same references directly, so pulling first is for staging rather than something an install needs.
–shasum checks the downloaded archive against a checksum you already hold, which is the check to use when the package is to be trusted on the strength of something other than its signature.
cargoship pull [Package] [flags]
Examples
# Pull a package into the current directory
$ cargoship pull oci://ghcr.io/my-org/my-package:1.0.0
# Pull it into ./build/ instead
$ cargoship pull oci://ghcr.io/my-org/my-package:1.0.0 --output ./build/
# Check the downloaded package against a known checksum
$ cargoship pull oci://ghcr.io/my-org/my-package:1.0.0 --shasum 4a4f1f5eb0a1e3f2c9b6d0b2b6a0d3f4c5e6a7b8c9d0e1f2a3b4c5d6e7f8a9b0
# Refuse to keep the package unless a signature validates against this key
$ cargoship pull oci://ghcr.io/my-org/my-package:1.0.0 --verify=always --key ./public-key.pem
Options
-a, --architecture string Architecture for OCI images and Zarf packages
-h, --help help for pull
--insecure-skip-tls-verify Skip checking server's certificate for validity. This flag should only be used if you have a specific reason and accept the reduced security posture.
-k, --key string Path to public key file for validating signed packages
--oci-concurrency int Number of concurrent layer operations when pulling or pushing images or packages to/from OCI registries. (default 6)
-o, --output string Specify the output (either a directory or an oci:// URL) for the created Cargoship distro package. (default ".")
--plain-http Allow OCI registry connections over HTTP instead of HTTPS. This flag should only be used if you have a specific reason and accept the reduced security posture.
--shasum string Shasum of the package to pull.
--tmpdir string Specify the temporary directory to use for intermediate files (default "/tmp")
--verify verifyMode Verify the Cargoship package signature. (default if-possible)
--zarf-cache string Specify the location of the Zarf cache directory (default "$HOME/.cache/cargoship")
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
redact-receipt
cargoship redact-receipt
Rewrites a run receipt so it can be attached to an issue
Synopsis
Reads the run receipt at RECEIPT and writes a copy with everything that identifies the fleet removed, for attaching to a bug report. Hosts are renumbered host-01, host-02 in the order the receipt lists them, consistently across every phase, so it still reads as a sequence of events on particular machines without naming them. Diff bodies are dropped entirely – an engine configuration diff carries join tokens and registry credentials – leaving the file paths and the reasons they changed, which is what a maintainer triages from. The cluster name, the operator, the management node and the command line go with them. What stays is the phase ledger, the package provenance, the outcome, and the architecture and OS of each host. Redaction is one way: the original is left untouched, because it is the audit record and this is only a copy to send.
cargoship redact-receipt [RECEIPT] [flags]
Examples
# Print a shareable copy of a run receipt
$ cargoship redact-receipt ./apply-20260928-143000.json
# Write one to a file, ready to attach to an issue
$ cargoship redact-receipt ./apply-20260928-143000.json -o ./share.json
Options
-h, --help help for redact-receipt
-o, --output string Write the redacted receipt to this path instead of standard output. The file is created readable only by its owner.
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
reset
cargoship reset
Reset a cluster, stopping, uninstalling, and removing all data for an engine
Synopsis
Removes the engine and the data it wrote from every host the cluster configuration names. Each node is deleted from the cluster, drained first where that is enabled, then the engine’s packages or binaries are uninstalled and the service manager is reloaded.
This is destructive and it backs nothing up. Nothing a workload kept on a host survives it, and no command puts the cluster back: what follows a reset is an apply, which bootstraps a new one.
–distro names the engine to remove. A reset loads no package, so it has nothing else to read the engine’s identity from.
This needs –confirm. Pass –dry-run to report every host it would reset, and what it would do to each, without touching one.
cargoship reset [flags]
Examples
# Reset an RKE2 cluster, uninstalling the engine and removing its data
$ cargoship reset --config ./cargoship-config.yaml --distro rke2 --confirm
# Reset a K3s cluster
$ cargoship reset --config ./cargoship-config.yaml --distro k3s --confirm
# Reset at most five hosts at a time
$ cargoship reset --config ./cargoship-config.yaml --distro rke2 --confirm --concurrency 5
# Reset workers 25% at a time
$ cargoship reset --config ./cargoship-config.yaml --distro rke2 --confirm --work-concurrency 25%
Options
-c, --concurrency int Maximum number of hosts to configure in parallel, set to 0 for unlimited. (default 30)
--config string Config file used to bootstrap a cluster.
--confirm Proceed with the run. Without it, a command that would change a host reports what it needs and stops.
-D, --distro string What type of distro that will be reset. Valid options are: 'rke2', 'k3s'.
--dry-run Report what would be done without changing any host. Connects to every host and runs the preflight checks for real, then lists the phases it did not run. Does not need --confirm.
-h, --help help for reset
--receipt-path string Write a structured JSON record of the run to PATH: what ran, against which package, against which fleet, and what changed. The file is written whether the run succeeds or fails, and is created readable only by its owner. PATH may carry the placeholders {command} and {timestamp}, which expand to the command that ran and when it started, so a PATH set once in '.distro.receipt_path' gives every run its own file. It names every host in the cluster configuration, so run 'cargoship redact-receipt' over it before attaching it to a ticket.
--target-hosts strings Specific hosts to delete and uninstall, matched by hostname or connection address. Repeatable. Defaults to every host in the cluster configuration.
-w, --work-concurrency string Maximum number of workers that will be installed or updated in parallel, as a fixed count or a percentage (e.g. "25%"), set to 0 for unlimited. (default "0")
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
schema
cargoship schema
Writes out a JSON Schema for one of cargoship’s own file formats
Synopsis
Writes out the JSON Schema for KIND – ‘inventory’ for a cluster file, ‘package’ for a distro.yaml, ‘config’ for a cargoship config file – so that an editor can complete and check those files without reaching out to GitHub for a hosted copy. The schema comes out of the binary, so it is the schema this build of cargoship validates against, which a hosted one pinned to the default branch is not, and it is available on a network that has never seen the internet. The output goes to stdout, or to –output. Point an editor at a written file with a ‘# yaml-language-server: $schema=PATH’ comment on the first line. With –package, the package’s own values schema is grafted onto spec.config.values, which is the one block of an inventory whose vocabulary belongs to the package rather than to cargoship – so the addon names, chart settings, and everything else that package accepts complete in the editor too. That composed schema describes the overrides on their own, while cargoship validates them merged with the values the package ships; it catches an unknown key, a wrong type, or a name outside an enum, and does not reproduce install-time validation.
cargoship schema KIND [flags]
Examples
# Write the inventory schema next to an inventory file, for an editor to read
$ cargoship schema inventory -o ./cluster-v1alpha1-schema.json
# Then point the inventory at it, as its first line
$ head -1 ./inventory.yaml
# yaml-language-server: $schema=./cluster-v1alpha1-schema.json
# Compose in a package's own values, so spec.config.values completes too
$ cargoship schema inventory --package ./package.tar.zst -o ./inventory.schema.json
# The same, straight from a package source directory, before it is built
$ cargoship schema inventory --package ./example/rke2-cilium-vsphere/v1_37/v1.37.0-rke2r1
# The schema for a distro.yaml, and for a cargoship config file
$ cargoship schema package -o ./distro.schema.json
$ cargoship schema config -o ./cargoship.schema.json
Options
-a, --architecture string Architecture for OCI images and Zarf packages
-h, --help help for schema
--insecure-skip-tls-verify Skip checking server's certificate for validity. This flag should only be used if you have a specific reason and accept the reduced security posture.
-k, --key string Path to public key file for validating signed packages
-o, --output string Path to write the schema to. Omit it to write to stdout.
--package string Graft a package's own values schema onto spec.config.values. Takes anything 'cargoship apply' takes -- a tarball, an oci:// or https:// reference -- or a package source directory or distro.yaml, so it works before the package is built. Only valid for the 'inventory' schema.
--plain-http Allow OCI registry connections over HTTP instead of HTTPS. This flag should only be used if you have a specific reason and accept the reduced security posture.
--tmpdir string Specify the temporary directory to use for intermediate files (default "/tmp")
--verify verifyMode Verify the Cargoship package signature. (default if-possible)
--zarf-cache string Specify the location of the Zarf cache directory (default "$HOME/.cache/cargoship")
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
sha256sum
cargoship sha256sum
Generates a SHA256SUM for the given file
cargoship sha256sum [ FILE | URL ] [flags]
Examples
# Checksum a local file
$ cargoship sha256sum ./build/cargoship-rancher-rke2-amd64-1.0.0.tar.zst
# Checksum a remote file, downloading it first
$ cargoship sha256sum https://example.com/artifact.tar.gz
# Checksum one file inside an archive, for use with a component's files.extractPath
$ cargoship sha256sum ./artifact.tar.gz --extract-path ./bin/tool
# Same thing, using the shorter alias
$ cargoship sum ./build/cargoship-rancher-rke2-amd64-1.0.0.tar.zst
Options
-e, --extract-path string The path inside of an archive to use to calculate the sha256sum (i.e. for use with "files.extractPath")
-h, --help help for sha256sum
--tmpdir string Specify the temporary directory to use for intermediate files (default "/tmp")
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
sign
cargoship sign
Signs an existing Cargoship distro package
Synopsis
Signs an existing Cargoship distro package with a private key. The package can be a local tarball or pulled from an OCI registry. The signature is created by signing the distro.yaml file and does not modify the package checksums.
cargoship sign PACKAGE_SOURCE [flags]
Examples
# Sign an unsigned package
$ cargoship sign cargoship-rancher-rke2-amd64-1.0.0.tar.zst --signing-key ./private-key.pem
# Re-sign with a new key (overwrite existing signature)
$ cargoship sign cargoship-rancher-rke2-amd64-1.0.0.tar.zst --signing-key ./new-key.pem --overwrite
# Sign a package from an OCI registry and output to a local directory
$ cargoship sign oci://ghcr.io/my-org/my-package:1.0.0 --signing-key ./private-key.pem --output ./signed/
# Sign a package and publish directly to an OCI registry
$ cargoship sign cargoship-rancher-rke2-amd64-1.0.0.tar.zst --signing-key ./private-key.pem --output oci://ghcr.io/my-org/signed-packages
# Sign with a key held in an environment variable rather than on disk
$ cargoship sign cargoship-rancher-rke2-amd64-1.0.0.tar.zst --signing-key env://COSIGN_PRIVATE_KEY
Options
-a, --architecture string Architecture for OCI images and Zarf packages
--confirm Skip the interactive confirmation prompt before uploading to the Rekor transparency log (equivalent to cosign --yes).
-h, --help help for sign
--insecure-skip-tls-verify Skip checking server's certificate for validity. This flag should only be used if you have a specific reason and accept the reduced security posture.
-k, --key string Public key to verify the existing signature before re-signing (optional)
--oci-concurrency int Number of concurrent layer operations when pulling or pushing images or packages to/from OCI registries. (default 6)
-o, --output string Output destination for the signed package. Can be a local directory or an OCI registry URL (oci://). Default: same directory as source package for files, current directory for OCI sources
--overwrite Overwrite an existing signature if the package is already signed
--plain-http Allow OCI registry connections over HTTP instead of HTTPS. This flag should only be used if you have a specific reason and accept the reduced security posture.
--retries int Number of retries to perform for Cargoship's operations, such as package publishes.
--signing-key string Private key for signing packages. Accepts either a local file path or env://ENV_VAR to read the key from an environment variable. Cloud KMS references are not supported.
--signing-key-pass string Password for encrypted private key
--tmpdir string Specify the temporary directory to use for intermediate files (default "/tmp")
--verify verifyMode Verify the Cargoship package signature. (default if-possible)
--zarf-cache string Specify the location of the Zarf cache directory (default "$HOME/.cache/cargoship")
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
validate
cargoship validate
Checks a cluster inventory, package definition, or config file against its schema
Synopsis
Checks one or more files against the JSON Schema cargoship generates for that file format, reporting every problem rather than the first. Cargoship parses these files without strict key checking, so a misspelled key is dropped rather than reported; this catches that, along with a wrong type and a value outside an enumerated list. The schema is read from the binary, so no network access is needed.
The schema is chosen from the document’s own ‘kind’ field. A cargoship config file declares no kind and has to be named with –kind.
spec.config.values is untyped in the inventory schema, because its shape belongs to whichever package is being installed. Pass –package to check it as well.
cargoship validate FILE... [flags]
Examples
# Check a cluster inventory against the inventory schema:
$ cargoship validate ./inventory.yaml
# Check every inventory in a directory, reporting all of them in one pass:
$ cargoship validate ./inventories/*.yaml
# Check a cargoship config file, which declares no kind of its own:
$ cargoship validate --kind config ./cargoship-config.yaml
# Check an inventory's spec.config.values against the package it will be installed with:
$ cargoship validate ./inventory.yaml --package ./package.tar.zst
Options
-a, --architecture string Architecture for OCI images and Zarf packages
-h, --help help for validate
--insecure-skip-tls-verify Skip checking server's certificate for validity. This flag should only be used if you have a specific reason and accept the reduced security posture.
-k, --key string Path to public key file for validating signed packages
--kind string Name the schema to check against instead of reading the document's 'kind' field. Required for a cargoship config file, which declares no kind.
--package string Check spec.config.values against a package's own values schema. Takes anything 'cargoship apply' takes -- a tarball, an oci:// or https:// reference -- or a package source directory or distro.yaml, so it works before the package is built. Only valid for the 'inventory' schema.
--plain-http Allow OCI registry connections over HTTP instead of HTTPS. This flag should only be used if you have a specific reason and accept the reduced security posture.
--tmpdir string Specify the temporary directory to use for intermediate files (default "/tmp")
--verify verifyMode Verify the Cargoship package signature. (default if-possible)
--zarf-cache string Specify the location of the Zarf cache directory (default "$HOME/.cache/cargoship")
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
vault
cargoship vault
Encrypts and decrypts cluster configuration values with Ansible Vault or age
Synopsis
Groups the commands that read and write encrypted values for a cluster configuration, in either of the two formats cargoship supports: Ansible Vault, keyed by a shared password given via –vault-password-file or the CARGOSHIP_VAULT_PASSWORD environment variable, and age, keyed by public keys given via –age-recipient and read back with –age-identity-file. Cargoship decrypts a registry’s user/pass/token and tls.ca fields at apply time. Which format a value is in is read from the value itself, so one configuration can hold both, and moving between them is ‘cargoship vault rekey’. The group is still called vault because renaming it would break every script that calls it.
Options
-h, --help help for vault
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
- cargoship vault decrypt - Decrypts an Ansible Vault or age value, printing the plaintext
- cargoship vault decrypt-file - Decrypts every registry credential in a config file, in place
- cargoship vault decrypt-path - Decrypts the values a config file holds at one or more YAML paths, in place
- cargoship vault encrypt - Encrypts a value with Ansible Vault or age, for use in a registry’s user/pass/token fields
- cargoship vault encrypt-file - Encrypts every registry credential in a config file, in place
- cargoship vault encrypt-path - Encrypts the values a config file already holds at one or more YAML paths, in place
- cargoship vault keygen - Generates an age key pair, or prints the public key of one you already hold
- cargoship vault rekey - Re-wraps every encrypted registry credential in a config file, optionally under a new key
decrypt-file
cargoship vault decrypt-file
Decrypts every registry credential in a config file, in place
Synopsis
Decrypts every encrypted registry credential FILE holds – each registry’s auth.user, auth.pass, auth.token and tls.ca – and writes the plaintext back to FILE, leaving comments, key order, and the rest of the document untouched. A field that is not encrypted is skipped. This is the inverse of ‘cargoship vault encrypt-file’, and leaves the credentials readable to anyone who can read the file.
cargoship vault decrypt-file FILE [flags]
Examples
# Decrypt every registry credential in a cluster configuration
$ cargoship vault decrypt-file ./cluster.yaml --vault-password-file ./vault-pass.txt
# Check that every encrypted credential decrypts, without writing plaintext to disk
$ cargoship vault decrypt-file ./cluster.yaml --vault-password-file ./vault-pass.txt --dry-run > /dev/null
# Decrypt a configuration encrypted to an age recipient
$ cargoship vault decrypt-file ./cluster.yaml --age-identity-file ./key.txt
# Decrypt one encrypted to an SSH public key, using the private key it pairs with
$ cargoship vault decrypt-file ./cluster.yaml --age-identity-file ~/.ssh/id_ed25519
# Rotate the password a whole configuration is vaulted with
$ cargoship vault decrypt-file ./cluster.yaml --vault-password-file ./old-pass.txt
$ cargoship vault encrypt-file ./cluster.yaml --vault-password-file ./new-pass.txt
Options
--age-identity-file stringArray Path to an age identity file holding the private keys that decrypt registry credentials, or to an SSH private key such as ~/.ssh/id_ed25519. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.identity_files in the cargoship config file, or as a single path in CARGOSHIP_AGE_IDENTITY_FILE.
--age-recipient stringArray A public key to encrypt registry credentials to: either an age recipient such as age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p, or an SSH public key such as 'ssh-ed25519 AAAAC3Nza...'. Repeatable; also settable as age.recipients in the cargoship config file, or space-separated in CARGOSHIP_AGE_RECIPIENTS. Giving any recipient makes cargoship write age ciphertext instead of Ansible Vault.
--age-recipients-file stringArray Path to a file holding public keys, one per line, which may mix age recipients and SSH public keys; an authorized_keys file works as it is. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.recipients_files in the cargoship config file.
--dry-run Print the resulting document to stdout instead of writing it back to FILE.
-h, --help help for decrypt-file
--vault-password-file string Path to a file containing the Ansible Vault password. Falls back to the CARGOSHIP_VAULT_PASSWORD, then ANSIBLE_VAULT_PASSWORD, environment variable.
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship vault - Encrypts and decrypts cluster configuration values with Ansible Vault or age
decrypt-path
cargoship vault decrypt-path
Decrypts the values a config file holds at one or more YAML paths, in place
Synopsis
Decrypts the encrypted values FILE holds at each YAML_PATH and writes the plaintext back to FILE, leaving comments, key order, and the rest of the document untouched. Each YAML_PATH names a value inside FILE rather than a file on disk: it is a YAML path such as ‘.spec.config.registries[0].auth.pass’, and the leading ‘$’ go-yaml uses is optional. Quote it, since it usually contains characters a shell would otherwise expand. Give as many as you like – FILE is written once, after every one of them has decrypted, so a path that is missing or plaintext already leaves FILE as it was. This is the inverse of ‘cargoship vault encrypt-path’, and leaves the values readable to anyone who can read the file.
cargoship vault decrypt-path FILE YAML_PATH [YAML_PATH...] [flags]
Examples
# Put the plaintext password back into a config, in place of the ciphertext
$ cargoship vault decrypt-path ./cluster.yaml '.spec.config.registries[0].auth.pass' --vault-password-file ./vault-pass.txt
# Decrypt several values in one pass
$ cargoship vault decrypt-path ./cluster.yaml '.x-tra.test.user' '.x-tra.test.pass' --vault-password-file ./vault-pass.txt
# Check that a vaulted value decrypts, without writing the plaintext to disk
$ cargoship vault decrypt-path ./cluster.yaml '.spec.config.registries[0].auth.pass' --vault-password-file ./vault-pass.txt --dry-run
# Rotate the password a config is vaulted with
$ cargoship vault decrypt-path ./cluster.yaml '.spec.config.registries[0].auth.pass' --vault-password-file ./old-pass.txt
$ cargoship vault encrypt-path ./cluster.yaml '.spec.config.registries[0].auth.pass' --vault-password-file ./new-pass.txt
Options
--age-identity-file stringArray Path to an age identity file holding the private keys that decrypt registry credentials, or to an SSH private key such as ~/.ssh/id_ed25519. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.identity_files in the cargoship config file, or as a single path in CARGOSHIP_AGE_IDENTITY_FILE.
--age-recipient stringArray A public key to encrypt registry credentials to: either an age recipient such as age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p, or an SSH public key such as 'ssh-ed25519 AAAAC3Nza...'. Repeatable; also settable as age.recipients in the cargoship config file, or space-separated in CARGOSHIP_AGE_RECIPIENTS. Giving any recipient makes cargoship write age ciphertext instead of Ansible Vault.
--age-recipients-file stringArray Path to a file holding public keys, one per line, which may mix age recipients and SSH public keys; an authorized_keys file works as it is. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.recipients_files in the cargoship config file.
--dry-run Print the resulting document to stdout instead of writing it back to FILE.
-h, --help help for decrypt-path
--vault-password-file string Path to a file containing the Ansible Vault password. Falls back to the CARGOSHIP_VAULT_PASSWORD, then ANSIBLE_VAULT_PASSWORD, environment variable.
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship vault - Encrypts and decrypts cluster configuration values with Ansible Vault or age
decrypt
cargoship vault decrypt
Decrypts an Ansible Vault or age value, printing the plaintext
Synopsis
Decrypts VALUE, a string produced by ‘cargoship vault encrypt’ in either format, and prints the plaintext to stdout. The format is read from the value itself: a $ANSIBLE_VAULT prefix needs the vault password, an —–BEGIN AGE ENCRYPTED FILE—– prefix needs –age-identity-file. If VALUE is omitted, it is read from stdin, which is the easier way to hand over an age value, since one begins with dashes and is read as a flag when given as an argument; after the flags, ‘–’ works too. The plaintext is written as it is, with a trailing newline added only when it does not already end in one, so that a multi-line value can be redirected straight into a file.
cargoship vault decrypt [VALUE] [flags]
Examples
# Decrypt a value copied out of a config file
$ cargoship vault decrypt '$ANSIBLE_VAULT;1.1;AES256
3862...' --vault-password-file ./vault-pass.txt
# Decrypt a value held in a file
$ cargoship vault decrypt --vault-password-file ./vault-pass.txt < ./encrypted-token.txt
# Write a decrypted CA certificate straight out to a PEM file
$ cargoship vault decrypt --vault-password-file ./vault-pass.txt < ./encrypted-ca.txt > ./ca.pem
# Check a value round-trips under the password a config will be applied with
$ cargoship vault encrypt hunter2 --vault-password-file ./vault-pass.txt | cargoship vault decrypt --vault-password-file ./vault-pass.txt
# Decrypt an age value; pipe it in, since an age value begins with dashes and reads as a flag
$ cargoship vault decrypt --age-identity-file ./key.txt < ./encrypted-token.txt
# The same value as an argument, with "--" after the flags to end flag parsing
$ cargoship vault decrypt --age-identity-file ./key.txt -- "$(cat ./encrypted-token.txt)"
Options
--age-identity-file stringArray Path to an age identity file holding the private keys that decrypt registry credentials, or to an SSH private key such as ~/.ssh/id_ed25519. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.identity_files in the cargoship config file, or as a single path in CARGOSHIP_AGE_IDENTITY_FILE.
--age-recipient stringArray A public key to encrypt registry credentials to: either an age recipient such as age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p, or an SSH public key such as 'ssh-ed25519 AAAAC3Nza...'. Repeatable; also settable as age.recipients in the cargoship config file, or space-separated in CARGOSHIP_AGE_RECIPIENTS. Giving any recipient makes cargoship write age ciphertext instead of Ansible Vault.
--age-recipients-file stringArray Path to a file holding public keys, one per line, which may mix age recipients and SSH public keys; an authorized_keys file works as it is. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.recipients_files in the cargoship config file.
-h, --help help for decrypt
--vault-password-file string Path to a file containing the Ansible Vault password. Falls back to the CARGOSHIP_VAULT_PASSWORD, then ANSIBLE_VAULT_PASSWORD, environment variable.
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship vault - Encrypts and decrypts cluster configuration values with Ansible Vault or age
encrypt-file
cargoship vault encrypt-file
Encrypts every registry credential in a config file, in place
Synopsis
Encrypts every registry credential FILE holds that cargoship decrypts at apply time – each registry’s auth.user, auth.pass, auth.token and tls.ca – and writes them back to FILE as block scalars, leaving comments, key order, and the rest of the document untouched. A field that is absent, empty, or encrypted already is skipped, so running this over a partly encrypted configuration finishes the job and running it twice changes nothing the second time. That skip applies to a value in the other format too: this command never moves a credential between Ansible Vault and age, which is what ‘cargoship vault rekey’ is for. Every credential skipped for a reason worth knowing about is reported, so a run that changes nothing says which values it left alone and why.
cargoship vault encrypt-file FILE [flags]
Examples
# Encrypt every registry credential in a cluster configuration
$ cargoship vault encrypt-file ./cluster.yaml --vault-password-file ./vault-pass.txt
# See what the file would become without writing it
$ cargoship vault encrypt-file ./cluster.yaml --vault-password-file ./vault-pass.txt --dry-run
# Vault a configuration before committing it
$ cargoship vault encrypt-file ./cluster.yaml --vault-password-file ./vault-pass.txt && git add ./cluster.yaml
# Encrypt to the age recipients a team keeps in its config file, with no keys on the command line
$ cargoship vault encrypt-file ./cluster.yaml
# Encrypt to the SSH keys a team already distributes, using its authorized_keys file as it is
$ cargoship vault encrypt-file ./cluster.yaml --age-recipients-file ./authorized_keys
Options
--age-identity-file stringArray Path to an age identity file holding the private keys that decrypt registry credentials, or to an SSH private key such as ~/.ssh/id_ed25519. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.identity_files in the cargoship config file, or as a single path in CARGOSHIP_AGE_IDENTITY_FILE.
--age-recipient stringArray A public key to encrypt registry credentials to: either an age recipient such as age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p, or an SSH public key such as 'ssh-ed25519 AAAAC3Nza...'. Repeatable; also settable as age.recipients in the cargoship config file, or space-separated in CARGOSHIP_AGE_RECIPIENTS. Giving any recipient makes cargoship write age ciphertext instead of Ansible Vault.
--age-recipients-file stringArray Path to a file holding public keys, one per line, which may mix age recipients and SSH public keys; an authorized_keys file works as it is. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.recipients_files in the cargoship config file.
--dry-run Print the resulting document to stdout instead of writing it back to FILE.
--force Encrypt values that are encrypted already, wrapping them a second time.
-h, --help help for encrypt-file
--vault-password-file string Path to a file containing the Ansible Vault password. Falls back to the CARGOSHIP_VAULT_PASSWORD, then ANSIBLE_VAULT_PASSWORD, environment variable.
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship vault - Encrypts and decrypts cluster configuration values with Ansible Vault or age
encrypt-path
cargoship vault encrypt-path
Encrypts the values a config file already holds at one or more YAML paths, in place
Synopsis
Encrypts the values FILE holds at each YAML_PATH and writes them back to FILE as block scalars, leaving comments, key order, and the rest of the document untouched. Each YAML_PATH names a value inside FILE rather than a file on disk: it is a YAML path such as ‘.spec.config.registries[0].auth.pass’, and the leading ‘$’ go-yaml uses is optional. Quote it, since it usually contains characters a shell would otherwise expand. Give as many as you like – FILE is written once, after every one of them has encrypted, so a path that is missing or encrypted already leaves FILE as it was rather than partly rewritten. Cargoship decrypts a registry’s user/pass/token and tls.ca fields at apply time, and warns about a path anywhere else, because nothing unwraps a value encrypted elsewhere.
cargoship vault encrypt-path FILE YAML_PATH [YAML_PATH...] [flags]
Examples
# Encrypt the password a config already holds for its first registry
$ cargoship vault encrypt-path ./cluster.yaml '.spec.config.registries[0].auth.pass' --vault-password-file ./vault-pass.txt
# Encrypt several values in one pass; the file is written once, after all of them have encrypted
$ cargoship vault encrypt-path ./cluster.yaml '.x-tra.test.user' '.x-tra.test.pass' --vault-password-file ./vault-pass.txt
# Encrypt a registry's inline CA certificate, however many lines of PEM it runs to
$ cargoship vault encrypt-path ./cluster.yaml '.spec.config.registries[0].tls.ca' --vault-password-file ./vault-pass.txt
# See what the file would become without writing it
$ cargoship vault encrypt-path ./cluster.yaml '.spec.config.registries[0].auth.token' --vault-password-file ./vault-pass.txt --dry-run
# Encrypt to an age recipient instead of a vault password
$ cargoship vault encrypt-path ./cluster.yaml '.spec.config.registries[0].auth.pass' --age-recipient age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
Options
--age-identity-file stringArray Path to an age identity file holding the private keys that decrypt registry credentials, or to an SSH private key such as ~/.ssh/id_ed25519. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.identity_files in the cargoship config file, or as a single path in CARGOSHIP_AGE_IDENTITY_FILE.
--age-recipient stringArray A public key to encrypt registry credentials to: either an age recipient such as age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p, or an SSH public key such as 'ssh-ed25519 AAAAC3Nza...'. Repeatable; also settable as age.recipients in the cargoship config file, or space-separated in CARGOSHIP_AGE_RECIPIENTS. Giving any recipient makes cargoship write age ciphertext instead of Ansible Vault.
--age-recipients-file stringArray Path to a file holding public keys, one per line, which may mix age recipients and SSH public keys; an authorized_keys file works as it is. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.recipients_files in the cargoship config file.
--dry-run Print the resulting document to stdout instead of writing it back to FILE.
--force Encrypt the value even though it is encrypted already, wrapping it a second time.
-h, --help help for encrypt-path
--vault-password-file string Path to a file containing the Ansible Vault password. Falls back to the CARGOSHIP_VAULT_PASSWORD, then ANSIBLE_VAULT_PASSWORD, environment variable.
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship vault - Encrypts and decrypts cluster configuration values with Ansible Vault or age
encrypt
cargoship vault encrypt
Encrypts a value with Ansible Vault or age, for use in a registry’s user/pass/token fields
Synopsis
Encrypts VALUE, producing a string that cargoship decrypts automatically at apply time when placed in a registry’s user/pass/token field. With –age-recipient or –age-recipients-file the output is armored age ciphertext beginning with —–BEGIN AGE ENCRYPTED FILE—–; otherwise it is Ansible Vault ciphertext beginning with $ANSIBLE_VAULT. If VALUE is omitted, it is read from stdin, or prompted for with hidden input when stdin is a terminal.
cargoship vault encrypt [VALUE] [flags]
Examples
# Encrypt a registry password for a config file's user/pass/token field
$ cargoship vault encrypt my-registry-password --vault-password-file ./vault-pass.txt
# Omit the value to be prompted for it, with the input hidden
$ cargoship vault encrypt --vault-password-file ./vault-pass.txt
# Encrypt a value piped in on stdin
$ printf my-registry-password | cargoship vault encrypt --vault-password-file ./vault-pass.txt
# Encrypt the contents of a file
$ cargoship vault encrypt --vault-password-file ./vault-pass.txt < ./registry-token.txt
# Encrypt to age public keys instead, so no shared password has to be handed around
$ cargoship vault encrypt my-registry-password --age-recipient age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
# Encrypt to every key a team keeps in one file, each of whom can decrypt on their own
$ cargoship vault encrypt my-registry-password --age-recipients-file ./recipients.txt
Options
--age-identity-file stringArray Path to an age identity file holding the private keys that decrypt registry credentials, or to an SSH private key such as ~/.ssh/id_ed25519. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.identity_files in the cargoship config file, or as a single path in CARGOSHIP_AGE_IDENTITY_FILE.
--age-recipient stringArray A public key to encrypt registry credentials to: either an age recipient such as age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p, or an SSH public key such as 'ssh-ed25519 AAAAC3Nza...'. Repeatable; also settable as age.recipients in the cargoship config file, or space-separated in CARGOSHIP_AGE_RECIPIENTS. Giving any recipient makes cargoship write age ciphertext instead of Ansible Vault.
--age-recipients-file stringArray Path to a file holding public keys, one per line, which may mix age recipients and SSH public keys; an authorized_keys file works as it is. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.recipients_files in the cargoship config file.
-h, --help help for encrypt
--vault-password-file string Path to a file containing the Ansible Vault password. Falls back to the CARGOSHIP_VAULT_PASSWORD, then ANSIBLE_VAULT_PASSWORD, environment variable.
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship vault - Encrypts and decrypts cluster configuration values with Ansible Vault or age
keygen
cargoship vault keygen
Generates an age key pair, or prints the public key of one you already hold
Synopsis
Generates an age key pair in the format age-keygen writes, so that the age distribution is not a prerequisite for encrypting a configuration with age, and the file stays readable by ‘age –decrypt’ and anything else that speaks the format. The identity – the private key, AGE-SECRET-KEY-1… – goes to –output, or to stdout when that is omitted; the recipient – the public key, age1… – goes to stderr, so that the file holds the private key alone and the public key can be copied straight into a recipients file. With –public-key it generates nothing and instead prints the public keys held in IDENTITY_FILE, or in stdin when no file is given, which is how a recipient is recovered from a private key you still have. A key pair is not registered anywhere: pass the public key to ‘cargoship vault encrypt’ as –age-recipient, and the file back as –age-identity-file to read those values again.
cargoship vault keygen [IDENTITY_FILE] [flags]
Examples
# Generate a key pair, writing the private key to a file and printing the public key
$ cargoship vault keygen --output ~/.age/cargoship.key
# The same, redirecting instead, which leaves the file mode up to your shell
$ cargoship vault keygen > ~/.age/cargoship.key
# Recover the public key from an identity file you still hold
$ cargoship vault keygen --public-key ~/.age/cargoship.key
# Encrypt a configuration to the key that was just generated, and read it back
$ cargoship vault encrypt-file ./cluster.yaml --age-recipient age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
$ cargoship vault decrypt-file ./cluster.yaml --age-identity-file ~/.age/cargoship.key
Options
-h, --help help for keygen
-o, --output string Path to write the generated identity to, created with mode 0600. An existing file is never overwritten, because anything encrypted to the key it holds would become unreadable. Omit it to write to stdout instead.
-y, --public-key Print the public keys held in IDENTITY_FILE instead of generating a key pair. Reads stdin when no file is given. This reads age identity files; the public key of an SSH key is in the ".pub" file beside it.
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship vault - Encrypts and decrypts cluster configuration values with Ansible Vault or age
rekey
cargoship vault rekey
Re-wraps every encrypted registry credential in a config file, optionally under a new key
Synopsis
Re-wraps every encrypted registry credential FILE holds – each registry’s auth.user, auth.pass, auth.token and tls.ca – under the key named by –new-vault-password-file or by –age-recipient, leaving comments, key order, and the rest of the document untouched. Naming age recipients while reading with the vault password is how a configuration moves from Ansible Vault to age; naming neither re-wraps the values under the key they already carry, which gives every one of them fresh ciphertext without changing the key. The plaintext is never written to FILE: each value is decrypted and encrypted again in memory, which is what makes this safer than a decrypt-file followed by an encrypt-file, where the file holds the credentials in the clear in between. Every encrypted value has to be readable with the keys given, and the command stops without touching FILE if one is not, because a configuration encrypted under two keys is one no single key can read back.
cargoship vault rekey FILE [flags]
Examples
# Move every encrypted registry credential in a cluster configuration to a new vault password
$ cargoship vault rekey ./cluster.yaml --vault-password-file ./old-pass.txt --new-vault-password-file ./new-pass.txt
# Re-salt every encrypted registry credential, keeping the password the file already uses
$ cargoship vault rekey ./cluster.yaml --vault-password-file ./vault-pass.txt
# Check what a rotation would produce without writing it back
$ cargoship vault rekey ./cluster.yaml --vault-password-file ./old-pass.txt --new-vault-password-file ./new-pass.txt --dry-run
# Move a vaulted configuration onto age, reading with the old password and writing to the recipients
$ cargoship vault rekey ./cluster.yaml --vault-password-file ./vault-pass.txt --age-recipient age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
# Re-encrypt to a smaller recipient set, which is how access is revoked
$ cargoship vault rekey ./cluster.yaml --age-identity-file ./key.txt --age-recipients-file ./recipients.txt
Options
--age-identity-file stringArray Path to an age identity file holding the private keys that decrypt registry credentials, or to an SSH private key such as ~/.ssh/id_ed25519. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.identity_files in the cargoship config file, or as a single path in CARGOSHIP_AGE_IDENTITY_FILE.
--age-recipient stringArray A public key to encrypt registry credentials to: either an age recipient such as age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p, or an SSH public key such as 'ssh-ed25519 AAAAC3Nza...'. Repeatable; also settable as age.recipients in the cargoship config file, or space-separated in CARGOSHIP_AGE_RECIPIENTS. Giving any recipient makes cargoship write age ciphertext instead of Ansible Vault.
--age-recipients-file stringArray Path to a file holding public keys, one per line, which may mix age recipients and SSH public keys; an authorized_keys file works as it is. A leading ~ is expanded to your home directory wherever the path comes from. Repeatable; also settable as age.recipients_files in the cargoship config file.
--dry-run Print the resulting document to stdout instead of writing it back to FILE.
-h, --help help for rekey
--new-vault-password-file string Path to a file containing the Ansible Vault password to move to. Omit it to re-salt every value under the key the file already uses, or pass --age-recipient instead to move the file onto age. Deliberately without an environment fallback: the environment holds the password the file is vaulted under now, so an omitted flag would otherwise look like a rotation that never happened.
--vault-password-file string Path to a file containing the Ansible Vault password. Falls back to the CARGOSHIP_VAULT_PASSWORD, then ANSIBLE_VAULT_PASSWORD, environment variable.
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship vault - Encrypts and decrypts cluster configuration values with Ansible Vault or age
verify
cargoship verify
Verifies the signature on a Cargoship distro package
Synopsis
Verifies the signature on an existing Cargoship distro package. The package can be a local tarball or pulled from an OCI registry. Pass –key to verify against a cosign public key, which is the only supported signing method. An unsigned package is reported as a failure. The other commands that load packages verify as part of loading, gated on their –verify flag – this command verifies and nothing else.
cargoship verify PACKAGE_SOURCE [flags]
Examples
# Verify a signed package against a public key
$ cargoship verify cargoship-rancher-rke2-amd64-1.0.0.tar.zst --key ./cosign.pub
# Verify a package in an OCI registry
$ cargoship verify oci://ghcr.io/my-org/my-package:1.0.0 --key ./cosign.pub
# Match the signer identity by pattern instead
Options
-h, --help help for verify
--insecure-skip-tls-verify Skip checking server's certificate for validity. This flag should only be used if you have a specific reason and accept the reduced security posture.
-k, --key string Path to public key file for validating signed packages
--plain-http Allow OCI registry connections over HTTP instead of HTTPS. This flag should only be used if you have a specific reason and accept the reduced security posture.
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
version
cargoship version
Shows the version of the running binary
Synopsis
Displays the version of the release that the current binary was built from.
cargoship version [flags]
Examples
# Print the version of the running binary
$ cargoship version
# Print the full build information as YAML
$ cargoship version --output yaml
# Print it as JSON, for piping into jq
$ cargoship version -o json
Options
-h, --help help for version
-o, --output outputFormat Output format. Valid options are: yaml, json.
Options inherited from parent commands
--log-file Always write a full-verbosity debug log to a file, regardless of --log-level.
-L, --log-format string Select a logging format. Defaults to 'console'. Valid options are: 'console', 'json', 'dev'. (default "console")
-l, --log-level string Log level when running cargoship. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable terminal color codes in logging and stdout prints.
SEE ALSO
- cargoship - CLI for cargoship installs
cluster
Cluster configuration
is the root object of a cluster configuration document.
Rendered from schema/cluster-v1alpha1-schema.json.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
apiVersion | string | no | identifies the API group and version of this configuration document. One of colonel-byte.dev/v1alpha1, zarf.dev/v1alpha1. | |
kind | string | yes | identifies the document type. The value must be Cluster, or ZarfCluster on a document written before the move to colonel-byte.dev. One of Cluster, ZarfCluster. | |
metadata | Metadata | yes | holds identifying information for the cluster. | |
spec | Spec | yes | holds the configuration and hosts for the cluster. |
AgeEncryption
records the age recipients a document’s credentials were encrypted to.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
lastModified | string | no | is when cargoship last rewrote the credentials in this document, in RFC 3339. | |
recipients | array of string | no | lists the age public keys cargoship encrypted to, in the order they were given. An SSH key keeps its authorized_keys comment, since that is the part that says whose key it is. |
Config
holds cluster-wide configuration.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
loadbalancer | string | yes | is the hostname clients use to reach the cluster control plane. | |
profiles | map[string]Profiles | no | maps a profile name to host and engine overrides that a host can select. | |
registries | array of Registries | no | lists the container registries the cluster uses. | |
values | object | no | overrides the values the distro package was built with. It is the same nested structure the package ships, addressed by the same dotted paths, and it must still satisfy the package’s values schema. Keys the package does not define are kept, so a cluster can carry values a later package version reads. |
Encryption
records how the credentials in a document were encrypted.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
age | AgeEncryption | no | records the age recipients this document’s credentials were encrypted to. Nothing verifies this list and nothing can: an age header carries no recipient identifier, so editing it changes what the file claims and not a byte of what it holds. |
Files
defines a file to write to a host.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
data | string | no | holds inline content for the file, as an alternative to Source. | |
dst | string | no | is the path on the host where cargoship writes the file. | |
dstDir | string | no | is the directory on the host where cargoship writes the file. | |
group | string | no | identifies the file group on the host. | |
name | string | yes | identifies the file in the cluster configuration. | |
perm | string | no | sets the file mode cargoship applies on the host. | |
src | string | no | is the local path or URL cargoship reads the file from. | |
user | string | no | identifies the file owner on the host. |
FirewallConfig
is the backend-neutral firewall configuration for a node.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
rules | array of FirewallRule | no | lists the firewall rules cargoship applies to the node. |
FirewallPolicyConfig
configures a firewalld policy that opens ports from one zone to another.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
egress | FirewallZone | yes | is the zone the policy allows traffic to. | |
ingress | FirewallZone | yes | is the zone the policy allows traffic from. | |
ports | array of FirewallPort | no | lists the ports this policy allows. | |
target | string | yes | is the action taken on traffic that matches the policy. One of CONTINUE, ACCEPT, REJECT, DROP. |
FirewallPort
defines a port allowed through the firewalld policy.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
port | string or integer | yes | is the port number, or port range, cargoship opens. | |
protocol | string | yes | is the type of allowed traffic. One of tcp, udp, sctp, dccp. |
FirewallRule
is a single backend-neutral firewall rule.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
action | string | yes | is what cargoship does with traffic that matches the rule. One of allow, deny, reject. | |
destination | string | no | is the address or CIDR the traffic goes to. | |
direction | string | no | in | selects which traffic the rule matches. It defaults to in. One of in, out, forward. |
egress | string | no | names where forward traffic leaves. It is a zone on firewalld hosts and an interface on ufw hosts. It applies to forward rules only. | |
ingress | string | no | names where forward traffic enters. It is a zone on firewalld hosts and an interface on ufw hosts. It applies to forward rules only. | |
name | string | no | identifies the rule. Cargoship uses it to name the artifacts it writes on the node, so it must be unique within a host. Cargoship generates one when it is empty. | |
port | string or integer | no | is the port number, or inclusive port range, the rule matches. | |
protocol | string | no | is the type of traffic the rule matches. One of tcp, udp, sctp, dccp. | |
source | string | no | is the address or CIDR the traffic comes from. |
FirewallZone
is the name of either the ingress or egress zone for a policy.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
name | string | yes | identifies the firewalld zone. |
Host
is a remote connection to a node
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
engine | HostEngine | no | holds the node label and taint overrides for this node. | |
environment | map[string]string | no | maps environment variables cargoship sets on the host. | |
files | array of Files | no | lists files cargoship uploads to the host. | |
host | HostConfig | no | holds the host-level configuration overrides for this node. | |
hostname | string | no | overrides the discovered name of the node. | |
localhost | boolean | no | ||
openSSH | OpenSSHConfig | no | ||
privateAddress | string | no | overrides the discovered private address of the node. | |
privateInterface | string | no | overrides the discovered private interface of the node. | |
profile | string | no | selects a profile by name from the cluster config’s Profiles map. | |
role | string | yes | sets the node role when cargoship adds this host to the cluster. It must be controller or worker. One of controller, worker. | |
ssh | SSHConfig | no | ||
winRM | WinRMConfig | no |
HostConfig
defines the configuration for a specific host, including firewall policies and the ports cargoship opens on the node.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
firewall | FirewallConfig | no | holds the backend-neutral firewall rules cargoship renders onto whichever firewall the node runs, firewalld, ufw, or nftables. | |
policy | map[string]FirewallPolicyConfig | no | maps a policy name to a firewalld policy that allows traffic from one interface to another. It is firewalld-only; prefer forward rules under firewall.rules for new configuration. | |
ports | array of HostPort | no | lists the ports and protocols cargoship opens on the node. |
HostEngine
defines configuration options for node-level metadata, specifically Kubernetes node labels and taints applied to a cluster host.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
labels | map[string]string | no | maps Kubernetes node label keys to their values. | |
taints | array of string | no | lists the Kubernetes node taints cargoship applies to the node. |
HostPort
is a port cargoship opens on the public side of the firewall.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
port | string or integer | yes | is the port number, or port range, cargoship opens. | |
protocol | string | yes | is the type of allowed traffic. One of tcp, udp. |
Hosts
is an ordered list of hosts that cargoship manages together.
Metadata
holds identifying information for a cluster.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
encryption | Encryption | no | records how the credentials in this document were encrypted. Cargoship writes it and never reads it back as key material, because an age header names no recipient: this records what was done, not a fact anything can check against the ciphertext beside it. | |
name | string | yes | sets the cluster name. If you allow cargoship to update the kubeconfig, cargoship uses this name there. |
OpenSSHConfig
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
address | string | yes | Address of the remote host | |
configPath | string | no | Path to SSH config file | |
disableMultiplexing | boolean | no | false | Disable SSH connection multiplexing |
keyPath | string | no | Path to SSH private key | |
options | OptionArguments | no | Additional SSH options as ssh_config key-value pairs | |
port | integer | no | Optional SSH port | |
user | string | no | Optional SSH user |
OptionArguments
Profiles
holds the host and engine overrides for one profile.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
concurrency | string or integer | no | limits how many hosts using this profile cargoship processes at once, e.g. draining, upgrading, initializing, or uninstalling. It accepts a fixed count (“1”) or a percentage of the hosts sharing this profile (“25%”). Empty falls back to the phase’s default concurrency. | |
engine | HostEngine | no | holds the node label and taint overrides applied to a host that selects this profile. | |
host | HostConfig | no | holds the configuration overrides applied to a host that selects this profile. |
Registries
holds the credentials and pull proxy for one container registry.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
auth | RegistryAuth | no | holds the credentials for the registry. | |
name | string | yes | identifies the registry. With a proxy it is the registry pulls are redirected away from; without one it is the registry the credentials authenticate to. | |
proxy | RegistryProxy | no | holds the pull redirect settings for the registry. Omit it to configure credentials or TLS for the registry without redirecting pulls away from it. | |
tls | RegistryTLS | no | configures TLS verification and certificates for the registry endpoint. |
RegistryAuth
holds the credentials for a container registry.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
pass | string | no | is the login secret for the remote registry. Sensitive. Treat this value as a secret: it is a credential rather than a path to one. | |
token | string | no | authenticates to the remote registry instead of a username and password. Sensitive. Treat this value as a secret: it is a credential rather than a path to one. | |
user | string | no | is the login name for the remote registry. Sensitive. Treat this value as a secret: it is a credential rather than a path to one. |
RegistryProxy
redirects pulls for a registry to a different URL.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
rewrite | map[string]string | no | maps a regex pattern to a replacement, transforming the image name (not the tag) before it is pulled from this mirror. See https://docs.rke2.io/install/private_registry#rewrites | |
url | string | yes | is the registry address the engine pulls from instead of the original registry. |
RegistryTLS
holds the TLS connection settings for a container registry.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
ca | string | no | is the PEM-encoded CA certificate used to verify the registry certificate. Cargoship writes it to a file on each host and points the engine at that path, so the certificate travels with the cluster configuration instead of having to be distributed separately. Use CAFile instead when the certificate is already on the hosts. It may also be given as an Ansible Vault-encrypted string, which cargoship decrypts at apply time. | |
caFile | string | no | is the path on the host to the CA bundle used to verify the registry certificate. | |
certFile | string | no | is the path on the host to the client certificate used to authenticate to the registry. | |
insecureSkipVerify | boolean | no | disables TLS certificate verification. | |
keyFile | string | no | is the path on the host to the client private key used to authenticate to the registry. |
SSHConfig
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
address | string | yes | Address of the remote host (IP or hostname) | |
bastion | SSHConfig | no | Optional bastion host | |
ignoreSshConfig | boolean | no | Do not read ~/.ssh/config or the system ssh_config for this host | |
keyPath | string | no | Path to SSH private key | |
options | OptionArguments | no | Additional SSH options as ssh_config key-value pairs | |
port | integer | no | 22.0 | SSH port (default 22) |
user | string | no | root | User to log in as |
Spec
holds the configuration and hosts for a cluster.
WinRMConfig
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
address | string | yes | Address of the remote host | |
bastion | SSHConfig | no | Optional SSH bastion | |
caCertPath | string | no | Path to CA certificate | |
certPath | string | no | Path to client certificate | |
insecure | boolean | no | false | Accept invalid TLS certificates |
keyPath | string | no | Path to client key | |
password | string | no | Password for WinRM authentication | |
port | integer | no | WinRM port (default 5985; 5986 when useHTTPS is true) | |
tlsServerName | string | no | TLS server name override | |
useHttps | boolean | no | false | Use HTTPS for WinRM |
useNtlm | boolean | no | false | Use NTLM authentication |
user | string | no | Administrator | User to authenticate as |
config
Cargoship configuration file
holds the values for the ., or root, section of the config file
Rendered from schema/config-distro-schema.json.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
age | AgeOptions | no | are the age keys used to encrypt and decrypt registry credentials | |
architecture | string | no | the CPU architecture to use for OCI operations | |
distro | DistroOptions | no | are various options used by the command | |
log_file | boolean | no | enables always writing a full-verbosity debug log to a file, regardless of LogLevel | |
log_format | string | no | console | how the logs are displayed well running One of console, json, dev. |
log_level | string | no | info | the level of logs that will be displayed One of warn, info, debug, trace. |
no_color | boolean | no | whether to disable terminal color codes in logging and stdout prints | |
timeout | string | no | 20m | the longest we will run long ran tasks before failing |
tmp_dir | string | no | /tmp | the directory where we store stuff before deleting them |
zarf_cache | string | no | is the folder where cargoship caches what it fetches: oras artifacts, and the release assets the example generation targets read |
AgeOptions
holds the values for the .age section of the config file, the age alternative to the Ansible Vault password given by –vault-password-file.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
identity_files | array of string | no | are paths to identity files, each holding either the age private keys that decrypt registry credentials or an SSH private key. A leading ~ is expanded to the home directory of the user running cargoship. Equivalent to repeating –age-identity-file. | |
recipients | array of string | no | are the public keys registry credentials are encrypted to, each either an age recipient or an SSH public key. Equivalent to repeating –age-recipient. | |
recipients_files | array of string | no | are paths to files holding public keys, one per line, with # comments allowed, mixing age recipients and SSH public keys freely. An authorized_keys file works as it is. A leading ~ is expanded to the home directory of the user running cargoship. Equivalent to repeating –age-recipients-file. |
DistroCreateOptions
holds the values for the .distro.create section of the config file
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
file_override | FileOverrideMap | no | maps a source URL prefix to the location cargoship downloads from instead, for example {“https://rpm.rancher.io”: “https://mirror.example.com/rpm-rancher”}. The value may also be a local directory, for builds from pre-staged assets. | |
registry_override | RegistryOverrideMap | no | maps a source registry to the registry cargoship uses instead when pulling images, for example {“docker.io”: “mirror.example.com”} |
DistroOptions
holds the values for the .distro section of the config file
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
allow_downgrade | boolean | no | whether a run continues when a host already runs an engine version newer than the one the package carries | |
allow_unmanaged_nodes | boolean | no | whether an apply continues when the cluster holds a node that no host in the config accounts for | |
concurrency | integer | no | how many nodes we will try to interact with at a time, 0 means that all nodes will be done at once | |
create | DistroCreateOptions | no | are options used by the create subcommand | |
firewall | boolean | no | true | whether we will update the host firewall |
hosts | boolean | no | true | whether we will update the etc host file |
kubeconfig | string | no | path of the kubeconfig file the admin creds are merged into, the standard location – KUBECONFIG, else ~/.kube/config – when empty | |
label_nodes | boolean | no | true | whether we will check and add the node-role.kubernetes.io/<profile> label on nodes |
oci_concurrency | integer | no | is how many concurrent oci artifacts that will be pushed at a time | |
output | string | no | the folder that we will create the distro tar balls in | |
public_key | string | no | path to public key file for validating signed packages | |
publish | DistroPublishOptions | no | are options used by the publish subcommand | |
receipt_path | string | no | is where the lifecycle commands – apply, prepare, reset and engine-config-sync – write the structured JSON record of a run, empty for no receipt. Equivalent to –receipt-path, and templated the same way: {command} expands to the command that ran and {timestamp} to when it started, so one configured value gives every run its own file rather than overwriting the last. | |
retry | integer | no | how many times an operation is retried before the run fails, including a package push to an OCI registry – it is what –retries defaults to on publish and sign | |
type | string | no | of distro we are interacting with One of rke2, k3s. | |
update_kubeconfig | boolean | no | true | whether we will update a kubeconfig file with the admin creds for the cluster |
verify | string | no | the Cargoship package signature One of never, if-possible, always. | |
worker_concurrency | string or integer | no | number of worker nodes that will be upgraded at once, as a fixed count (“5”) or a percentage of the batch (“25%”) |
DistroPublishOptions
holds the values for the .distro.publish section of the config file
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
signing_key | string | no | is the private key used to sign, or re-sign, the package: a local file path, or env://ENV_VAR to read the key from an environment variable. Cloud KMS references are not supported – they cannot be reached from an air-gapped management node. | |
signing_key_password | string | no | the password for the private key used for signing Sensitive. Treat this value as a secret: it is a credential rather than a path to one. |
FileOverrideMap
maps a source URL prefix to the location cargoship downloads from instead.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
https://download.docker.com | string | no | ||
https://get.k3s.io | string | no | ||
https://github.com | string | no | ||
https://pkgs.k8s.io | string | no | ||
https://rpm.rancher.io | string | no |
RegistryOverrideMap
maps a source registry to the registry cargoship uses instead.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
docker.io | string | no | ||
gcr.io | string | no | ||
ghcr.io | string | no | ||
public.ecr.aws | string | no | ||
quay.io | string | no | ||
registry.k8s.io | string | no |
distro
Distro package
is the root object of a distro package configuration document.
Rendered from schema/distro-v1alpha1-schema.json.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
apiVersion | string | no | identifies the API group and version of this configuration document. One of colonel-byte.dev/v1alpha1, zarf.dev/v1alpha1. | |
build | BuildData | no | holds information recorded when the package was built. | |
kind | string | yes | identifies the document type. The value must be Distro, or ZarfDistro on a document written before the move to colonel-byte.dev. One of Distro, ZarfDistro. | |
metadata | Metadata | yes | holds identifying information for the distro package. | |
spec | Spec | yes | holds the configuration for the distro package. |
Actions
defines the actions cargoship runs during specific phases of building the distro package.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
onCreate | ZarfComponentActionSet | no | lists the actions cargoship runs when it creates the package. |
Arches
BinarySelector
selects which files to upload to a host, based on host role and install method.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
arch | Arches | no | lists the CPU architectures this file applies to. An empty list means every architecture the package targets. | |
package | string | no | selects which install method receives this file: rpm, apt, or binary. One of rpm, apt, binary. | |
profile | string | no | selects which host role receives this file: worker or controller. One of worker, controller. | |
roles | array of string | no | lists the host roles this file applies to. | |
when | string | no | is an optional values path (e.g. “.tools.etcdctl.enabled”) gating whether this file is uploaded. |
BuildData
holds information recorded when the package was built.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
architecture | string | no | is the CPU architecture used to build the package. Populated only when the package targets a single architecture. One of amd64, arm64, riscv64. | |
architectures | Arches | no | lists the CPU architectures used to build the package. | |
fileOverrides | map[string]string | no | maps each source URL prefix to the mirror or local directory files were downloaded from instead. Like RegistryOverrides, this is the configuration the build was given. For what each override actually did, see FileSources. | |
fileSources | array of FileSource | no | records every file whose source an override redirected: what the definition declared, where the bytes were read from instead, and the checksum they were verified against. It is empty for a build that used no overrides, so a package built without them is unchanged. Entries are in the order the files were staged, which is the order they appear in the definition, so a reproducible build still produces identical output. | |
provenanceFiles | array of string | no | lists files in the package that checksums.txt does not cover. These are files added after cargoship generates checksums, for example signature files. The signed distro.yaml authenticates this list. | |
registryOverrides | map[string]string | no | maps each original registry to the registry actually used to build the package. | |
reproducible | boolean | no | indicates Build.Timestamp was pinned to a fixed value (config.Timestamp) instead of the actual build time, so identical package inputs produce byte-identical output. | |
signed | boolean | no | indicates whether the package was signed. A nil value means the signing status was not recorded. | |
timestamp | string | no | is the time the package was created. | |
version | string | no | records the distro version used to build the package. |
Config
holds the configuration for the distro engine.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
engine | object | no | holds configuration passed through to the distro engine. | |
files | Files | no | lists files that cargoship writes to every host, no matter which install method it uses. | |
imageConfig | ImageConfig | no | holds settings for the images bundled with the package. | |
manifests | array of string | no | lists on-host paths – each one the Target of a Files entry – that cargoship kubectl-applies, in order, from the leader once it bootstraps. Delivery is already solved by Files; this is only the marker that says “and apply this one,” e.g. for a CNI a package declares. A distro with no apply mechanism (rke2/k3s use HelmChartConfig instead) ignores it. | |
os | OS | no | holds settings applied to the host operating system. |
File
defines a file shared by the distro and cluster APIs.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
data | string | no | holds the inline file content. | |
executable | boolean | no | sets executable permissions on the file when true. On a host running fapolicyd, the prepare phase also writes an allow rule for the file’s target path, so a package’s own executables are not denied by policy. | |
extractPath | string | no | is the path cargoship extracts from a tarball. | |
group | string | no | is the group that owns the file. | |
isDirectory | boolean | no | indicates that Target is a directory. Cargoship uses this mainly for image uploads. | |
name | string | no | identifies the file entry. It is not the file name on disk. | |
perm | any | no | sets the permissions cargoship applies to the file. It accepts an octal string or an integer, for example “0644” or 644. | |
selector | BinarySelector | no | controls whether cargoship uploads this file to a given node. | |
shasum | string | no | verifies the file when cargoship sources the distro package. | |
source | string | yes | is the path where cargoship finds the file when it creates the package. | |
symlinks | array of string | no | lists the symlinks cargoship creates that point to Target. | |
target | string | yes | is the path on the remote host where cargoship writes the file. | |
template | boolean | no | renders the file’s contents as a Go template against the package’s values before cargoship uploads it. It defaults to false, because most files are binaries or archives that a template pass would corrupt. | |
user | string | no | is the user that owns the file. |
FileSource
records where one file in the package was actually read from, for a file whose declared source a –file-override redirected.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
declared | string | yes | is the source the distro definition named. | |
override | string | yes | is the Source prefix of the override that matched, which is the one that won when several could have. | |
path | string | yes | is where the file was staged inside the package, for example “files/0/k3s”. It identifies the entry even when several files share a declared source. | |
resolved | string | yes | is where cargoship read the bytes from instead: a mirror URL, or a path on the build host. | |
shasum | string | yes | is the checksum the bytes were verified against. An override is refused for a file that declares none, so this is never empty. |
Files
is an ordered list of files.
ImageConfig
holds settings for the images cargoship writes to a host.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
compression | string | no | none | sets the compression format for the image tarballs. One of none, gz, zstd. |
images | array of string | no | lists the offline images required by the package. | |
path | string | no | is the upload destination for the image tarballs. |
Mapping
Metadata
holds identifying information for a distro package.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
aggregateChecksum | string | no | is the checksum of the checksums.txt file, which lists the checksum for every layer in the package. | |
annotations | map[string]string | no | holds key-value pairs added to the OCI manifest. | |
architecture | string | no | amd64 | is the CPU architecture this distro package targets. Use Architectures to target more than one. One of amd64, arm64, riscv64. |
architectures | Arches | no | lists the CPU architectures this distro package targets. It supersedes Architecture, which stays valid for a package targeting a single architecture. | |
athors | string | no | sets the OCI annotation for the contact details of the people or organization responsible for the image. | |
description | string | no | explains what this distro package does. | |
documentation | string | no | sets the OCI annotation for the URL to the image documentation. | |
name | string | yes | identifies the distro package. | |
source | string | no | sets the OCI annotation for the URL to the image source code. | |
uncompressed | boolean | no | disables compression for this package when true. | |
url | string | no | sets the OCI annotation for more information about the image. | |
vendor | string | no | sets the OCI annotation for the name of the organization or individual that distributes the image. | |
version | string | no | is the distro version cargoship installs. We recommend matching it to the Kubernetes version you install. |
OS
holds settings applied to a host.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
env | map[string]string | no | maps environment variables cargoship sets on the host. | |
fapolicyd | string | no | holds the fapolicyd config file contents cargoship writes to a host. | |
files | Files | no | lists files cargoship uploads to a host. | |
kernel | array of string | no | lists the kernel modules cargoship enables on the host. | |
selinux | SELinux | no | holds the custom SELinux policy cargoship applies to a host. | |
sysctl | map[string]string or number | no | maps sysctl keys to the values cargoship applies to a host. |
SELinux
holds the custom SELinux policy cargoship applies to hosts running SELinux in enforcing mode.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
booleans | map[string]boolean | no | maps SELinux boolean names to the values cargoship sets persistently on a host. | |
fileContexts | array of SELinuxFileContext | no | lists the file context mappings cargoship adds on a host. | |
modules | array of SELinuxModule | no | lists the CIL policy modules cargoship installs on a host. |
SELinuxFileContext
holds a single file context mapping cargoship adds with semanage.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
fileType | string | no | all | restricts the mapping to one kind of filesystem object. One of all, file, dir, symlink, pipe, socket, block, char. |
path | string | yes | holds the path regular expression the mapping applies to. | |
type | string | yes | names the SELinux type the matching paths are labeled with. |
SELinuxModule
holds a single CIL policy module cargoship installs with semodule.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
cil | string | yes | holds the Common Intermediate Language policy source semodule installs. | |
name | string | yes | identifies the module to semodule, and names the file cargoship writes to the host. | |
priority | integer | no | 400.0 | holds the semodule priority the module installs at. |
SetValue
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
key | string | no | ||
type | string | no | ||
value | any | no | Accepts any value; cargoship does not constrain its shape here. |
Shell
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
darwin | string | no | (default ‘sh’) Indicates a preference for the shell to use on macOS systems | |
linux | string | no | (default ‘sh’) Indicates a preference for the shell to use on Linux systems | |
windows | string | no | (default ‘powershell’) Indicates a preference for the shell to use on Windows systems (note that choosing ‘cmd’ will turn off migrations like touch -> New-Item) |
Spec
holds the configuration for a distro package.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
actions | Actions | no | defines the actions cargoship runs while building the package. | |
config | Config | yes | holds the distro engine configuration. | |
type | string | yes | selects the distro engine: rke2, k3s, or upstream. One of rke2, k3s, upstream. | |
values | Values | no | holds the default values the package is built with, and the schema they must satisfy. | |
version | string | yes | is the version of the distro engine. |
ValueMapping
projects one value onto one place in the engine configuration.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
source | string | yes | is the dotted path to read from the resolved values, e.g. .cilium.encryption.enabled | |
target | string | yes | is the dotted path to write, relative to spec.config.engine, e.g. .manifest.rke2-cilium.encryption.enabled |
Values
declares the values a package ships with.
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
files | array of string | no | lists the YAML values files cargoship merges, in order, to build the package’s default values. Each entry is a path relative to the package definition, an absolute path, or a URL. A later file overrides an earlier one, key by key. | |
mappings | array of ValueMapping | no | project values onto the engine configuration, so that one value a cluster sets reaches wherever the distro needs it. Each mapping reads the source path out of the resolved values and writes it to the target path, which is relative to spec.config.engine. A source the values do not define is left alone, so the package’s own engine configuration stands as the default. | |
schema | string | no | is the JSON Schema document the merged values must satisfy. Cargoship checks the values against it when it builds the package, and refuses to build when they do not match. The schema must not use $ref. |
Variable
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
autoIndent | boolean | no | ||
name | string | yes | ||
pattern | string | no | ||
sensitive | boolean | no | ||
type | string | no | One of raw, file. |
ZarfComponentAction
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
cmd | string | no | ||
description | string | no | ||
dir | string | no | ||
env | array of string | no | ||
maxRetries | integer | no | ||
maxTotalSeconds | integer | no | ||
mute | boolean | no | ||
setValues | array of SetValue | no | ||
setVariable | string | no | ||
setVariables | array of Variable | no | ||
shell | Shell | no | ||
template | boolean | no | ||
wait | ZarfComponentActionWait | no |
ZarfComponentActionDefaults
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
dir | string | no | ||
env | array of string | no | ||
maxRetries | integer | no | ||
maxTotalSeconds | integer | no | ||
mute | boolean | no | ||
shell | Shell | no |
ZarfComponentActionSet
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
after | array of ZarfComponentAction | no | ||
before | array of ZarfComponentAction | no | ||
defaults | ZarfComponentActionDefaults | no | ||
onFailure | array of ZarfComponentAction | no | ||
onSuccess | array of ZarfComponentAction | no |
ZarfComponentActionWait
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
cluster | ZarfComponentActionWaitCluster | no | ||
network | ZarfComponentActionWaitNetwork | no |
ZarfComponentActionWaitCluster
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
condition | string | no | ||
kind | string | yes | ||
name | string | yes | ||
namespace | string | no |
ZarfComponentActionWaitNetwork
| Property | Type | Required | Default | Description |
|---|---|---|---|---|
address | string | yes | ||
code | integer | no | ||
protocol | string | yes | One of tcp, http, https. |
The colonel_byte.cargoship collection
Cargoship runs inside a playbook as an Ansible collection. The cargoship binary is the module: it is installed as a set of symlinks named cargoship_<action>, and a symlink’s name selects the action. There is no separate module package and nothing to install on the managed nodes.
Ansible supplies the inventory and nothing else. It does not connect to the fleet, gather facts on it, or run a task per host. Every task in this collection runs on one management node outside the cluster – the node the package and images were staged onto – and cargoship opens every SSH connection itself from there. See choice-ansible-module for why the work is split that way.
That boundary is what the collection is for, and it is the whole of what the collection is for. A role that configures fleet hosts directly belongs in the playbook that consumes this collection, not in it.
What is here
| Page | What it covers |
|---|---|
| Modules | The five execution primitives, one per fleet action. |
| Roles | cluster, the shorter way to write the same task. |
| Module guide | Installing the collection, check mode, and what the result means. |
| Inventory guide | The inventory document the modules take. |
Which to reach for
Reach for the cluster role first. It picks the module from cargoship_action, assembles the inventory parameter, and sets run_once, delegate_to, and no_log for you.
Reach for a module directly when you need control the role does not give you. Then you must set run_once: true yourself, or the cluster is converged once per host in the fleet, and delegate_to and no_log: true with it.
Reference pages are generated
Every module_*.md and role_*.md page here is generated from the collection itself – a role’s interface from its meta/argument_specs.yml, a module’s from the DOCUMENTATION and EXAMPLES blocks in its action plugin – by mage generate:document. Edit the collection and regenerate; an edit to a generated page does not survive the next commit.
This page, modules.md, and roles.md are written by hand. See ansible/colonel_byte/cargoship/AGENTS.md for the contract.
The other collection here
Modules
Five modules ship, one per fleet action. Package creation is not among them – it builds an artifact from a definition in a repository, which belongs in a pipeline rather than in a convergence run.
| Module | What it does |
|---|---|
cargoship_apply | Installs or converges the whole cluster. The widest action there is. |
cargoship_prepare | Stages a package onto the fleet and readies the hosts. |
cargoship_engine_config_sync | Converges engine configuration across the fleet. |
cargoship_reset | Removes the cluster from the fleet. |
cargoship_kube_config | Fetches the cluster kubeconfig onto the management node. |
The surfaces differ because the commands do
cargoship prepare reads no encrypted value, so it takes no vault_password_file. cargoship reset installs nothing, so it takes no package and no values. cargoship engine-config-sync does not update the hosts themselves, so it takes none of the three host switches. A parameter offered to the wrong module is an error naming it, rather than a flag quietly dropped.
Each module’s page lists its full parameter set and the flag each parameter renders. A parameter that renders no flag is listed as None: it is either the command’s positional argument, or something consumed before the command line is built.
What every task needs
A parameter left unset is left unset: the module passes no flag for it, so whatever your cargoship configuration file sets still applies. That is why label_nodes: false and omitting label_nodes are different.
Set run_once: true, delegate_to, and no_log: true on every direct module call. The cluster role sets all three for you. Key material is always named by a path and never given by value, because Ansible writes a module’s parameters into a file on disk.
The module guide covers installing the collection, check mode, what changed is worth, and the shape of the result.
apply
cargoship_apply
Install or converge a Kubernetes cluster across the fleet.
Write it as colonel_byte.cargoship.cargoship_apply. It runs on the management node; nothing is installed on the fleet.
Added in colonel_byte.cargoship 0.26.0.
Runs cargoship apply against the fleet described by an Ansible inventory. It is the widest action cargoship has.
The module runs on one management node. Ansible does not connect to the fleet; cargoship opens every SSH connection itself from there.
Parameters
| Parameter | Type | Required | Flag | Description |
|---|---|---|---|---|
inventory | dict | yes | None | The resolved Ansible inventory the module translates into a ZarfCluster document: groups, hostvars, and a cluster block. The action plugin fills groups and hostvars in from the play, so a task normally sets only cluster. Renders no flag of its own. The translated document is passed to the command as --config. |
parameters | dict | no | None | Further module parameters, merged into the ones given directly. It is how the cluster role passes a caller-supplied set without templating a whole task’s arguments. A parameter given both directly and inside parameters is an error naming it. The action plugin consumes this key; the module never sees it. |
package | str | yes | None | The distro package to install. The positional argument of the command, so it renders no flag. |
concurrency | int | no | --concurrency | How many hosts to work on at once. |
work_concurrency | str | no | --work-concurrency | How much work to run concurrently on each host. A count or a percentage, as a string. |
hosts | bool | no | --hosts | Whether to manage /etc/hosts entries on the fleet. |
firewall | bool | no | --firewall | Whether to manage the host firewall on the fleet. |
label_nodes | bool | no | --label-nodes | Whether to apply cargoship’s role labels to the cluster nodes. |
allow_unmanaged_nodes | bool | no | --allow-unmanaged-nodes | Whether to tolerate nodes in the cluster that the inventory does not describe. |
update_kubeconfig | bool | no | --update-kubeconfig | Whether to merge the cluster into the local kubeconfig after the run. |
kubeconfig | str | no | --kubeconfig | The kubeconfig to read or write. |
vault_password_file | str | no | --vault-password-file | A file holding the vault password used to decrypt encrypted inventory values. Key material is named by a path, never given by value: a module’s parameters are written into a file on disk whose mode nobody chose. |
age_identity_file | list of str | no | --age-identity-file | Age identity files, or SSH private keys, used to decrypt registry credentials. One flag per entry. |
values | list of str | no | --values | Package values files to apply, one flag per entry. |
timeout | str | no | --timeout | How long the run may take before it is abandoned, as a Go duration such as 45m. |
public_key | str | no | --key | The public key the package signature is verified against. |
verify | str | no | --verify | When to verify the package signature. Verification is key-based. A Cosign key pair is cargoship’s only signing method; keyless (Fulcio/OIDC) verification was removed because it needs network an air-gapped management node does not have. One of never, if-possible, always. |
inventory_path | str | no | --config | Where to write the generated ZarfCluster document. A private temporary file when unset, removed after a run that succeeded. A run that fails leaves the document on disk either way, because it is the first thing to look at when a failure reads like the wrong cluster. |
age_recipient | list of str | no | None | Age public keys used to encrypt registry credentials written into the generated inventory document. Consumed while that document is written, so it renders no flag. |
age_recipients_file | list of str | no | None | Files holding age public keys, read for the same purpose as age_recipient. |
log_level | str | no | --log-level | Cargoship’s log level. Defaults to debug when the play runs with -v. |
log_format | str | no | --log-format | Cargoship’s log format. |
log_file | bool | no | --log-file | Whether to always write a full-verbosity debug log to a file on the management node, regardless of log level. |
Notes
- Set
run_once: trueanddelegate_toon the task. Cargoship converges the whole fleet in a single run, so a play over the fleet’s own inventory would otherwise converge it once per host. - Set
no_log: trueon the task. A binary module has no per-parameterno_log, so the task-level setting is what suppresses the arguments and the result. - A parameter left unset renders no flag, so whatever the cargoship configuration file sets still applies. That is why
label_nodes: falseand omittinglabel_nodesdiffer.
Example
- name: Install the cluster
colonel_byte.cargoship.cargoship_apply:
package: /srv/staging/rke2-1.31.tar.zst
inventory_path: /srv/staging/generated-inventory.yaml
inventory:
cluster:
name: bubbles
loadbalancer: bubbles-kc.test.com
vault_password_file: /srv/staging/vault-pass
timeout: 45m
delegate_to: localhost
run_once: true
no_log: true
engine_config_sync
cargoship_engine_config_sync
Converge engine configuration across the fleet.
Write it as colonel_byte.cargoship.cargoship_engine_config_sync. It runs on the management node; nothing is installed on the fleet.
Added in colonel_byte.cargoship 0.26.0.
Runs cargoship engine-config-sync against the fleet described by an Ansible inventory. It is the most Ansible-shaped thing cargoship does, and the one action whose phases report whether they changed anything.
It does not update the hosts themselves, so it takes none of the three host switches.
Parameters
| Parameter | Type | Required | Flag | Description |
|---|---|---|---|---|
inventory | dict | yes | None | The resolved Ansible inventory the module translates into a ZarfCluster document: groups, hostvars, and a cluster block. The action plugin fills groups and hostvars in from the play, so a task normally sets only cluster. Renders no flag of its own. The translated document is passed to the command as --config. |
parameters | dict | no | None | Further module parameters, merged into the ones given directly. It is how the cluster role passes a caller-supplied set without templating a whole task’s arguments. A parameter given both directly and inside parameters is an error naming it. The action plugin consumes this key; the module never sees it. |
package | str | yes | None | The distro package to synchronise from. The positional argument of the command, so it renders no flag. |
concurrency | int | no | --concurrency | How many hosts to work on at once. |
work_concurrency | str | no | --work-concurrency | How much work to run concurrently on each host. A count or a percentage, as a string. |
label_nodes | bool | no | --label-nodes | Whether to apply cargoship’s role labels to the cluster nodes. |
update_kubeconfig | bool | no | --update-kubeconfig | Whether to merge the cluster into the local kubeconfig after the run. |
kubeconfig | str | no | --kubeconfig | The kubeconfig to read or write. |
vault_password_file | str | no | --vault-password-file | A file holding the vault password used to decrypt encrypted inventory values. Key material is named by a path, never given by value: a module’s parameters are written into a file on disk whose mode nobody chose. |
age_identity_file | list of str | no | --age-identity-file | Age identity files, or SSH private keys, used to decrypt registry credentials. One flag per entry. |
values | list of str | no | --values | Package values files to apply, one flag per entry. |
timeout | str | no | --timeout | How long the run may take before it is abandoned, as a Go duration such as 45m. |
public_key | str | no | --key | The public key the package signature is verified against. |
verify | str | no | --verify | When to verify the package signature. Verification is key-based. A Cosign key pair is cargoship’s only signing method; keyless (Fulcio/OIDC) verification was removed because it needs network an air-gapped management node does not have. One of never, if-possible, always. |
inventory_path | str | no | --config | Where to write the generated ZarfCluster document. A private temporary file when unset, removed after a run that succeeded. A run that fails leaves the document on disk either way, because it is the first thing to look at when a failure reads like the wrong cluster. |
age_recipient | list of str | no | None | Age public keys used to encrypt registry credentials written into the generated inventory document. Consumed while that document is written, so it renders no flag. |
age_recipients_file | list of str | no | None | Files holding age public keys, read for the same purpose as age_recipient. |
log_level | str | no | --log-level | Cargoship’s log level. Defaults to debug when the play runs with -v. |
log_format | str | no | --log-format | Cargoship’s log format. |
log_file | bool | no | --log-file | Whether to always write a full-verbosity debug log to a file on the management node, regardless of log level. |
Notes
- Set
run_once: trueanddelegate_toon the task. Cargoship converges the whole fleet in a single run, so a play over the fleet’s own inventory would otherwise converge it once per host. - Set
no_log: trueon the task. A binary module has no per-parameterno_log, so the task-level setting is what suppresses the arguments and the result. - A parameter left unset renders no flag, so whatever the cargoship configuration file sets still applies. That is why
label_nodes: falseand omittinglabel_nodesdiffer.
Example
- name: Converge engine configuration
colonel_byte.cargoship.cargoship_engine_config_sync:
package: /srv/staging/rke2-1.31.tar.zst
inventory:
cluster:
name: bubbles
loadbalancer: bubbles-kc.test.com
vault_password_file: /srv/staging/vault-pass
delegate_to: localhost
run_once: true
no_log: true
kube_config
cargoship_kube_config
Fetch the cluster kubeconfig onto the management node.
Write it as colonel_byte.cargoship.cargoship_kube_config. It runs on the management node; nothing is installed on the fleet.
Added in colonel_byte.cargoship 0.26.0.
Runs cargoship kube-config against the fleet described by an Ansible inventory.
It is the one module that changes the node the play runs on rather than the fleet.
cargoship kube-config has no dry run, so under --check the module reports skipped: true and runs nothing at all.
Parameters
| Parameter | Type | Required | Flag | Description |
|---|---|---|---|---|
inventory | dict | yes | None | The resolved Ansible inventory the module translates into a ZarfCluster document: groups, hostvars, and a cluster block. The action plugin fills groups and hostvars in from the play, so a task normally sets only cluster. Renders no flag of its own. The translated document is passed to the command as --config. |
parameters | dict | no | None | Further module parameters, merged into the ones given directly. It is how the cluster role passes a caller-supplied set without templating a whole task’s arguments. A parameter given both directly and inside parameters is an error naming it. The action plugin consumes this key; the module never sees it. |
distro | str | no | --distro | The distro the cluster runs. One of k3s, rke2. |
kubeconfig | str | no | --kubeconfig | The kubeconfig to read or write. |
inventory_path | str | no | --config | Where to write the generated ZarfCluster document. A private temporary file when unset, removed after a run that succeeded. A run that fails leaves the document on disk either way, because it is the first thing to look at when a failure reads like the wrong cluster. |
age_recipient | list of str | no | None | Age public keys used to encrypt registry credentials written into the generated inventory document. Consumed while that document is written, so it renders no flag. |
age_recipients_file | list of str | no | None | Files holding age public keys, read for the same purpose as age_recipient. |
log_level | str | no | --log-level | Cargoship’s log level. Defaults to debug when the play runs with -v. |
log_format | str | no | --log-format | Cargoship’s log format. |
log_file | bool | no | --log-file | Whether to always write a full-verbosity debug log to a file on the management node, regardless of log level. |
Notes
- Set
run_once: trueanddelegate_toon the task. Cargoship converges the whole fleet in a single run, so a play over the fleet’s own inventory would otherwise converge it once per host. - Set
no_log: trueon the task. A binary module has no per-parameterno_log, so the task-level setting is what suppresses the arguments and the result. - A parameter left unset renders no flag, so whatever the cargoship configuration file sets still applies. That is why
label_nodes: falseand omittinglabel_nodesdiffer.
Example
- name: Fetch the cluster kubeconfig
colonel_byte.cargoship.cargoship_kube_config:
distro: rke2
kubeconfig: /srv/staging/bubbles.kubeconfig
inventory:
cluster:
name: bubbles
loadbalancer: bubbles-kc.test.com
delegate_to: localhost
run_once: true
no_log: true
prepare
cargoship_prepare
Stage a package onto the fleet and ready the hosts.
Write it as colonel_byte.cargoship.cargoship_prepare. It runs on the management node; nothing is installed on the fleet.
Added in colonel_byte.cargoship 0.26.0.
Runs cargoship prepare against the fleet described by an Ansible inventory.
It takes no key material: it neither reads an encrypted value nor writes a kubeconfig, so there is no vault password and no age identity to name.
Parameters
| Parameter | Type | Required | Flag | Description |
|---|---|---|---|---|
inventory | dict | yes | None | The resolved Ansible inventory the module translates into a ZarfCluster document: groups, hostvars, and a cluster block. The action plugin fills groups and hostvars in from the play, so a task normally sets only cluster. Renders no flag of its own. The translated document is passed to the command as --config. |
parameters | dict | no | None | Further module parameters, merged into the ones given directly. It is how the cluster role passes a caller-supplied set without templating a whole task’s arguments. A parameter given both directly and inside parameters is an error naming it. The action plugin consumes this key; the module never sees it. |
package | str | yes | None | The distro package to stage. The positional argument of the command, so it renders no flag. |
concurrency | int | no | --concurrency | How many hosts to work on at once. |
hosts | bool | no | --hosts | Whether to manage /etc/hosts entries on the fleet. |
firewall | bool | no | --firewall | Whether to manage the host firewall on the fleet. |
values | list of str | no | --values | Package values files to apply, one flag per entry. |
timeout | str | no | --timeout | How long the run may take before it is abandoned, as a Go duration such as 45m. |
public_key | str | no | --key | The public key the package signature is verified against. |
verify | str | no | --verify | When to verify the package signature. Verification is key-based. A Cosign key pair is cargoship’s only signing method; keyless (Fulcio/OIDC) verification was removed because it needs network an air-gapped management node does not have. One of never, if-possible, always. |
inventory_path | str | no | --config | Where to write the generated ZarfCluster document. A private temporary file when unset, removed after a run that succeeded. A run that fails leaves the document on disk either way, because it is the first thing to look at when a failure reads like the wrong cluster. |
age_recipient | list of str | no | None | Age public keys used to encrypt registry credentials written into the generated inventory document. Consumed while that document is written, so it renders no flag. |
age_recipients_file | list of str | no | None | Files holding age public keys, read for the same purpose as age_recipient. |
log_level | str | no | --log-level | Cargoship’s log level. Defaults to debug when the play runs with -v. |
log_format | str | no | --log-format | Cargoship’s log format. |
log_file | bool | no | --log-file | Whether to always write a full-verbosity debug log to a file on the management node, regardless of log level. |
Notes
- Set
run_once: trueanddelegate_toon the task. Cargoship converges the whole fleet in a single run, so a play over the fleet’s own inventory would otherwise converge it once per host. - Set
no_log: trueon the task. A binary module has no per-parameterno_log, so the task-level setting is what suppresses the arguments and the result. - A parameter left unset renders no flag, so whatever the cargoship configuration file sets still applies. That is why
label_nodes: falseand omittinglabel_nodesdiffer.
Example
- name: Stage the package onto the fleet
colonel_byte.cargoship.cargoship_prepare:
package: /srv/staging/rke2-1.31.tar.zst
inventory:
cluster:
name: bubbles
loadbalancer: bubbles-kc.test.com
firewall: true
delegate_to: localhost
run_once: true
no_log: true
reset
cargoship_reset
Remove the cluster from the fleet.
Write it as colonel_byte.cargoship.cargoship_reset. It runs on the management node; nothing is installed on the fleet.
Added in colonel_byte.cargoship 0.26.0.
Runs cargoship reset against the fleet described by an Ansible inventory.
There is no package: it removes what is installed, and the distro to remove is named directly rather than read out of one. It installs nothing, so it takes no values and no signature parameters.
Parameters
| Parameter | Type | Required | Flag | Description |
|---|---|---|---|---|
inventory | dict | yes | None | The resolved Ansible inventory the module translates into a ZarfCluster document: groups, hostvars, and a cluster block. The action plugin fills groups and hostvars in from the play, so a task normally sets only cluster. Renders no flag of its own. The translated document is passed to the command as --config. |
parameters | dict | no | None | Further module parameters, merged into the ones given directly. It is how the cluster role passes a caller-supplied set without templating a whole task’s arguments. A parameter given both directly and inside parameters is an error naming it. The action plugin consumes this key; the module never sees it. |
distro | str | no | --distro | The distro to remove. One of k3s, rke2. |
concurrency | int | no | --concurrency | How many hosts to work on at once. |
work_concurrency | str | no | --work-concurrency | How much work to run concurrently on each host. A count or a percentage, as a string. |
inventory_path | str | no | --config | Where to write the generated ZarfCluster document. A private temporary file when unset, removed after a run that succeeded. A run that fails leaves the document on disk either way, because it is the first thing to look at when a failure reads like the wrong cluster. |
age_recipient | list of str | no | None | Age public keys used to encrypt registry credentials written into the generated inventory document. Consumed while that document is written, so it renders no flag. |
age_recipients_file | list of str | no | None | Files holding age public keys, read for the same purpose as age_recipient. |
log_level | str | no | --log-level | Cargoship’s log level. Defaults to debug when the play runs with -v. |
log_format | str | no | --log-format | Cargoship’s log format. |
log_file | bool | no | --log-file | Whether to always write a full-verbosity debug log to a file on the management node, regardless of log level. |
Notes
- Set
run_once: trueanddelegate_toon the task. Cargoship converges the whole fleet in a single run, so a play over the fleet’s own inventory would otherwise converge it once per host. - Set
no_log: trueon the task. A binary module has no per-parameterno_log, so the task-level setting is what suppresses the arguments and the result. - A parameter left unset renders no flag, so whatever the cargoship configuration file sets still applies. That is why
label_nodes: falseand omittinglabel_nodesdiffer.
Example
- name: Remove the cluster
colonel_byte.cargoship.cargoship_reset:
distro: rke2
inventory:
cluster:
name: bubbles
loadbalancer: bubbles-kc.test.com
delegate_to: localhost
run_once: true
no_log: true
Roles
| Role | What it does |
|---|---|
cluster | Runs one cargoship action from an Ansible inventory. |
cluster is the shorter way to write a module call. It picks the module from cargoship_action, assembles the inventory parameter from the play’s groups and hostvars, and sets run_once, delegate_to, and no_log for you.
- name: Install the cluster
ansible.builtin.include_role:
name: colonel_byte.cargoship.cluster
vars:
cargoship_action: apply
cargoship_package: /srv/staging/rke2-1.31.tar.zst
cargoship_cluster:
name: bubbles
loadbalancer: bubbles-kc.test.com
cargoship_args:
vault_password_file: /srv/staging/vault-pass
timeout: 45m
Anything the role does not name a variable for goes in cargoship_args, which is passed to the chosen module verbatim. The module pages list what each module takes.
There is only one
Every role here drives the cargoship binary from the management node. A role that configures fleet hosts directly – writing files on them, installing packages, managing services – does not belong in this collection, however convenient it would be to ship alongside; it belongs in the playbook that consumes the collection. That boundary is what lets the collection declare a controller range and no managed-node floor.
cluster
cluster
Run a cargoship cluster action from an Ansible inventory.
Include it as colonel_byte.cargoship.cluster.
Added in colonel_byte.cargoship 0.26.0.
Runs one cargoship action against the fleet described by the Ansible inventory, on the management node.
Every task carries run_once: true and delegates to cargoship_delegate_to. Cargoship converges the whole fleet in a single run, so a play over the fleet’s own inventory would otherwise converge it once per host.
Ansible supplies the inventory and nothing else. It does not connect to the fleet, and nothing is installed on the managed nodes.
Variables
| Variable | Type | Default | Description |
|---|---|---|---|
cargoship_action | str | apply | The cargoship action to run. It selects which module the role calls. One of apply, prepare, engine_config_sync, reset, kube_config. |
cargoship_package | str | "" | The distro package to install, stage, or synchronise from. Required by every action except reset and kube_config, which install nothing. |
cargoship_cluster | dict | {} | The settings that belong to the cluster rather than to any host. name and loadbalancer are required; profiles, registries, and values are optional and take the shape the inventory schema documents. See the inventory guide for the full document. |
cargoship_role_groups | dict | {} | The mapping from a cargoship role onto the Ansible groups that carry it, for an inventory that does not name its groups controller and worker. Left empty, the default mapping applies. |
cargoship_inventory_path | str | "" | Where to write the generated ZarfCluster document. A private temporary file when empty, removed after a run that succeeded. |
cargoship_age_recipients | list of str | [] | Age public keys used to encrypt registry credentials written to disk in the generated inventory document. |
cargoship_age_identity_files | list of str | [] | Paths to age identity files, or SSH private keys, used to decrypt registry credentials at apply time. |
cargoship_log_file | bool | false | Whether to always write a full-verbosity debug log to a file, regardless of log level. |
cargoship_args | dict | {} | Everything else the chosen module takes, passed through verbatim. See the generated module pages for the parameters of each module. |
cargoship_delegate_to | str | localhost | The node cargoship runs on. It is the management node: the one the package and images were staged onto, which reaches the fleet over SSH. Set this to the inventory name of that node when it is not the Ansible controller. |
cargoship_no_log | bool | true | Whether to suppress task parameter and result logging. Defaults to true to avoid exposing decrypted credentials and fleet inventories in logs. Set to false when debugging locally. |
cargoship_show_result | bool | false | Whether to print cargoship’s own report on the run. The task itself defaults to no_log: true, because a module’s arguments name the fleet; the report holds file paths and phase names and no key material. |
apply
apply phases
With --dry-run, cargoship connects to every host and runs the preflight checks for real, then reports the phases it would have run instead of running them. Each phase below is labelled with what a dry run does with it. A phase is only run when it declares that it is safe to, so a phase added later is reported until someone says otherwise.
The report is not a static list. Every phase still prepares itself and checks whether it has anything to do, and both only read, so work that is already done is filtered out against the live hosts. A phase that could not be assessed, because it reads state an earlier reported phase would have created, is reported as unassessed.
A dry run takes no cluster lock, so it does not block a real run, and it can report state that a concurrent run is already changing. It does not need --confirm.
- Connect to hosts
- Connects to a remote host via
github.com/k0sproject/rig - Dry run: runs, reads only. Connects opens the SSH session to each host and does nothing else. A dry run needs it, because a preflight that never reached a host would report on a cluster it never looked at.
- Connects to a remote host via
- Detect host operating systems
- Gathers information about the remote host, including: OS and OS version
- Dry run: runs, reads only. Reads
/etc/os-releaseand the kernel to pick a configurer for the host. Reporting what each host runs is half of what makes a dry run worth running.
- Gather host facts
- Gathers network related information about the remote host, including: Hostname, Private Address, Private Interface. Will also update the hosts based off the profile if configured in the config file.
- Dry run: runs, reads only. Gather facts about each host by asking for its hostname, private interface and private address. All three are reads, and the rest of the run decides what it would do from them.
- Validate hosts
- Verifying that each node in the cluster has a unique name and private address, that its CPU architecture is one the package carries, and that its firewall rules are usable,
- Dry run: runs, reads only. Validate the hosts is the preflight itself: sudo, unique hostnames and addresses, host architecture, firewall rules and clock skew. A dry run that skipped it would check nothing.
- Gathering facts about the distro installed
- Gathers information relating to the specific distro being installed, including: if the distro is installed, and what version it is running
- Dry run: runs, reads only. Gather distro related facts reads the engine version already on each host. That is what tells an upgrade apart from an install, and what catches a downgrade before a real run starts one.
- Checking for nodes no longer in the config
- Compares the nodes joined to the cluster against the hosts in the config and stops the apply when the cluster holds a node the config does not, since nothing later in an apply removes a node
- Dry run: runs, reads only. Checking for removed nodes lists the nodes joined to the cluster and compares them to the config. It writes nothing, and a dry run is exactly when an operator wants to be told that a host they deleted from the config is still running.
- Acquire exclusive host lock
- Runs a background task that touches a file every 30 seconds on each remote node. This prevents other
cargoshipsfrom making any changes until the lock file has not been touched for over a minute - Dry run: reported, not run
- Runs a background task that touches a file every 30 seconds on each remote node. This prevents other
- Prepare hosts
- Updates the remote nodes; environment variables and sysctl
- Dry run: reported, not run
- Prepare hosts - Enterprise Linux support
- Installs container-selinux on systems that have SELinux enabled on them
- Dry run: reported, not run
- Prepare hosts - Enterprise Linux support - Fapolicyd
- Writes the distro-supplied FAPolicy rules to /etc/fapolicyd/rules.d/31-cargoship.rules and executable rules to /etc/fapolicyd/rules.d/32-cargoship-executable.rules
- Dry run: reported, not run
- Prepare hosts - Enterprise Linux support - SELinux policy
- Installs the policy modules, booleans and file contexts from spec.config.os.selinux on hosts running SELinux in enforcing mode, using semodule, setsebool and semanage fcontext
- Dry run: reported, not run
- Updating hosts file for clusters nodes
- If enabled, then this will modify the
/etc/hostsfile on the remote nodes with the fully qualified domain name for each node in the cluster - Dry run: reported, not run
- If enabled, then this will modify the
- Updating hosts firewall
- If enabled, this configures the firewall on each node that runs one, firewalld, ufw, or nftables, preferring the front end the node’s OS ships. It trusts every other node in the cluster along with the engine’s pod and service CIDRs, opens the ports in the
.host.portssection, and applies the rules in the.host.firewall.rulessection - Dry run: reported, not run
- If enabled, this configures the firewall on each node that runs one, firewalld, ufw, or nftables, preferring the front end the node’s OS ships. It trusts every other node in the cluster along with the engine’s pod and service CIDRs, opens the ports in the
- Upload files to hosts
- Uploads the distro agnostic files to each remote node
- Dry run: reported, not run
- Upload files to hosts – RPM
- If the remote node is an Enterprise Linux and the Distro package includes any files for those systems
- Dry run: reported, not run
- Upload files to hosts – APT
- If the remote node is a Debian based Operating System and the Distro package includes any files for those systems
- Dry run: reported, not run
- Upload files to hosts – Binaries
- Catch-all phase for when the combination of Operating System and Distro has no other install method
- Dry run: reported, not run
- Import images
- For a distro whose engine does not import uploaded image tarballs on its own, imports them into the engine’s image store
- Dry run: reported, not run
- Configure engine
- Runs distro specific operations
- Dry run: reported, not run
- Initialize Controller
- If the remote node does not have a running controller service, and is a controller, install the engine and start each service sequentially
- Dry run: reported, not run
- Apply manifests
- For a distro whose package declares raw manifests – a CNI, typically – kubectl applies each one from the leader
- Dry run: reported, not run
- Initialize Worker
- If the remote node does not have a running worker service, and is not a controller, install the engine and start each service by the set concurrency limit
- Dry run: reported, not run
- Upgrade Controller
- If the remote node is a controller and is running an older version of the engine, drain the node, stop the service, upgrade the engine, start the service, and uncordon the node sequentially
- Dry run: reported, not run
- Upgrade Worker
- If the remote node is a worker and is running an older version of the engine, drain the node, stop the service, upgrade the engine, start the service, and uncordon the node by the set concurrency limit
- Dry run: reported, not run
- Install staged files
- Moves staged engine files into place on any host the initialize and upgrade phases did not install
- Dry run: reported, not run
- Sync Registry Config Controller
- If the remote node is a controller and its engine config (registries/audit/pss) has drifted from the desired state, drain the node, stop the service, write the new config, start the service, and uncordon the node sequentially. Chart values are written in place instead, since the engine reconciles them without a restart
- Dry run: reported, not run
- Sync Registry Config Worker
- If the remote node is a worker and its engine config (registries/audit/pss) has drifted from the desired state, drain the node, stop the service, write the new config, start the service, and uncordon the node by the set concurrency limit
- Dry run: reported, not run
- Updating kubeconfig file with the current cluster
- If enabled, this will update the local kubeconfig with the admin creds for the current distro
- Dry run: reported, not run
- Labeling nodes with their profile group
- If enabled, this checks each node’s
node-role.kubernetes.io/<profile>label and adds it, set to “true”, when missing or set to anything else - Dry run: reported, not run
- If enabled, this checks each node’s
- Release exclusive host lock
- Deletes the lock file from each node, allowing other
cargoshipsto run - Dry run: reported, not run
- Deletes the lock file from each node, allowing other
- Disconnect from hosts
- Deletes any lingering temp files and disconnects from the remote node
- Dry run: runs its own dry-run path
engine-config-sync
engine-config-sync phases
With --dry-run, cargoship connects to every host and runs the preflight checks for real, then reports the phases it would have run instead of running them. Each phase below is labelled with what a dry run does with it. A phase is only run when it declares that it is safe to, so a phase added later is reported until someone says otherwise.
The report is not a static list. Every phase still prepares itself and checks whether it has anything to do, and both only read, so work that is already done is filtered out against the live hosts. A phase that could not be assessed, because it reads state an earlier reported phase would have created, is reported as unassessed.
A dry run takes no cluster lock, so it does not block a real run, and it can report state that a concurrent run is already changing. It does not need --confirm.
- Connect to hosts
- Connects to a remote host via
github.com/k0sproject/rig - Dry run: runs, reads only. Connects opens the SSH session to each host and does nothing else. A dry run needs it, because a preflight that never reached a host would report on a cluster it never looked at.
- Connects to a remote host via
- Detect host operating systems
- Gathers information about the remote host, including: OS and OS version
- Dry run: runs, reads only. Reads
/etc/os-releaseand the kernel to pick a configurer for the host. Reporting what each host runs is half of what makes a dry run worth running.
- Acquire exclusive host lock
- Runs a background task that touches a file every 30 seconds on each remote node. This prevents other
cargoshipsfrom making any changes until the lock file has not been touched for over a minute - Dry run: reported, not run
- Runs a background task that touches a file every 30 seconds on each remote node. This prevents other
- Gather host facts
- Gathers network related information about the remote host, including: Hostname, Private Address, Private Interface. Will also update the hosts based off the profile if configured in the config file.
- Dry run: runs, reads only. Gather facts about each host by asking for its hostname, private interface and private address. All three are reads, and the rest of the run decides what it would do from them.
- Validate hosts
- Verifying that each node in the cluster has a unique name and private address, that its CPU architecture is one the package carries, and that its firewall rules are usable,
- Dry run: runs, reads only. Validate the hosts is the preflight itself: sudo, unique hostnames and addresses, host architecture, firewall rules and clock skew. A dry run that skipped it would check nothing.
- Gathering facts about the distro installed
- Gathers information relating to the specific distro being installed, including: if the distro is installed, and what version it is running
- Dry run: runs, reads only. Gather distro related facts reads the engine version already on each host. That is what tells an upgrade apart from an install, and what catches a downgrade before a real run starts one.
- Sync Registry Config Controller
- If the remote node is a controller and its engine config (registries/audit/pss) has drifted from the desired state, drain the node, stop the service, write the new config, start the service, and uncordon the node sequentially. Chart values are written in place instead, since the engine reconciles them without a restart
- Dry run: reported, not run
- Sync Registry Config Worker
- If the remote node is a worker and its engine config (registries/audit/pss) has drifted from the desired state, drain the node, stop the service, write the new config, start the service, and uncordon the node by the set concurrency limit
- Dry run: reported, not run
- Updating kubeconfig file with the current cluster
- If enabled, this will update the local kubeconfig with the admin creds for the current distro
- Dry run: reported, not run
- Labeling nodes with their profile group
- If enabled, this checks each node’s
node-role.kubernetes.io/<profile>label and adds it, set to “true”, when missing or set to anything else - Dry run: reported, not run
- If enabled, this checks each node’s
- Release exclusive host lock
- Deletes the lock file from each node, allowing other
cargoshipsto run - Dry run: reported, not run
- Deletes the lock file from each node, allowing other
- Disconnect from hosts
- Deletes any lingering temp files and disconnects from the remote node
- Dry run: runs its own dry-run path
kube-config
kube-config phases
- Connect to hosts
- Connects to a remote host via
github.com/k0sproject/rig
- Connects to a remote host via
- Detect host operating systems
- Gathers information about the remote host, including: OS and OS version
- Acquire exclusive host lock
- Runs a background task that touches a file every 30 seconds on each remote node. This prevents other
cargoshipsfrom making any changes until the lock file has not been touched for over a minute
- Runs a background task that touches a file every 30 seconds on each remote node. This prevents other
- Gather host facts
- Gathers network related information about the remote host, including: Hostname, Private Address, Private Interface. Will also update the hosts based off the profile if configured in the config file.
- Updating kubeconfig file with the current cluster
- If enabled, this will update the local kubeconfig with the admin creds for the current distro
- Release exclusive host lock
- Deletes the lock file from each node, allowing other
cargoshipsto run
- Deletes the lock file from each node, allowing other
- Disconnect from hosts
- Deletes any lingering temp files and disconnects from the remote node
prepare
prepare phases
With --dry-run, cargoship connects to every host and runs the preflight checks for real, then reports the phases it would have run instead of running them. Each phase below is labelled with what a dry run does with it. A phase is only run when it declares that it is safe to, so a phase added later is reported until someone says otherwise.
The report is not a static list. Every phase still prepares itself and checks whether it has anything to do, and both only read, so work that is already done is filtered out against the live hosts. A phase that could not be assessed, because it reads state an earlier reported phase would have created, is reported as unassessed.
A dry run takes no cluster lock, so it does not block a real run, and it can report state that a concurrent run is already changing. It does not need --confirm.
- Connect to hosts
- Connects to a remote host via
github.com/k0sproject/rig - Dry run: runs, reads only. Connects opens the SSH session to each host and does nothing else. A dry run needs it, because a preflight that never reached a host would report on a cluster it never looked at.
- Connects to a remote host via
- Detect host operating systems
- Gathers information about the remote host, including: OS and OS version
- Dry run: runs, reads only. Reads
/etc/os-releaseand the kernel to pick a configurer for the host. Reporting what each host runs is half of what makes a dry run worth running.
- Acquire exclusive host lock
- Runs a background task that touches a file every 30 seconds on each remote node. This prevents other
cargoshipsfrom making any changes until the lock file has not been touched for over a minute - Dry run: reported, not run
- Runs a background task that touches a file every 30 seconds on each remote node. This prevents other
- Prepare hosts
- Updates the remote nodes; environment variables and sysctl
- Dry run: reported, not run
- Prepare hosts - Enterprise Linux support
- Installs container-selinux on systems that have SELinux enabled on them
- Dry run: reported, not run
- Prepare hosts - Enterprise Linux support - Fapolicyd
- Writes the distro-supplied FAPolicy rules to /etc/fapolicyd/rules.d/31-cargoship.rules and executable rules to /etc/fapolicyd/rules.d/32-cargoship-executable.rules
- Dry run: reported, not run
- Prepare hosts - Enterprise Linux support - SELinux policy
- Installs the policy modules, booleans and file contexts from spec.config.os.selinux on hosts running SELinux in enforcing mode, using semodule, setsebool and semanage fcontext
- Dry run: reported, not run
- Enable the requested kernel modules
- Turns on the list of requested modules on the host, then reboots the box if modules are added
- Dry run: reported, not run
- Release exclusive host lock
- Deletes the lock file from each node, allowing other
cargoshipsto run - Dry run: reported, not run
- Deletes the lock file from each node, allowing other
- Disconnect from hosts
- Deletes any lingering temp files and disconnects from the remote node
- Dry run: runs its own dry-run path
refresh
refresh phases
- Connect to hosts
- Connects to a remote host via
github.com/k0sproject/rig
- Connects to a remote host via
- Detect host operating systems
- Gathers information about the remote host, including: OS and OS version
- Gather host facts
- Gathers network related information about the remote host, including: Hostname, Private Address, Private Interface. Will also update the hosts based off the profile if configured in the config file.
- Gathering facts about the distro installed
- Gathers information relating to the specific distro being installed, including: if the distro is installed, and what version it is running
- Disconnect from hosts
- Deletes any lingering temp files and disconnects from the remote node
reset
reset phases
With --dry-run, cargoship connects to every host and runs the preflight checks for real, then reports the phases it would have run instead of running them. Each phase below is labelled with what a dry run does with it. A phase is only run when it declares that it is safe to, so a phase added later is reported until someone says otherwise.
The report is not a static list. Every phase still prepares itself and checks whether it has anything to do, and both only read, so work that is already done is filtered out against the live hosts. A phase that could not be assessed, because it reads state an earlier reported phase would have created, is reported as unassessed.
A dry run takes no cluster lock, so it does not block a real run, and it can report state that a concurrent run is already changing. It does not need --confirm.
- Connect to hosts
- Connects to a remote host via
github.com/k0sproject/rig - Dry run: runs, reads only. Connects opens the SSH session to each host and does nothing else. A dry run needs it, because a preflight that never reached a host would report on a cluster it never looked at.
- Connects to a remote host via
- Detect host operating systems
- Gathers information about the remote host, including: OS and OS version
- Dry run: runs, reads only. Reads
/etc/os-releaseand the kernel to pick a configurer for the host. Reporting what each host runs is half of what makes a dry run worth running.
- Acquire exclusive host lock
- Runs a background task that touches a file every 30 seconds on each remote node. This prevents other
cargoshipsfrom making any changes until the lock file has not been touched for over a minute - Dry run: reported, not run
- Runs a background task that touches a file every 30 seconds on each remote node. This prevents other
- Gather host facts
- Gathers network related information about the remote host, including: Hostname, Private Address, Private Interface. Will also update the hosts based off the profile if configured in the config file.
- Dry run: runs, reads only. Gather facts about each host by asking for its hostname, private interface and private address. All three are reads, and the rest of the run decides what it would do from them.
- Validate hosts
- Verifying that each node in the cluster has a unique name and private address, that its CPU architecture is one the package carries, and that its firewall rules are usable,
- Dry run: runs, reads only. Validate the hosts is the preflight itself: sudo, unique hostnames and addresses, host architecture, firewall rules and clock skew. A dry run that skipped it would check nothing.
- Gathering facts about the distro installed
- Gathers information relating to the specific distro being installed, including: if the distro is installed, and what version it is running
- Dry run: runs, reads only. Gather distro related facts reads the engine version already on each host. That is what tells an upgrade apart from an install, and what catches a downgrade before a real run starts one.
- Reset Worker
- Deletes the worker from the cluster, if enabled it will try to drain node before removing the node
- Dry run: reported, not run
- Reset Controller
- Deletes the controller from the cluster, if enabled it will try to drain node before removing the node
- Dry run: reported, not run
- Uninstalling Engine
- Remove the rpm, apt, or binary files from all the hosts
- Dry run: reported, not run
- Repoint the controllers that were kept
- When a removal takes a controller out, rewrites
server:on the controllers that stayed so they name one that is still there, which is what lets them rejoin after a restart - Dry run: reported, not run
- When a removal takes a controller out, rewrites
- Remove the SELinux policy cargoship applied
- Removes the policy modules and file contexts recorded in /var/lib/cargoship/selinux/cargoship-state.json and returns the booleans to the values the host held before cargoship changed them
- Dry run: reported, not run
- Reload service manager
- Runs
systemctl daemon-reloador equivalent on all hosts. - Dry run: reported, not run
- Runs
- Release exclusive host lock
- Deletes the lock file from each node, allowing other
cargoshipsto run - Dry run: reported, not run
- Deletes the lock file from each node, allowing other
- Disconnect from hosts
- Deletes any lingering temp files and disconnects from the remote node
- Dry run: runs its own dry-run path
Build Flags and Environment
This document explains the compiler flags, linker flags, and environment variables used when compiling the cargoship binary, and why each one is set. They are defined in pkg/utils/build/utils.go, which the Mage host build path (magefiles/pkg/build/build.go) uses.
That package exposes two functions, LDFlags(version, commit string) string and GCFLags() string. The build site additionally sets CGO_ENABLED=0 and passes -trimpath directly in its go build invocation rather than through these shared helpers. .goreleaser.yaml and .github/workflows/e2e.yaml spell the same flags out inline, so a change here has to be made in those two files as well.
Why this matters
An unoptimized go build of this repo produces a binary well over 130MB on Linux/amd64, mostly because cargoship pulls in large transitive dependency trees (k8s.io/client-go, zarf’s signing stack, cloud SDKs, etc.) and, without CGO_ENABLED=0, dynamically links against glibc. The flags below were chosen specifically to keep the shipped binary as small and portable as possible without changing behavior.
Measured impact on a Linux/amd64 build of this repo:
| Configuration | Size | Linking |
|---|---|---|
go build with no flags | ~136MB | dynamic (glibc) |
+ CGO_ENABLED=0 | ~98.5MB | static |
+ -trimpath | ~98.2MB | static |
default gcflags instead of -l (for comparison) | ~113MB | static |
-l (shipped) | 97.95MB | static |
-l -B -C (former setting, for comparison) | 96.86MB | static |
Environment variables
CGO_ENABLED=0
Forces a pure-Go, statically-linked binary.
- Why: on a native Linux/amd64 host, Go’s default is
CGO_ENABLED=1whenever a C toolchain is present, which produces a dynamically-linked binary against glibc. This repo doesn’t need cgo (no imports rely on it), so leaving it enabled only adds size and a runtime dependency on the host’s libc/dynamic linker. Disabling it saved ~28% of binary size in testing and makes binaries fully static and portable across Linux distros/container base images. - Where set:
env["CGO_ENABLED"] = "0"inmagefiles/pkg/build/build.go’sBinary.
GOOS / GOARCH
Standard Go cross-compilation target selection, set per invocation from the os/arch arguments passed to the build functions. Not size-related; documented here only because they’re set alongside CGO_ENABLED in the same env map/chain.
go build flags
-trimpath
Strips local filesystem paths (e.g. /home/user/git/cargoship/...) from the compiled binary, replacing them with module paths.
- Why: without it, absolute build-machine paths get embedded in the binary (used for panic traces and debug info paths), which leaks local environment details and hurts build reproducibility. It also shaves a small amount of size (a few hundred KB) since fewer/shorter path strings end up in the binary.
- Where set: directly in the
go buildcommand string inmagefiles/pkg/build/build.go.
-ldflags (see LDFlags in pkg/utils/build/utils.go)
-
-s- omits the symbol table. Symbols aren’t needed at runtime and aren’t useful without-wanyway; this is one of the two biggest size wins available via linker flags. -
-w- omits DWARF debug info. Removes the ability to attach a source-level debugger (dlv) to the binary, but this is a release build, not a debug build. Combined with-s, this is what turns the ~157MB unstripped analysis build in testing into a much smaller shipped binary. -
-X github.com/colonel-byte/cargoship/config.CLIVersion=%s- embeds the release version string at link time. -
-X github.com/colonel-byte/cargoship/config.CLICommit=%s- embeds the short git commit SHA at link time.These two
-Xflags aren’t size-related; they exist socargoship versioncan report accurate build metadata without a separate version file shipped alongside the binary.
-gcflags=all="-l" (see GCFLags in pkg/utils/build/utils.go)
Applied to all packages (including the standard library and vendored dependencies), not just this module’s own code.
-
-l- disables inlining. Counterintuitively, this reduces binary size in this repo: inlining duplicates the inlined function’s code at every call site, and with a dependency tree this large, the code-size cost of inlining outweighs the runtime speed benefit for a CLI tool that isn’t CPU-bound in a hot loop. Measured:-lalone produced a binary ~13% smaller than the same build with defaultgcflags(97.95MB against ~113MB).-lis the onlygcflagsentry this build sets.-Band-Cwere set in an earlier revision and are gone - see Flags that were removed for why.Note that
all=means changing this flag invalidates the build cache for the entire dependency tree, so the first build after touching it is slow. That is a per-flag-change cost, not a per-build one.
Flags that were removed
These three were set in an earlier revision of this build and are no longer used.
-gcflags=all=-B(disable bounds checking) - removed. Measured at 1.09MB of 97.95MB, about 1.1%. In exchange it stripped bounds checks from every package in the tree, including the code whose job is parsing input this binary does not control: archive extraction, OCI manifests and layers, registry responses. That is exactly where a bounds check is the mechanism that turns a malformed length field into a panic instead of an out-of-bounds read, and 1% of binary size is not worth giving that up. If a hard-to-diagnose crash ever appeared only in release builds, this flag was the first suspect — one more reason it is gone.-gcflags=all=-C- removed. It never did anything for size.go tool compile -helpdocuments-Cas “disable printing of columns in error messages”: a compiler diagnostic formatting flag with no effect on generated code. An earlier revision of this document claimed it disabledunsafe.Pointer/checkptr-adjacent checks; that was wrong.-a(force rebuild of all packages) - removed. It was justified here as a guard against a stale build cache masking flag changes, but Go’s build cache is keyed on build flags, so that situation cannot arise: changing-gcflagsor-ldflagsalready rebuilds everything those flags affect. All-adid was make every build redo the standard library.
What was deliberately not changed
- UPX or other binary compressors - not used. UPX-compressed binaries unpack themselves into memory at startup, adding a small latency hit, and self-modifying/self-extracting binaries are frequently flagged by antivirus and endpoint security tools. Given
cargoshipoperates in cluster-management/infrastructure contexts where such scanning is common, this tradeoff wasn’t taken. - Trimming
zarf/k8s.io/client-godependencies - these are the largest remaining contributors to binary size, and they are not a build-flag question: shrinking them means changing what the binary links, which is tracked separately from this document.k8s.io/apiandk8s.io/client-goare load-bearing for cluster operations and reached frompkg/phasedirectly.zarf’s signing stack has since been partly addressed -pkg/signingis a fork that no longer registers the four cloud KMS plugins, worth about 7.4MB, because none of them can reach a cloud endpoint from an air-gapped node. The roughly 45MB still behind cosign’s CLI command structs (Open Policy Agent, cuelang, go-github, the GitLab client) is untouched, because replacing those structs means rewriting verification rather than trimming a dependency. Seesigning.mdand../agent/choice-signing-fork.md.
Running the cluster suite against your own fleet
The cluster phase suite normally provisions the hosts it tests: bootloose containers by default, or local virtual machines with CARGOSHIP_E2E_BACKEND=microvm. The third backend provisions nothing. Point it at a ZarfCluster document naming hosts that already exist and it runs the same phase assertions against them.
CARGOSHIP_E2E_BACKEND=inventory \
CARGOSHIP_E2E_INVENTORY=$PWD/my-fleet.yaml \
go test -mod=vendor -count=1 -v -timeout=90m ./test/e2e/cluster/...
That is what makes the suite usable as an acceptance test for an environment cargoship has not been proven against – EC2, GCE, vSphere, Proxmox, Hetzner, bare metal – rather than only as a regression test for the two fleets this repository knows how to build. The phases are the product; the hosts are yours.
The apply walk expects hosts with no engine on them. Test_12_GatherFactsDistro asserts every host reports no engine version before anything is installed, which is what makes the later upgrade-phase assertions mean anything. So the walk is a one-shot against a given fleet: running it a second time fails, correctly, and what follows an install is the upgrade or reset walk rather than another apply. Reset or rebuild the hosts before installing again.
It changes the hosts it is given, and it does not change them back. The walk installs an engine, rewrites /etc/hosts, configures the firewall, writes sysctl and fapolicyd files, and installs packages. Nothing is torn down on the way out – the suite did not create these machines, so it will not destroy them, and that cuts both ways. Use hosts you are willing to lose.
The document
The same ZarfCluster document cargoship apply --config takes. It is read through the loader the CLI uses, so anything cargoship accepts is accepted here and anything it rejects fails before a host is touched.
apiVersion: colonel-byte.dev/v1alpha1
kind: Cluster
metadata:
name: my-fleet
spec:
config:
loadbalancer: 10.0.1.10
hosts:
- ssh:
address: 10.0.1.10
user: ubuntu
port: 22
keyPath: /home/me/.ssh/fleet
hostname: ip-10-0-1-10
privateAddress: 10.0.1.10
role: controller
profile: controller
- ssh:
address: 10.0.1.11
user: ubuntu
port: 22
keyPath: /home/me/.ssh/fleet
hostname: ip-10-0-1-11
privateAddress: 10.0.1.11
role: worker
profile: worker
test/e2e/cluster/testdata/inventory-external.yaml is a working example of the shape, and is what the unit tests read.
What the suite requires of it:
- At least one host with
role: controller. Without one there is no control plane to install, and the run says so rather than failing inside a phase. roleon every host. This backend reads roles from the document and never from the hostnames. The other two fleets name their machineskc0,kw0and so on and let a prefix carry the role; your hostnames are whatever your fleet calls them, so the field is the only source.privateAddresson every host, andprivateInterfacewhere the default route is not on the interface the nodes talk to each other over. The firewall and/etc/hostsphases distribute these to every peer. Fact gathering will discover an address if you leave it out, and on a multi-homed host it discovers the one holding the default route, which may not be the one the cluster should use.- Key-based SSH, as a user who can
sudowithout a password. Every phase runs privileged commands throughSudo(). SSH_KNOWN_HOSTS=""in the environment if your hosts’ keys are not in a known-hosts file the test binary will read.
Environment
| Variable | What it does |
|---|---|
CARGOSHIP_E2E_BACKEND | inventory to select this backend |
CARGOSHIP_E2E_INVENTORY | Path to the ZarfCluster document |
CARGOSHIP_E2E_JOIN_HOST | Hostname the join walk should join. Unset skips that walk; see below |
CARGOSHIP_E2E_DISTRO | k3s (default), rke2 or upstream, which engine to install |
SSH_KNOWN_HOSTS | Set to "" when your hosts’ keys are not in a known-hosts file the test binary reads |
What runs, and what stands down
Every phase runs. Some assertions do not, because they are assertions about a fleet this suite chose:
| Assertion | Against your fleet |
|---|---|
| Each host runs the OS its image declared | Stands down. Your hosts’ operating systems are not knowable until phase 09 has asked them, so the test asserts only that detection produced an answer, and logs the mix it found |
| The fleet runs every OS family | Stands down, for the same reason |
| An Enterprise Linux host exists for the RPM phase, a Debian host for the APT phase | Stands down. Those guards protect a fleet that was meant to cover a branch and lost the host for it, which cannot be said about hosts the suite did not choose |
| The join walk | Skipped unless CARGOSHIP_E2E_JOIN_HOST names a host for it, because the suite cannot create one. See “Adding a host” below |
| Everything else – the prepare phases, SELinux, fapolicyd, the firewall, all four upload phases, the engine config, the engine install, the upgrade and reset walks | Runs, and asserts |
The run logs which of these it stood down, so a green result does not quietly read as broader coverage than it was.
Adding a host, and running the join walk against it
The join walk installs the engine on a host that was not in the cluster when the install ran, then asserts the cluster is one node larger rather than one node different. The other two backends create that host on demand. Here you create it, and it takes two runs – the suite’s walks run in one process, so there is no moment in a single run at which a host can appear.
First run: install the cluster on the hosts you have. Stop after the apply walk, because the join walk has nothing to join yet. The hosts have to be carrying no engine at this point, per the note above.
CARGOSHIP_E2E_BACKEND=inventory CARGOSHIP_E2E_INVENTORY=$PWD/my-fleet.yaml go test -mod=vendor -count=1 -v -timeout=90m -run 'TestClusterPhases/apply' ./test/e2e/cluster/...
Then provision the new host yourself – another instance in the same subnet or VLAN, reachable over SSH by the same key, able to reach the other nodes on the ports the engine uses – and add it to the document. A worker, since that is what the walk joins:
- ssh:
address: 10.0.1.12
user: ubuntu
port: 22
keyPath: /home/me/.ssh/fleet
hostname: ip-10-0-1-12
privateAddress: 10.0.1.12
role: worker
profile: worker
Second run: name it, and run only the join walk.
CARGOSHIP_E2E_BACKEND=inventory CARGOSHIP_E2E_INVENTORY=$PWD/my-fleet.yaml CARGOSHIP_E2E_JOIN_HOST=ip-10-0-1-12 go test -mod=vendor -count=1 -v -timeout=60m -run 'TestClusterPhases/join' ./test/e2e/cluster/...
CARGOSHIP_E2E_JOIN_HOST is the hostname as the document spells it, not the address. The walk resolves the host out of the inventory by that name, so a mismatch fails before any phase runs rather than after half a walk.
What that does with the document
The named host is held out of the fleet the install is considered to have produced, and put back for the join. One file, read twice: the apply walk’s view has every host except that one, the join walk’s view has all of them, and the host counts the assertions compare against come out exactly as they do for a fleet the suite provisioned. That is why the new host must be in the document for the second run and may be absent from the first – adding it earlier is harmless, because the first run does not look at it either way.
The walk then asserts what it always asserts: the phases claim the new node and not the established ones, the engine comes up on it at the version the cluster already runs, every established node’s firewall learns the new node’s address, and the cluster reports one more Ready node than before.
Two ways naming it goes wrong, both reported before a host is touched:
- A name the document does not hold. Check the
hostnamefield rather than the address. - The only host in the document. There is no cluster for it to join.
Leave CARGOSHIP_E2E_JOIN_HOST unset and the join walk is skipped with a line saying why, which is the right default: a document whose hosts are all already in the cluster has nothing to join.
Removing a host, and adding it back
A removal and a re-join are the same host taken out of the cluster and put back, and running them in sequence is the most useful exercise this backend offers: it is the only one that covers the phases in both directions against hosts you did not create.
Taking it out
cargoship reset scoped to the host. Keep the whole fleet in the document and name the one to remove:
cargoship reset --config my-fleet.yaml --distro rke2 --confirm --dry-run --target-hosts ip-10-0-1-12
cargoship reset --config my-fleet.yaml --distro rke2 --confirm --target-hosts ip-10-0-1-12
The dry run first: it reports every host it would reset and what it would do to each, and touches none.
The full inventory matters. The node is drained and deleted from the cluster before its engine comes down, and cargoship does that by driving kubectl through a controller that is staying – so one has to be in the document. A run that targets every running controller while keeping some other host is refused rather than attempted, because it would delete nodes through a machine it is about to uninstall. Targeting every host is allowed: that is a whole-cluster reset, which is what plain reset is. Removing more than one running controller in a single run is also refused; one run per controller.
What the host is left in is the point for what follows. The engine is uninstalled, its data and config directories go with it, and so does /etc/cargoship – the machine is back to the state the install walk expects, which is exactly the precondition a re-join needs. Verify it on the host rather than trusting the run, and check the engine-specific paths rather than their parents:
ssh ip-10-0-1-12 'for d in /var/lib/rancher/rke2 /etc/rancher/rke2 /etc/cargoship; do
printf "%-26s %s\n" "$d" "$(sudo test -e $d && echo present || echo gone)"; done'
/var/lib/rancher and /etc/rancher themselves stay, which is correct: the engine owns the subdirectory under each, and that is what CleanupPaths names. /etc/rancher/node stays too – the node-password material the engine’s own package put there is not cargoship’s to remove, and a re-join writes a fresh one.
Removing a controller also repoints the ones that stayed: the survivors carry server: naming the controller they joined through, so taking the first one out rewrites that on the rest and drops it from whichever survivor becomes first. Without that the cluster keeps working and cannot come back, because server: is read at startup. Check it the same way:
ssh ip-10-0-1-11 'sudo grep -H "^server:" /etc/rancher/rke2/config.yaml' # names a host that is still there
Checking what is left
Take the host out of the document, then check the remaining fleet. Not with the apply walk: those hosts run the engine and that walk asserts they do not. A CLI dry run reports over live state and changes nothing:
cargoship apply --config my-fleet.yaml --dry-run <package>
It should name no install work – the remaining hosts are already at the packaged version – and no host you removed. Confirm the cluster agrees from inside a node, which on a fleet whose load balancer is private is the only place the API is reachable:
kubectl get nodes # the removed node is gone, not NotReady
A node deleted from the cluster but left in the document shows up as a host the dry run still plans for, which is why the order matters.
Putting it back
The host is clean, so it re-joins the way any new host does – put it back in the document and name it to the join walk:
CARGOSHIP_E2E_BACKEND=inventory \
CARGOSHIP_E2E_INVENTORY=$PWD/my-fleet.yaml \
CARGOSHIP_E2E_JOIN_HOST=ip-10-0-1-12 \
go test -mod=vendor -count=1 -v -timeout=60m -run 'TestClusterPhases/join' ./test/e2e/cluster/...
That is the same invocation as “Adding a host” above, and it is the same work: the walk prepares the host, uploads to it, installs the engine and waits for the node to report Ready alongside the others. What makes the round trip worth running is that it is the one path through this suite where a host’s second install has to behave like its first – anything the reset left behind shows up here as a phase that finds work already done, or as an engine that starts against stale configuration.
Two things worth watching on the way back in:
- The node name. It rejoins under the name the inventory gives it, so a host put back under a different
hostnamearrives as a new node and the old object is not reused. That is usually what you want after a rebuild; it is a surprise if you expected the name to follow the machine. - Whether it was a controller. A controller that rejoins is not automatically the first one again: the survivor elected when it left keeps that role, and the returning host joins through it. The cluster is correct either way, and
cargoship applyis what re-renders the whole fleet’s view of who is first.
Removing the whole cluster is the reset walk, which runs last in a full TestClusterPhases and needs no flags. It is destructive and backs nothing up; what follows a reset is an apply.
Reading a failure
A failure here is more likely to be about your environment than about cargoship, and the suite is not in a position to tell the difference. Three things worth checking before filing anything:
- Node-to-node traffic. The engine phases are the first that need it, and a security group or host firewall that allows your SSH but not the cluster’s own ports fails at the worker join – the agent never starts, and the real reason is in the node’s journal rather than in the test output. The phase captures the journal of the host whose wait timed out.
- Which address the nodes got. If
privateAddressis wrong or discovered from the wrong interface, the firewall and/etc/hostsphases distribute addresses no peer can reach, and everything up to the engine still passes. dnf versionlockand the SELinux policy packages. A minimal Enterprise Linux image carries neither the versionlock plugin norcontainer-selinux, and phase 21 is fatal when it cannot install the latter. Both need a reachable package repository, which an air-gapped fleet does not have by definition – stage them into the image instead.
Verifying work that is not on main yet
This backend and the VM fleet are also how a change to the lifecycle phases gets exercised before it merges, which previously had no local path at all: the container backend cannot form a multi-node cluster on a host whose firewall drops container-to-container traffic, so the full walk – the join, upgrade and reset walks, and every phase after the engine starts – has never run outside CI, and CI runs only the staging half.
Check out the branch, bring a fleet up, and run the suite against it. --target-hosts is the worked example, and it is worth knowing what came out of running it this way before it merged. A subset reset’s whole risk is in which hosts it claims and which controller it drives kubectl through, and neither is observable without a real multi-node cluster to take one node out of – the first run against one found three defects that every unit test passed over:
- the host was matched to its node by the name the machine reports, which on anything with a search domain is not the node’s name, so the phase claimed no hosts and the engine was uninstalled while the node stayed in the cluster
- the worker path deleted the node without stopping the engine first, so the kubelet re-registered it seconds later
- the uninstall guarded every directory removal on a
test -fcheck, which is false for a directory, so a reset left the whole data directory behind while reporting success
None of those is visible from a single-node cluster or from a container, which is the argument for this backend in one paragraph.
The other two backends
microvm is the local VM fleet, which is the one to reach for first: it is a real kernel with SELinux enforcing and costs a minute to bring up. e2e-tests and e2e-phase-tests cover the suite itself and the default container backend.
Adding an E2E Test for a New Apply Phase
The cluster suite in test/e2e/cluster walks the apply phase list one phase at a time against a live bootloose cluster and asserts what each phase left on the hosts. Adding a phase to pkg/action/apply.go without adding a test here leaves a gap that nothing else covers, so this page is the checklist for closing it.
One top-level test, TestClusterPhases, runs five walks against the same cluster in the order they work in: dryrun runs whole actions with --dry-run over nodes with nothing on them and asserts it left them that way, apply installs the distro, join starts one more machine and brings it into the running cluster, upgrade walks the same phases again with a newer package, and reset takes the distro back off. A new phase needs a test in the apply walk, and usually one in the join and upgrade walks too – see Adding the join half and Adding the upgrade half.
For why the suite is built this way, see choice-phase-e2e-tests. For running the e2e suites generally, see e2e-tests.
One number per phase
A phase is identified by its source file’s number, and that number is used twice:
- The test file name mirrors the phase’s source file. A test for
pkg/phase/25_modify_hosts_file.gogoes intest/e2e/cluster/25_modify_hosts_file_test.go. Same number, same name, one file each. - The method name carries the same number. That file contains
Test_25_ModifyHosts.
Testify runs suite methods in lexicographic order of the method name, so the number is also the order the phases run in here. The numbers are zero-padded to two digits, which makes lexicographic order numeric order.
That order matches apply’s everywhere except the lock. Apply takes the lock third, right after OS detection, and holds it for the whole run; phase/91_lock.go’s number puts it after the install instead, with phase/92_unlock.go right behind it. The lock test still asserts what the phase writes on each host. What it no longer does is hold the lock across the phases in between, so a phase that misbehaves while the cluster is locked is not something this suite would see.
Steps that are not phases – creating the package, prepare, the health check – have no phase number to take, so they use the two ends of the ordering: Test_00 and Test_01 sort before every phase, and Test_ZZ1 onwards sort after every phase, because a letter sorts after a digit. They live in cluster_lifecycle_test.go, alongside ResetSuite, which is the whole of the reset walk. Those steps run whole actions rather than single phases – distro.Create, action.NewPrepare, action.NewApply, action.NewReset, action.NewKubeConfig – each with its own manager, and are given e2e.ClusterConfigPath, the engine-only inventory, rather than the full one the harness is built from; see the section on the upload-only host below.
Adding a phase
Take the new phase’s file number for both the test file and the method. Nothing else is renumbered: inserting phase/23_something.go gives you 23_something_test.go containing Test_23_Something, and it lands between Test_22_PrepareFapolicy and Test_25_ModifyHosts on its own.
If a phase is renumbered in pkg/phase, rename its test file and its method to match, and check that its new number still puts it after everything it depends on – the number is the ordering, so a phase that moves in the source moves here too.
Writing the test
Start from the phase’s source file and ask what an operator would look at to confirm it did its job. That is the assertion. A phase returning nil is not evidence of anything – read the host back.
A minimal test:
// Test_NN_YourPhase covers phase/NN_your_phase.go. <One or two sentences on what the phase
// does and, therefore, what artifact this asserts.>
func (s *ApplyPhaseSuite) Test_NN_YourPhase() {
s.runPhase(&phase.YourPhase{Distro: s.harness.distro})
for _, host := range s.harness.hosts() {
s.Require().Truef(host.FileExist(someArtifactPath),
"%s: nothing at %s", host, someArtifactPath)
}
}
s.runPhase runs the phase through the shared manager and fails the test if it errors. Do not call s.harness.run directly unless you need the error itself.
Use s.Require() rather than s.Assert(). These tests are ordered and stateful, so continuing after a failed assertion produces noise about state the failed assertion already told you was wrong.
Each assertion body lives on phaseWalk, the embedded type every suite is built from, with a one-line Test_NN method on each suite calling it. Testify collects methods by their Test prefix, so the shared method is called, not run as a test of its own. Start a shared body with s.T().Helper().
What the harness gives you
phase_harness.go holds everything shared. The pieces a new test is likely to want:
s.harness.hosts() | every host currently on the manager |
s.harness.controllers() / s.harness.workers() | the role split |
s.harness.engineWorkers() | the workers that join the cluster, so workers() minus the upload-only hosts |
s.harness.uploadOnly() | the hosts that receive uploads and never join |
s.harness.manager | the live phase.Manager, for Distro, DistroID, TempDirectory |
s.harness.distro | the distro module, for phases that take a Distro field |
s.harness.dropUploadOnlyHosts() | removes those hosts from the manager; already called once, in Test_60 |
s.harness.carriesFilesFor(selector) | whether the package ships OS files for config.SelectorRPM / SelectorAPT / SelectorBIN |
s.harness.opts | the apply options the suite runs with (ModifyHosts, ModifyFirewall, WorkerConcurrent, …) |
readOnHosts(hosts, path) | reads one path on every host, keyed by host string |
lockFileContent() | the hostname-pid string the lock phase writes |
ran(p) | whether the manager executed the phase or skipped it |
For anything else, the hosts carry a live rig connection: host.ExecOutput, host.ReadFile, host.FileExist, and host.Configurer are all available and are how most assertions are written.
Phases that are expected to skip
Some phases legitimately do nothing on this cluster – SELinux where the runtime exposes none, fapolicyd when no host runs it, config sync when nothing has drifted. Do not write a test that asserts nothing; assert the gate instead. Compute what the hosts report, then require that ran(p) matches it:
var fapolicyd int
for _, host := range s.harness.hosts() {
if host.Configurer.ServiceIsRunning(host, phase.FAPOLICYD) {
fapolicyd++
}
}
carriesRules := s.harness.manager.Distro.Spec.Config.OS.FAPolicyd != ""
p := &phase.PrepareFapolicy{}
s.runPhase(p)
s.Require().Equalf(fapolicyd > 0 && carriesRules, ran(p),
"phase ran=%v with %d fapolicyd hosts and rules=%v", ran(p), fapolicyd, carriesRules)
if !ran(p) {
return
}
// ... assert the artifact where it did fire
This tests the routing, which is the part most likely to break, and it keeps working when the cluster gains a host that does trip the gate. 21_prepare_selinux_test.go and 22_prepare_fapolicyd_test.go are the worked examples.
ran(p) is only meaningful after s.runPhase, because the manager calls Prepare first and most gates are computed there.
Phases that route on the OS family
The cluster runs multiple OS families on purpose. In the default full-walk inventory (k3sOS), all five machines join the cluster – one controller and four workers, split between Ubuntu (kc0, kw0, kw1) and Fedora (kwf0, kwf1). The stage-only inventory (stageOS) is not that set plus a machine: it is a different five, one per family per role – an Ubuntu controller (kc0), a Fedora controller (kcf0), an Ubuntu worker (kw0), a Fedora worker (kwf0), and an Alpine machine (kwa0) that receives uploads but never joins the cluster, covered in the section on the upload-only host below. So a Fedora controller only exists in the stage-only walk, and k3sOS has no Alpine host at all. A phase that treats Enterprise Linux differently from Debian must be tested on both sides, not just on the side it claims.
Derive the host set from the same filters the phase uses, so the test cannot disagree with it:
enterprise := s.harness.hosts().Filter(utils.FilterEnterpriseLinux)
debian := s.harness.hosts().Filter(utils.FilterDebianLinux)
Then assert the split: the hosts the phase claims got what it promises, the hosts it does not came back unchanged. 57_rpm_install_test.go and 58_apt_install_test.go are the worked examples.
Require the set you are testing to be non-empty. If someone collapses the cluster to one image, that turns a test which silently stopped covering anything into a failure that says so.
The upload-only host, and the boundary at Test_60
Alpine is neither Enterprise Linux nor Debian, so both of the filters above decline it and it falls through to the BIN upload phase. That is why it is in the inventory: without it, 59_bin_install.go is only ever exercised as the path every host happens to take, never as the fallback it is written to be.
It cannot run the engine, which links against glibc. So Test_60_ConfigureEngine calls s.harness.dropUploadOnlyHosts() before it does anything else, and from that point on the host is gone from the manager entirely, when there is one to drop. Phases up to and including Test_59 see every host in the inventory; everything after sees one fewer, if the inventory carried an upload-only host at all.
The default full-walk inventory (k3sOS) has no upload-only host, so that drop is a no-op there; the Alpine host, and the boundary it exercises, live in the smaller stage-only inventory (stageOS) instead. See clusterConfig in main_test.go.
This matters when adding a phase:
- An upload phase goes before the drop, and should assert against
s.harness.uploadOnly()explicitly if it is meant to claim those hosts. - Anything that installs, starts, configures or queries the engine goes after the drop, and needs nothing special –
hosts()andworkers()already exclude the dropped hosts by then. - Do not “fix” a phase that fails on Alpine by skipping the host in the assertion. The phases select their own hosts in
Prepare, with no OS gate in most cases, so the phase will still have run against it and spent its retry budget there. If a phase genuinely cannot run on a host, that host must not be on the manager when the phase runs.
s.harness.engineWorkers() exists for the one case where order is not obvious: a test that runs before the drop but is only meaningful for hosts that join.
The dry-run walk
DryRunSuite (dry_run_test.go) is the one walk that does not step through phases, and the one whose assertions are negative. It runs action.NewApply, action.NewPrepare and action.NewReset whole, with Manager.DryRun set, and then asserts that none of the artifacts the other walks look for arrived: no upload manifest, no sysctl file, no lock file, no running engine. It runs first because “the hosts are unchanged” is only checkable before the apply walk changes them.
Two things about it are easy to get wrong when extending it.
The hosts are disconnected by the time a run returns, because every action ends with the disconnect phase and its dry-run path disconnects as well. A read taken after the run reports every file as absent whether it is there or not, which would make each assertion pass over a host a dry run had written to. reconnectHosts runs Connect and DetectOS again – both read-only – and the assertions go after that.
The walk also asserts the positive half, which is what stops the negative half from being vacuous: the read-only phases have to have gathered their facts, so each host carries a configurer, a hostname, an architecture, and phase.UnknownVersion for the engine it is not running. A dry run that never connected to anything would satisfy an untouched-hosts check perfectly.
A new phase needs nothing added here unless it is read-only, in which case the facts it gathers belong in requireFactsGathered, or it writes an artifact worth naming, in which case its absence belongs in requireHostsUntouched next to the path the install test asserts exists.
The two reset cases are split by what they need. reset --dry-run against a cluster that is up is in ResetSuite as Test_0, immediately before the real reset, because that is the only point where a live cluster exists. reset --dry-run with no running controller is in this walk, because nodes nothing was installed on are the natural fixture for the nil-leader path PR #312 found against a live cluster.
Adding the join half
JoinPhaseSuite walks the same phase list a second time, against the cluster the apply walk installed and one machine larger. It carries the same package at the same version, so nothing routes to the upgrade phases and the only host with work left is the one that just appeared. It is the only coverage of apply against a cluster that already exists.
So a new phase usually needs a join test as well. Which kind depends on what it does the second time:
-
The phase does the same thing either way – an upload phase, a config render, anything that does not read what is already running. Move the body to an unexported method on
phaseWalkand give each suite a one-lineTest_NNmethod calling it.50_uploadfiles_test.gois the worked example. -
The phase behaves differently – anything gated on what the hosts report. Write two methods with the same name, one per suite, in the same file.
61_initialize_controller_test.gois the worked example: the apply walk requiresran(p), the join walk requires!ran(p)because the control plane is already up, and each says in its comment why.
// Test_NN_YourPhase, on the join walk, <what is different and why>.
func (s *JoinPhaseSuite) Test_NN_YourPhase() {
p := &phase.YourPhase{Distro: s.harness.distro}
s.runPhase(p)
s.Require().False(ran(p), "<the routing this proves>")
}
Phases that assert over the whole matrix of hosts are worth a second look here rather than treating as repetition: Test_25 and Test_26 on the join walk are what prove the established nodes learned about the new one, not only that the new node learned about them.
The join machine is kwf2, a Fedora worker, provisioned on first use by requireJoinMachine. A Fedora worker is deliberate: the new node then comes through the SELinux, fapolicyd, firewalld and dnf branches on its own rather than inheriting anything the apply walk proved on the nodes beside it.
Adding the upgrade half
UpgradePhaseSuite walks the same phase list a third time, against the cluster the apply and join walks left running, with a package one patch release newer. It is what covers the code inside phase/66_upgrade_controller.go and phase/67_upgrade_worker.go: on a fresh install those phases claim no hosts, so the apply walk can only ever assert that they did nothing.
It starts at Test_12_GatherFactsDistro. Connect, DetectOS, GatherFacts and ValidateHosts run as a single Test_01_Reconnect step, because the apply walk already asserts them one at a time and they behave identically the second time round. The lock phases are left out for the same reason they are left out of the join walk.
So a new phase needs an upgrade test if it is numbered 12 or higher, and the two kinds are the same two the join walk has: a shared body on phaseWalk where the phase does the same thing either way, and a method per suite where it does not. 61_initialize_controller_test.go is the worked example again – the apply walk requires ran(p), the upgrade walk requires !ran(p), because an initialize phase that claimed a running node would re-bootstrap a live cluster.
// Test_NN_YourPhase, on the upgrade walk, <what is different and why>.
func (s *UpgradePhaseSuite) Test_NN_YourPhase() {
p := &phase.YourPhase{Distro: s.harness.distro}
s.runPhase(p)
s.Require().False(ran(p), "<the routing this proves>")
}
Four things the upgrade tests use that the others do not:
installedVersion | the version the apply walk put on the cluster, so what a host should report before the upgrade phase runs |
s.harness.manager.Distro.Spec.Version | the version the upgrade package carries, so what it should report after |
s.harness.engineHosts() | every host that runs the engine, so hosts() minus the upload-only ones – the upload-only host still reports phase.UnknownVersion |
s.requireSchedulable(hosts) | asserts each host’s node is registered and not left cordoned, which is what the drain/uncordon half of an upgrade has to leave behind |
Assert version routing by calling the phases’ own comparison rather than comparing strings:
compare := &phase.GenericPhase{}
s.Require().True(compare.VersionLess(host, s.harness.manager.Distro.Spec.Version))
VersionLess reads no manager state, so a zero-valued phase is enough, and using it means the test cannot disagree with the routing it is asserting.
Running it
The upgrade walk is off unless CARGOSHIP_E2E_UPGRADE is set. It installs a second complete set of engine images onto every node, which roughly doubles the disk the run needs and adds tens of minutes.
$ CARGOSHIP_E2E_UPGRADE=1 go test -mod=vendor -count=1 -v -timeout=195m ./test/e2e/cluster/...
Without it, TestClusterPhases/upgrade skips with a message naming the variable, and the reset walk runs against the installed cluster as before.
Prerequisites and running it
The suite needs Docker and a real distro package, and it is not fast: it provisions five containers and installs k3s onto all of them, one controller and four workers, then starts a sixth and joins it. It needs no prebuilt binary – every step calls the cargoship packages directly, so a bare checkout is enough.
$ go test -mod=vendor -count=1 -v -timeout=105m ./test/e2e/cluster/...
Or through mage, which also clears leftover containers from a run that was killed before teardown. Unlike the other e2e mage targets, it builds nothing first:
$ mage test:endToEndCluster # the dry-run, install, join and reset walks
$ mage test:endToEndClusterDryRun # only the dry-run walk, against the staging cluster
$ mage test:endToEndClusterUpgrade # the same, with the upgrade walk in between
-short skips the whole suite, and CARGOSHIP_E2E_UPGRADE adds the upgrade walk. TestMain deletes the bootloose cluster on the way out even when tests fail.
This full walk, the one that installs and bootstraps k3s, is not run in CI – see below. It is a local-box check: run mage test:endToEndCluster (or the go test invocation above) on a machine with Docker before trusting a change to the engine-bootstrap phases, phase/61 onward.
For the smaller staging walk on its own, without the engine phases:
$ mage test:endToEndClusterStage
In CI
.github/workflows/e2e-cluster.yaml runs the staging half of this suite – CARGOSHIP_E2E_STAGE_ONLY=1, stopping at the boundary Test_60_ConfigureEngine draws – on its own trigger, separate from e2e.yaml. It runs on every pull request, the same as e2e.yaml, and on demand from the Actions tab. No label is needed.
The engine-bootstrap half, including the join and upgrade walks, does not run in CI at all. It was not reliable enough on a free hosted runner to be a required check, even on the single-controller k3s topology that replaced the original three-controller rke2 one – see choice-e2e-stage-split. Trusting a change to the engine-bootstrap phases, the join walk, or the upgrade walk means running mage test:endToEndCluster or mage test:endToEndClusterUpgrade locally; CI cannot cover that for you.
The upgrade walk has no CI path either. .github/workflows/e2e-cluster.yaml declares an upgrade boolean input for workflow_dispatch, but nothing in the file reads it: the single e2e-cluster-stage job hardcodes CARGOSHIP_E2E_STAGE_ONLY: "1" and runs on a 30-minute budget, so setting the input changes nothing. Run the upgrade walk locally with mage test:endToEndClusterUpgrade when a change touches the upgrade phases, the version comparison, or anything the install walk can only assert has stayed out of the way.
The workflow has no build step and takes no artifact from e2e.yaml. Nothing in the suite runs a binary: Test_00_CreatePackage calls distro.Create, and the prepare, apply, reset and kube-config steps call the matching action.New* entry points. That is what makes the two workflows independent, which is the point of the split.
The job frees disk before it starts, because five containerd image stores do not fit in what a hosted runner leaves free. If it fails with nodes stuck mid-upload, check the diagnostics step for a full disk or an OOM kill before reading the phase failure as a real one, and a larger runner is the fix.
Iterating while writing a new phase test
A new phase test needs the suite up to that point, not the whole run. Four environment variables, combined, cut a write-run-fix loop down to what your phase actually needs.
Run only the walk your phase is in. TestClusterPhases runs apply, join, upgrade and reset as separate subtests of one parent, so -run at that path segment excludes the ones you are not touching:
$ go test -mod=vendor -count=1 -v -timeout=30m -run 'TestClusterPhases/apply' ./test/e2e/cluster/...
This still walks every phase from Test_00 onward – there is no way to start partway through a walk, because each phase depends on state the ones before it left on the host, Test_07_Connect first among them. What this buys you is skipping the walks after the one you are writing for.
Stay in stage-only mode if your phase is before the Test_60 boundary. CARGOSHIP_E2E_STAGE_ONLY=1 uses the five-machine stageOS inventory and never starts the engine, so it finishes in a few minutes instead of tens:
$ CARGOSHIP_E2E_STAGE_ONLY=1 go test -mod=vendor -count=1 -v -timeout=30m -run 'TestClusterPhases/apply' ./test/e2e/cluster/...
A phase numbered 60 or higher needs the full k3sOS walk instead, so drop this variable for those.
Keep the containers after a failure to inspect them by hand. CARGOSHIP_E2E_KEEP_CLUSTER=1 leaves the machines running instead of deleting them when the run fails, so you can docker exec into one and check what your phase actually left, rather than only what the test’s own assertions could see:
$ CARGOSHIP_E2E_STAGE_ONLY=1 CARGOSHIP_E2E_KEEP_CLUSTER=1 \
go test -mod=vendor -count=1 -v -timeout=30m -run 'TestClusterPhases/apply' ./test/e2e/cluster/...
$ docker ps --filter "label=io.k0sproject.bootloose.owner=bootloose"
$ docker exec -it <container> sh
Run against virtual machines instead of containers. CARGOSHIP_E2E_BACKEND=microvm, or mage test:endToEndClusterVM, provisions the hosts as local Rocky 10 virtual machines rather than bootloose containers. Everything downstream of the inventory is identical; what changes is that the hosts have a real kernel, so the phases that correctly do nothing against a container – SELinux, fapolicyd, the firewalld backend – run as the paths they are:
$ CARGOSHIP_E2E_BACKEND=microvm go test -mod=vendor -count=1 -v -timeout=60m ./test/e2e/cluster/...
It needs KVM, and a bring-up costs a minute or two against a container’s seconds. It is also one distribution, so it covers none of the family routing the container fleet’s Ubuntu, Fedora and Alpine mix exists for – run both, not one instead of the other. See microvm for the fleet itself and choice-microvm-backend for the reasoning.
Run against a fleet the suite did not create. CARGOSHIP_E2E_BACKEND=inventory with CARGOSHIP_E2E_INVENTORY pointing at a ZarfCluster document runs every phase against hosts that already exist, wherever they are. The assertions that are about a fleet this suite chose – the declared OS per host, the OS mix, the per-family upload guards, the join walk – stand down and say so; everything else asserts as usual. See e2e-external-inventory, including the warning that it changes those hosts and does not change them back.
Walk the suite against rke2 instead of k3s. CARGOSHIP_E2E_DISTRO=rke2 switches which distro every step installs, and which package cluster_lifecycle_test.go builds. The default is k3s. A phase that branches on the engine needs a run each way:
$ CARGOSHIP_E2E_DISTRO=rke2 go test -mod=vendor -count=1 -v -timeout=90m -run 'TestClusterPhases/apply' ./test/e2e/cluster/...
See distroEnvVar and distroID in test/e2e/cluster/05_manager_test.go.
Each go test invocation provisions fresh containers – there is no “resume the cluster from last time” path, so a kept cluster from a failed run is for reading, not for the next run to build on. Clean it up before running again:
$ mage test:cleanCluster
or, without mage, the same removal stopBootlooseContainers does – see mage-test-manual.
Things that will cost you time
- You cannot run one phase test on its own.
-run 'TestClusterPhases/apply/Test_25_ModifyHosts'fails: the hosts were never connected, becauseTest_07_ConnectandTest_09_DetectOSare earlier methods that-runfiltered out. Run the whole suite, or accept that reproducing one phase means reproducing its predecessors. - The first failure stops the rest.
SetupTestskips every remaining step once one has failed, so the output names one phase rather than twenty-five. If the first failure looks like a symptom, it is the cause – everything after it was skipped, not passed. - Constants the phase package keeps unexported. Where a path is not exported, the test file redeclares it with a comment saying where it came from (
uploadManifestPathin50_uploadfiles_test.go,sysctlConfPathin20_prepare_host_test.go). Prefer exporting fromphasewhen the constant is genuinely part of the contract; duplicate it, with the comment, when it is not. - Six containers is still a lot of memory once the join walk has added its machine. Each runs a k3s node, which is the reason this suite is not part of the default
go test ./...path.
Running the End-to-End Tests
The noncluster e2e suite drives the built cargoship binary as a subprocess, the same way a user would. It does not import the command packages, so what it proves is that the shipped binary behaves, not that the internals compile. Everything below is about that suite and is a plain go test invocation, which is what you want when a test fails and you need to poke at it. The cluster suite is the other way round – it calls the cargoship packages directly and needs no binary at all; see e2e-phase-tests.
Layout
test/e2e/noncluster/ misc + package command groups: version, sha256sum, the vault and age suites, ssh, create,
publish, pull, sign, verify, schema, validate, the Ansible module, and the zarf info modules
test/e2e/cluster/ the install group: a bootloose cluster walked one phase at a time -- install, join, optionally upgrade, then reset
test/e2e/tofu/ the OpenTofu provider: a three-machine bootloose cluster installed, reduced and destroyed through the real `tofu` CLI -- see [e2e-tofu-tests](e2e-tofu-tests.md)
test/common.go the CargoE2ETest harness (e2e.Cargoship) shared by the suites
test/container_package.go copies a shipped example definition and strips the sysctls a container cannot apply, for the suites that build a package and install it
test/bootstrap.go TestMain's chdir-to-repo-root, with (Bootstrap) and without (BootstrapInProcess) the binary lookup
test/registry.go in-process OCI registry used by the publish/pull/sign/verify tests
test/kube.go kubernetes helpers for the suites that read a cluster back
test/e2e/AGENTS.md the fixture rule: test documents live in testdata/, not in Go string constants
The suites are separate Go packages so that a group can be selected by package path rather than by test-name filters. The noncluster package starts no containers and makes no network calls: it needs nothing but the binary. The cluster package needs Docker and takes tens of minutes, and needs no binary; everything below is about noncluster, and e2e-phase-tests covers the cluster suite and how to extend it when a phase is added.
Prerequisites
Build the binary first. The suite looks for it at build/cargoship_<goos>_<goarch> and TestMain aborts immediately if it is missing:
$ go build -mod=vendor -o "build/cargoship_$(go env GOOS)_$(go env GOARCH)" ./cmd/cargoship
This is enough for the tests. A build without the release linker flags leaves cargoship version reporting the unset placeholder values, which the tests tolerate - they assert the fields are present and non-empty, not what they contain.
The binary is not rebuilt by go test. After changing anything under api/, cmd/, config/, fuzz/, internal/, pkg/, or types/, rebuild it, or you will be testing the previous binary against the new expectations.
Running
$ go test -mod=vendor -count=1 ./test/e2e/noncluster/... # the whole group, about 6 seconds
$ go test -mod=vendor -count=1 -short ./test/e2e/noncluster/... # same, minus the real example package
$ go test -mod=vendor -count=1 -v -timeout=30m ./test/e2e/noncluster/...
-count=1disables the test result cache. Without it, a green run is cached and a rebuilt binary will not re-trigger it, since the binary is not one of the inputsgo testhashes.-shortskipsTestCargoshipCreateExample, which buildsexample/rke2-multi-cni-ciliumfor real and downloads the engine artifacts and images for both of the architectures that example covers. Everything else usestestdata/minimal, an image-free distro that builds into a 386-byte package in milliseconds.-timeoutdefaults to 10 minutes, which is ample for a-shortrun and not necessarily enough for a full one on a cold cache.
One test, or one subtest
Subtests are named as sentences, and go test replaces spaces with underscores in the -run pattern:
$ go test -mod=vendor -count=1 -v -run TestCargoshipSign ./test/e2e/noncluster/...
$ go test -mod=vendor -count=1 -v -run 'TestCargoshipSign/re-signing_requires_overwrite' ./test/e2e/noncluster/...
-run takes a regular expression matched against each path element, so -run 'TestCargoship(Publish|Pull)' selects a couple of suites and -run 'TestCargoshipSign/.*overwrite.*' selects by substring when the exact name is a mouthful.
Those patterns are unanchored, which bites here because several suite names are prefixes of others: -run TestCargoshipSign also runs TestCargoshipSignedRoundTrip, and -run TestCargoshipPull also runs TestCargoshipPullHTTP. Anchor both elements to isolate exactly one subtest:
$ go test -mod=vendor -count=1 -v -run '^TestCargoshipSign$/^re-signing_requires_overwrite$' ./test/e2e/noncluster/...
Environment variables
-
CARGOSHIP_E2E_TMPDIR- parent directory for the temp dirs the harness creates, one pere2e.Cargoshipcall, plus the shared minimal package. Unset means the system temp directory. Point it atbuild/tmpto keep all test scratch inside the repo, which makes it easy to see what a run left behind:CARGOSHIP_E2E_TMPDIR=$PWD/build/tmp TMPDIR=$PWD/build/tmp go test .... -
TMPDIR- respected by the binary itself for anything it does not put under its own staging directory. Worth setting alongside the above for the same reason. -
CARGOSHIP_E2E_BACKEND- what provisions the cluster suite’s hosts:bootloose(the default) for privileged containers,microvmfor local Rocky 10 virtual machines, orinventoryfor hosts that already exist. The VM backend is the only way the SELinux, fapolicyd and firewalld phases run as the paths they are rather than as gates that correctly do nothing, and it needs KVM. It is additional coverage rather than a replacement, since it runs one distribution; see microvm. -
CARGOSHIP_E2E_VM_MEMORY- withCARGOSHIP_E2E_BACKEND=microvm, the per-node memory in MiB. The suite brings its fleet up at the size the join walk needs rather than the size the apply walk uses, so the default of 4096 asks for 16GiB at once and the bring-up refuses rather than letting the OOM killer choose which process dies. A value that is not a positive number is ignored; see microvm. -
CARGOSHIP_E2E_INVENTORY- withCARGOSHIP_E2E_BACKEND=inventory, the path to aZarfClusterdocument naming hosts that already exist. The suite provisions nothing and tears nothing down, which is how the phase assertions are run against a fleet on EC2, vSphere, Proxmox or bare metal. It changes those hosts and does not change them back; see e2e-external-inventory. -
CARGOSHIP_E2E_KEEP_REGISTRY_LOG- set to any non-empty value to keep the in-memory registry’s request log for passing tests as well as failing ones. See “Artifacts and logs” below. -
CARGOSHIP_CONFIG- the config file the binary loads.TestCargoshipCreateExamplesets it (totest/e2e/cargoship-config.yaml), and the Ansible module suite’s “reports a broken config file as a module failure” case sets it to a deliberately malformed file to check that a broken config surfaces as a module-level failure rather than a crash; every other test passes flags explicitly so that what is being tested is visible in the test.
Seeing what the binary actually did
e2e.Cargoship runs the binary with exec.PrintCfg(), so its stdout and stderr are written through to the test process’s stdout and stderr, not into t.Log. go test buffers that per package and prints it only when the package fails; add -v to see it as it happens.
One flag is appended to every invocation automatically: --no-color, so assertions are matching plain text. Each invocation also gets a fresh staging directory, passed as DISTRO_TMP_DIR and removed when the call returns. It is the environment variable rather than --tmpdir because only the commands that stage package content register that flag - passing it to version or vault fails argument parsing.
Artifacts and logs
- In-memory registry log. The registry used by the publish, pull and sign tests writes one line per HTTP request to a file under the user cache directory,
~/.cache/cargoship/e2e-logs/registry-<timestamp>.log($XDG_CACHE_HOME/cargoship/e2e-logsif that is set), named the way the CLI names its own log files inlogs/, rather than to stderr where it would bury the test output. A passing test deletes its log; a failing one keeps it and prints the path in the failure output. SetCARGOSHIP_E2E_KEEP_REGISTRY_LOG=1to keep the logs of passing tests too - the path is then printed bygo test -vfor every test that started a registry. That log is usually what explains a publish or pull that failed for a non-obvious reason. - Example packages.
TestCargoshipCreateExamplewrites wheretest/e2e/cargoship-config.yamlpoints it,test/e2e/. Those.tar.zstfiles are gitignored, and they are large - delete them when done. - Per-call temp dirs are removed when each
e2e.Cargoshipcall returns, including on failure, so nothing the binary wrote underDISTRO_TMP_DIRsurvives for inspection. To keep it, reproduce the command by hand as described below.
Debugging a failure
The most useful property of this suite is that every test is a CLI invocation. When one fails, take the command from the test output and run it yourself against the same binary:
$ ./build/cargoship_linux_amd64 sign path/to/package.tar.zst --signing-key cosign.key --verify=always -k cosign.pub --log-level debug
Now nothing is cleaned up, --log-level debug is available, and you can iterate without recompiling anything.
A few things that have cost time before:
--verifyneeds its value attached. It is declared with a cobraNoOptDefVal, so--verify alwaysparsesalwaysas a positional argument and fails withaccepts 1 arg(s), received 2. Use--verify=always.- A stale binary produces failures that make no sense against the source you are reading. Rebuild, then rerun with
-count=1. - Signing keys are generated per test, so two calls to the keypair helper deliberately produce unrelated keys. A “verification failed” in a test that expects success usually means the wrong pair was threaded through.
To step through the harness itself (not the binary, which is a separate process), delve works on the test package as usual:
$ dlv test ./test/e2e/noncluster -- -test.run 'TestCargoshipPull' -test.v
Running the Fuzz Tests
The fuzz package holds Go’s native fuzz targets. It is not an e2e suite and does not sit with them: it does not drive the built binary and starts no containers, and the targets call the cargoship packages in process, so they run at tens of thousands of executions per second and reach the encoding decisions – scalar style, quoting, indentation, byte offsets – where a wrong answer produces a file that still parses and fails much later. Nothing here needs build/, a cluster, or the network.
It also has to stay out of src/test/. OpenSSF Scorecard’s file walker discards every path beginning src/test/ before any check reads it – isTestdataFile in checks/fileparser/listing.go, which carries the Maven src/test/java convention – so for as long as these targets lived at src/test/e2e/fuzz the Fuzzing check scored 0 and reported the project as not fuzzed. Moving the package back under src/test/ would silently take it to 0 again. The e2e suites themselves moved out of src/test/ to test/ for unrelated reasons, which loses that same exclusion for their tree – worth knowing if a future Scorecard run starts flagging something in test/ that it did not before.
Layout
One file per concern: 49 targets across 24 target files. The shared scaffolding:
fuzz/main_test.go package documentation and TestMain
fuzz/keyring_test.go the shared keyrings, one per format, built once in TestMain
fuzz/testdata/fuzz/<Target>/ committed crashers, one directory per target
fuzz/AGENTS.md the naming rule: no target name may be a prefix of another
The targets, grouped by what they guard:
vault_value_fuzz_test.go EncryptValue/DecryptValue/FormatOf round trips and salting
vault_path_fuzz_test.go EncryptAtPath/DecryptAtPath/RekeyAtPath and path canonicalisation
vault_document_fuzz_test.go the splice against varying document shapes, arbitrary documents, arbitrary paths
vault_config_fuzz_test.go EncryptConfig/DecryptConfig over a whole document
vault_meta_fuzz_test.go the recipients recorded in a document's vault metadata
keymaterial_fuzz_test.go AgeRecipientsIn, ResolveKeyring, RekeyTarget
parse_recipients_fuzz_test.go age recipient lines and recipient lists
cfg_fuzz_test.go cfg.Parse and cfg.ParseMultiDoc over arbitrary document bytes
clustercfg_parse_fuzz_test.go the cluster inventory parser
schema_decode_fuzz_test.go schema decoding
url_and_config_fuzz_test.go ExtractBasePathFromURL and ReadByteStrict over a distro config
dns_fuzz_test.go ParseServiceURL and IsLocalhost
layout_path_fuzz_test.go the path sanitisers: IsContainedPath and isCleanPath
parse_manifest_fuzz_test.go OCI manifest parsing
parse_checksum_fuzz_test.go checksum lines and Retry-After headers
parse_registry_overrides_fuzz_test.go the --registry-override key=value list
file_overrides_fuzz_test.go file override parsing and resolution
identify_source_fuzz_test.go package source identification
helmvalues_parse_fuzz_test.go values file shape and schema external-ref rejection
helmvalues_path_fuzz_test.go value path round trips, mapping application, merge immutability
helmvalues_template_fuzz_test.go template rendering, including that it denies host state
nft_split_families_fuzz_test.go nftables family splitting
ansible_inventory_fuzz_test.go the projected request, one host variable, the role mapping
ansiblemod_fuzz_test.go module argument reading and module naming
The Ansible inventory targets
ansible_inventory_fuzz_test.go fuzzes internal/ansibleinv, which is the translation an Ansible module runs on input it did not write. The vault targets guard what a wrong answer does to a file; these guard what a wrong answer does to a cluster, so their properties are about topology rather than encoding:
FuzzAnsibleRequesttakes the JSON an action plugin projects, decodes it, and translates it. The first host in the result has to be a controller, becauseConfigureEnginemakes the first controller the leader; every host has to reach the document with an address, a user and a non-zero port; and translating the same input twice has to produce the same bytes, since the walk goes over Go maps.FuzzAnsibleHostVarvaries one host variable against an inventory that is otherwise fixed. A variable under thecargoship_prefix that cargoship does not read is refused by name. A variable outside the prefix that cargoship does not read leaves the generated document byte-for-byte unchanged.FuzzAnsibleRoleGroupsvaries the role mapping and the group names it points at. No host reaches the document twice, and controllers come before workers.
They need no key material, but they are not all fast: FuzzAnsibleRequest translates a request and manages tens of thousands of executions per second, while FuzzAnsibleHostVar translates the inventory twice on every execution, with and without the variable, and manages under a thousand. Measured numbers are in the table under Running; a minute each is a real sweep for the first two and a thin one for FuzzAnsibleHostVar.
Both formats, every time
cargoship writes two ciphertext formats, Ansible Vault and age, and one keyring can carry both. Every target that encrypts therefore runs its body once per format rather than picking one: encryptKeyrings() returns the two encrypting keyrings and rekeyPairs() returns the four rekey directions (vault to vault, vault to age, age to age, age to vault). Both are in keyring_test.go, and both are built once in TestMain so that key generation is not paid per execution.
TestMain also unsets CARGOSHIP_AGE_IDENTITY_FILE, CARGOSHIP_AGE_RECIPIENTS, CARGOSHIP_VAULT_PASSWORD and ANSIBLE_VAULT_PASSWORD before it builds anything. A developer who has those set in their shell would otherwise be fuzzing against different key material than CI, which is the kind of difference that makes a reported crasher unreproducible.
Running
A plain go test runs the seed corpus only – the f.Add values in each target, plus anything committed under testdata/fuzz/<Target>/. That is about half a second and is what mage test:fuzz and go test ./... do. .github/workflows/unit-tests.yaml runs it on every pull request, as a second step beside test:unit, because it needs nothing that step did not already need:
$ mage test:fuzz
$ go test -mod=vendor -count=1 ./fuzz/ # the same thing, spelled out
Actual fuzzing needs -fuzz, which takes one target at a time and runs until it finds a failure or the clock runs out:
$ go test -mod=vendor -count=1 -run=XXX -fuzz=FuzzDecryptAtPathRoundTrip -fuzztime=60s ./fuzz/
-run=XXXmatches no unit test, so the run spends its whole budget on the target rather than on the rest of the package.-fuzztimedefaults to forever. Give it a value unless you mean to sit and watch.-fuzzminimizetimebounds the shrinking pass that runs after a failure is found. The default is fine; raise it when the reported input is bigger than it needs to be.-count=1disables the test cache, which otherwise makes a re-run of the seed corpus a no-op.
Because only one target runs per invocation, a sweep is a loop:
$ for t in $(grep -ho '^func \(Fuzz[A-Za-z]*\)' fuzz/*_test.go | cut -d' ' -f2); do
go test -mod=vendor -count=1 -run=XXX -fuzz=$t -fuzztime=5m ./fuzz/ || break
done
Expect very different throughput between targets. The ones that only push strings through a parser run at tens of thousands of executions per second; the ones that encrypt a value run the Ansible Vault key derivation on every execution and manage fewer than two hundred; the inventory targets fall between the two, since generating a document is cheaper than deriving a key and dearer than parsing a path. The spread across a 30 second budget is roughly three orders of magnitude:
FuzzRekeyTargetPicksOneKey 1,806,844 no encryption, key selection only
FuzzPathIsDecryptable 2,033,626 path parsing only
FuzzDecryptValueRejectsGarbage 2,531,710 almost every input is rejected before any key derivation
FuzzAnsibleRequest 401,831 decode, translate, translate again, encode both to compare bytes
FuzzAnsibleRoleGroups 93,138 one translation over a varying role mapping
FuzzAnsibleHostVar 24,171 two translations per execution, both encoded to YAML on the unchanged path
FuzzSpliceAcrossDocumentShapes 3,664 one vault encryption per execution
FuzzDecryptAtPathRoundTrip 2,082 encrypt, splice, decrypt, re-encrypt
FuzzRekeyAtPathPreservesValue 1,498 four rekey directions per execution
Both kinds are useful, but a target that encrypts needs minutes where a parsing target needs seconds. The cost is Ansible Vault’s PBKDF2 and is not worth engineering around: it is the same derivation an operator pays, and lowering it would mean fuzzing against key material the product does not use.
The corpus
There are two corpora, and only one of them is in the repository.
Inputs the fuzzer generates live in the build cache, under $(go env GOCACHE)/fuzz/. They are shared across runs on the same machine, are not portable, and are not meant to be committed; go clean -fuzzcache discards them, which is worth doing when a target has been rewritten and its cached corpus is exercising a shape that no longer exists.
Failing inputs are the committed corpus. When a target fails, go test writes the input to fuzz/testdata/fuzz/<Target>/<hash> and prints the path. Commit that file. From then on it is replayed by a plain go test, which is how a crash found by a long fuzz run becomes a permanent regression test that costs milliseconds. There are 24 committed crashers across 9 targets, the bulk of them under FuzzEncryptAtPathArbitraryDocument (12) and FuzzDecryptAtPathRoundTrip (3).
Every one of them is fixed – go test ./fuzz/ is green. A red run here means a new defect, not a known one. The five that came out of writing the original targets are worth reading as a sample of the shape of defect this package finds:
FuzzDecryptAtPathRoundTrip/89831cc049267b2c string("\n") a credential of one line break, written as an empty block scalar, read back as ""
FuzzPathIsDecryptable/477bd66902831e69 string("0[0") an unclosed index, which panicked inside go-yaml instead of erroring
FuzzPathIsDecryptable/ebc8a2cafef15425 string("'$'") a path whose canonical spelling, "$.$", the parser then rejects
FuzzEncryptAtPathArbitraryDocument/d370f7a72740b34b a flow mapping holding a bare entry, into which the splice wrote a block scalar
FuzzAnsibleRequest/1aea00cb191b2f7c a group listing a host with an empty name, which became an address, a hostname and a hostvars key
Two are worth the detail. The Ansible one: a group entry of "" produced a host whose address, hostname and hostvars key were all the empty string, and the schema accepted all three because each asks for a string; nothing downstream could tell that host from one an operator meant to install. deriveRoles now refuses an empty host name and an empty group name in the mapping.
The flow-mapping one: EncryptAtPath wrote a literal block scalar into a flow mapping such as {0, pass: 00}, producing a document that no longer parses, with the credential already encrypted into it and the plaintext gone. inFlowCollection in internal/clustercfg/vaultpath.go is what catches this, and it missed when a bare entry – a key with an implicit null value – preceded the target key. That is fixed; the corpus entry is what keeps it fixed.
Those same failures are also written up as ordinary table cases next to the code they broke – see TestEncryptAtPathReadsBackWhatDecryptWrote and TestCanonicalYAMLPath in internal/clustercfg/vaultpath_test.go. Keep doing both: the corpus file is what stops the target regressing, and the named case with a comment is what explains the defect to the next reader.
Writing a target
A target is a function taking *testing.F, seeded with f.Add and run with f.Fuzz. The fuzzed arguments may only be []byte, string, the integer and float types, bool, and rune – no structs, no slices of anything else. Build the structure you need inside the body from those pieces.
func FuzzEncryptValueRoundTrip(f *testing.F) {
f.Add("hunter2")
f.Add("")
f.Add("\xff\xfe not valid utf-8")
f.Fuzz(func(t *testing.T, value string) {
encrypted, err := clustercfg.EncryptValue(value, fuzzKeyring)
require.NoError(t, err)
decrypted, err := clustercfg.DecryptValue(encrypted, fuzzKeyring)
require.NoError(t, err)
require.Equal(t, value, decrypted, "value did not survive the round trip")
})
}
Four properties are worth stating, and each of them has already caught a real defect in this repo:
- Round trip.
decode(encode(x)) == x. Caught a credential holding a control character, whichdecrypt-pathwrote as a"\xNN"escape thatencrypt-paththen refused to measure, and a credential of one line break, which was written as an empty block scalar and read back as the empty string. - Idempotence, and output that is legal input.
f(f(x)) == f(x), andf’s output parses. CaughtCanonicalYAMLPathturning'$'into$.$, a spelling the parser rejects, which was then handed to the rest of the run. - Two code paths that must agree. A predicate against the thing it predicts, or a validator against its consumer.
FuzzPathIsDecryptablestates that every pathvault encrypt-pathaccepts canonicalises to one the apply-time decryptor visits, because a path that fails that test vaults a credential nothing unwraps. - No panic on untrusted input. Anything reached by a hand-edited file or a command-line argument. Caught go-yaml reading past the end of a path whose index is never closed –
registries[0– which reached the operator as a stack trace rather than a usage error.
Seed with the shapes the value actually takes plus the ones most likely to be mishandled: empty, whitespace a trim would eat, a PEM block, a control character, invalid UTF-8, and a real fixture such as test/e2e/noncluster/testdata/inventory-vault.yaml. Seeds are also the whole of what CI runs, so a seed is the cheapest place to pin a shape that matters.
Keep the body fast and self-contained. A target that touches the network or a real file drops from tens of thousands of executions per second to hundreds and finds nothing in the time it is given; use t.TempDir() where a file is unavoidable.
Include the input in failure messages. require.NoError(t, err, "%q canonicalises to %q, which does not parse", path, canonical) names the defect in the failure line; without it the message says only that something did not parse, and the input has to be dug out of the corpus by hand.
Gotchas
- A target whose oracle is regenerated per run will disagree with the cached corpus. Vault ciphertext is salted, so a value encrypted now does not equal the same value encrypted in the run that produced the corpus entry. State the property (
cluster.IsVaultEncrypted(blob)) rather than comparing against a freshly computed ciphertext. - Reading a single YAML path is not the same as decoding the document.
goyaml.Path.Readdrops the trailing newline of a clipped block scalar wheregoyaml.Unmarshalkeeps it, which fails a PEM round trip for a reason that has nothing to do with the code under test. Decode intocluster.Cluster, which is also what an apply does. - A failing seed stops the run before the fuzzing starts.
failure while testing seed corpus entrymeans the target is red on input it was given, not on input it found. Fix that first; until it is fixed the target is not fuzzing at all. - A target that varies the document shape has to vary it the way a real file does.
blockScalarinvault_document_fuzz_test.gotakes a chomp indicator because the TLS CA neighbour is a PEM block that keeps its trailing newline; rendering it with|-strips that newline and fails the neighbour assertion for a reason that has nothing to do with the splice. - Line endings are deliberately not one of the shape knobs. A document saved with CRLF is one the splice mishandles today:
DecryptAtPathrefuses a multi-line value in such a document, andspliceScalarwrites bare line feeds into it when it does splice one. Fuzzing that axis reports the same known defect on every input rather than finding a new one. Put the knob back once the splice carries the document’s own line ending.
Where to look next
Fuzzing pays where a wrong answer still parses, so the targets worth writing next are the places that choose an encoding, canonicalise a string, or hand a path to a parser:
distrocfg.marshalRegistriesYAMLandquoteRegistryKeys(types/distrocfg/distro_common.go) choose a quoting style for keys the engine reparses – the same shape as the bugs above, over registry names andrewriteregex patterns.
GoReleaser Release Pipeline
This document describes how Cargoship turns a pushed tag into published artifacts: archives, Linux packages, container images, and signatures. The pipeline is defined by .goreleaser.yaml at the repository root and driven by .github/workflows/release.yaml.
This is the release path. It is distinct from the Mage pipeline, which builds binaries for local development and CI checks.
Trigger
The workflow runs on any pushed tag. Tags are normally created by release-please, so the practical sequence is: merge the release PR, release-please pushes the tag, the goreleaser workflow fires.
What Gets Published
A single goreleaser release run produces:
- Archives -
tar.gzper OS/arch, named afterunameconventions (cargoship_Linux_x86_64.tar.gz). - Packages -
apk,deb, andrpm, from thenfpmssection. Packages can be signed when the corresponding signing keys are configured. - Container images - four multi-platform manifests covering
linux/amd64andlinux/arm64:ghcr.io/colonel-byte/cargoship:<tag>(Chainguard static, bare binary),ghcr.io/colonel-byte/cargoship-ubi:<tag>(AlmaLinux, installs the RPM),ghcr.io/colonel-byte/cargoship-deb:<tag>(Debian slim, installs the DEB), andghcr.io/colonel-byte/cargoship-ansible:<tag>(AlmaLinux, installs the RPM and Ansible dependencies). - Ansible collection - a tarball,
colonel_byte-cargoship-<version>.tar.gz, built fromansible/colonel_byte/cargoshipand attached to the release with its own cosign bundle. It is not published to Galaxy: the management node that runs the playbooks is inside the airlock and installs it from the tarball. - SBOMs - one per archive, via Syft.
- Signatures - cosign
sigstore.jsonbundles over the checksums file and every archive, plus registry signatures over every container image. Package signatures are configured separately through nfpm.
The Ansible Collections
Neither collection is built by a builds entry, because nothing about a collection is compiled. hack/ansible-build-collection.sh runs as a before hook - once per collection, so twice, taking the collection directory as its third argument - calls ansible-galaxy collection build, writes the tarball to bin/ansible, and signs it when COSIGN_PRIVATE_KEY is set. Four release.extra_files globs then attach both tarballs and both bundles.
The script is given the release version and fails when that collection’s galaxy.yml disagrees with it. release-please keeps both versions in step through the extra-files entries in release-please-config.json, so a disagreement means the release PR did not update what it should have, and publishing a collection whose version is not the one being released is worse than a failed release. A snapshot passes no version, because a snapshot’s version is not in galaxy.yml and never will be.
The tarball is signed by the script rather than by the signs section, and is absent from checksums.txt, because both of those cover artifacts the pipeline itself produced. Verify it with its bundle:
cosign verify-blob --key cosign.pub \
--bundle colonel_byte-cargoship-<version>.tar.gz.sigstore.json \
colonel_byte-cargoship-<version>.tar.gz
Build Matrix
The builds section has one entry. cargoship (./cmd/cargoship) compiles linux, darwin and windows against amd64, arm64 and riscv64. Three settings there matter more than they look:
CGO_ENABLED=0- produces a statically linked binary. This is what lets the default image sit oncgr.dev/chainguard/static, which ships no libc, no shell, and no package manager; a dynamically linked binary fails there with a confusingno such file or directoryon exec.ldflags -X- stampsconfig.CLIVersionandconfig.CLICommit, which is whatcargoship versionprints.mod_timestamp: {{ .CommitTimestamp }}- pins file mtimes to the commit for reproducibility.nfpms.mtimeand the image’sorg.opencontainers.image.createdlabel use.CommitDatefor the same reason.
Container Images
Images are configured under dockers_v2, which replaced the older dockers + docker_manifests pair. GoReleaser emits every platform and the manifest list that fans out to them from a single docker buildx build --push.
Three consequences are worth internalizing:
- Images build during the publish phase, not the build phase.
goreleaser buildandgoreleaser release --skip=publishnever touch them. Use--snapshotto exercise the Dockerfile. - A multi-platform-capable buildx builder is required. The default
dockerdriver cannot produce a manifest, hence thesetup-buildx-actionstep in the workflow. - Snapshots emit per-platform tags (
:<tag>-amd64,:<tag>-arm64) rather than a manifest, because buildx cannot create a manifest without pushing. Real releases publish the single manifest tag.
dockers_v2 is still flagged experimental upstream and prints a warning on every run. It becomes plain dockers in GoReleaser v3.
The Dockerfile Contract
All four Dockerfiles live under containers/ - containers/base/Dockerfile, containers/ubi/Dockerfile, containers/deb/Dockerfile, and containers/ansible/Dockerfile - and each is named explicitly by its dockers_v2 entry’s dockerfile: key. None compiles anything: GoReleaser has already built the binaries and packages by the time they run. The static image copies the binary; the UBI and Ansible images install the RPM, while the Debian image installs the DEB. Adding a compiling stage to any of them would duplicate that work and break the guarantee described under Binary Identity.
The directory name is deliberately generic: containers/base/ is the plain-binary image whatever base it happens to sit on, so a future base swap does not force another rename. If it is ever renamed anyway, three other files have to move with it in the same commit - the dockerfile: key in .goreleaser.yaml, the lint matrix in .github/workflows/scan-lint.yaml, and the directories: list of the docker ecosystem in .github/dependabot.yaml.
dockers_v2 hands the Dockerfile a temporary context laid out one directory per platform, which is why the binary is copied from $TARGETPLATFORM rather than the context root:
<context>/Dockerfile
<context>/linux/amd64/cargoship
<context>/linux/arm64/cargoship
The source tree is not in that context. Anything else the image needs must be declared in extra_files.
The UBI file is single-stage: a FROM, a COPY of the rpm, and RUNs that install it and lay out the unprivileged user’s directories.
The static file has to be two-stage. cgr.dev/chainguard/static carries no shell and no package manager, so the runtime image cannot mkdir, chmod, or chown anything for itself. A small Alpine skeleton stage builds the directory tree under /skeleton and the runtime stage copies it in. That stage creates empty directories only - nothing architecture-specific - so it is pinned to --platform=$BUILDPLATFORM and never runs under emulation.
Copy the skeleton at its root, not directory by directory:
COPY --from=skeleton /skeleton/ /
A COPY whose destination directory does not yet exist creates that directory root-owned and carries ownership only on the entries beneath it. COPY --from=skeleton /skeleton/workspace /workspace therefore produces a root:root /workspace that uid 65532 cannot write to - the failure surfaces at runtime, not at build time. Merging the tree into / preserves every directory’s own uid, gid, and mode.
A $BUILDPLATFORM stage is not automatically cheaper, and speed is not why the split exists. On the older Alpine-based version of this file, hoisting its one RUN into a build-platform stage measured slower on a --no-cache linux/arm64 build (~1.9s vs ~1.3s): busybox adduser/mkdir under QEMU cost less than an extra stage and its COPYs. The split is here because the runtime base cannot execute anything at all.
Base Image Pinning
Every FROM pins its base to a digest, in tag@sha256:... form:
FROM cgr.dev/chainguard/static:latest@sha256:<index digest>
The tag stays in the reference for readability; the digest is what actually resolves. Keep the digest a manifest-list (index) digest, not a per-platform one, or the linux/arm64 build will fail to find its platform:
docker buildx imagetools inspect cgr.dev/chainguard/static:latest --format '{{ .Manifest.Digest }}'
One convention here exists to keep dependabot working, and it looks redundant until you know why:
- The reference is repeated on every
FROMrather than hoisted into anARG. Dependabot’s Dockerfile parser reads literalFROMlines only - it has noARGresolution, soFROM ${BASE_IMAGE}is invisible to it and the pin would never be refreshed. It is also not stage-aware, so aliasing (FROM alpine@sha256:... AS basethenFROM base) is no better: the bare stage reference parses as an image and produces a spuriousbase:latestdependency.
The images carry no LABEL instructions of their own. Every org.opencontainers.image.* label, base.name and base.digest included, is set by .goreleaser.yaml from the {{ .BaseImage }} and {{ .BaseImageDigest }} templates goreleaser resolves at build time. That is why there is nothing here to drift and no check for it: a hand-written label could disagree with the FROM it describes, and a templated one cannot.
One base needs more care than the others. cgr.dev/chainguard/static publishes :latest and nothing else on Chainguard’s free tier, and old digests are garbage-collected as the tag moves. A stale pin there does not merely go out of date - it eventually stops resolving, and the build fails with manifest unknown. That is the failure to expect if this image breaks without the Dockerfile changing; refresh the digest with the command above.
.github/dependabot.yaml tracks all four Dockerfile directories - /containers/ansible, /containers/base, /containers/deb, /containers/ubi - under its docker ecosystem, on the same daily schedule as the other ecosystems, which is what keeps that pin inside the retention window.
Linting
Dockerfiles are linted by hadolint in two places:
.pre-commit-config.yaml- thehadolint-dockerhook, which runs the linter in a container and so needs a working Docker daemon. It picks up any file pre-commit identifies as a Dockerfile, excludingvendor/..github/workflows/scan-lint.yaml- thecontainersjob, a matrix over the four Dockerfile paths.
The CI job lists paths explicitly rather than globbing. The action’s recursive mode searches from the repository root, which would pull in the vendored Dockerfiles under vendor/ - several of which do not pass. Add a new path to that matrix when adding an image variant; the pre-commit hook picks it up on its own.
Default rules and the default info failure threshold apply; there is no .hadolint.yaml.
RPM Variant
A second dockers_v2 entry builds containers/ubi/Dockerfile and publishes ghcr.io/colonel-byte/cargoship-ubi:<tag>. It is based on AlmaLinux and installs the signed Cargoship RPM so the package is registered in the image’s RPM database.
The static image copies the binary directly; this image installs the same RPM produced by the nfpms section.
$ docker run --rm --entrypoint rpm \
ghcr.io/colonel-byte/cargoship-ubi:<tag> -q cargoship
cargoship-<version>-1.x86_64
rpm -V cargoship and rpm -ql cargoship work as well, which is the main
reason to use an RPM-based image.
Details worth knowing before editing it:
ids: [packagers]selects the nfpms artifacts for the build context instead of the raw binary. All three formats share one nfpms id, so the.apkand.debalso land in the context; the Dockerfile selects the target-platform RPM.- The RPM signature is verified at build time.
crypto/software@conlon.dev.gpgis supplied throughextra_files, imported into the image keyring, and checked withrpm --checksigbefore installation. - The install step must run on the target platform. RPM unpacks an architecture-specific binary into the target root filesystem, so the install cannot be moved to a
$BUILDPLATFORMstage. This is why thelinux/arm64build requires QEMU. - The binary lands in
/usr/binaccording tonfpms.rpm.prefixes, not/usr/local/binas in the static image. The entrypoint differs accordingly; thenonrootaccount,$HOME, and/workspacelayout remain compatible. - Package versions are pinned via Dockerfile
ARGdefaults (BASH_COMPLETION_VERSION, andANSIBLE_CORE_VERSIONfor the ansible image) and explicitly passed in.goreleaser.yamlunderbuild_argsto comply with HadolintDL3041and ensure reproducible image builds. Runmage dev:dnfPinsto query the AlmaLinux repodata and refresh the pins when upstream packages roll forward. Only packages an image still installs are pinned, so dropping an install means dropping its pin frommagefiles/pkg/devtools/dnfpinstoo.
The AlmaLinux base is substantially larger than the static image. Recheck exact image sizes after base-image updates rather than relying on a fixed comparison.
Debian Variant
containers/deb/Dockerfile, published as ghcr.io/colonel-byte/cargoship-deb:<tag>, is the Debian-family counterpart to the RPM image. It installs the DEB produced by the nfpms section with dpkg, registering Cargoship in the image’s dpkg database.
$ docker run --rm --entrypoint dpkg \
ghcr.io/colonel-byte/cargoship-deb:<tag> -s cargoship
Package: cargoship
Status: install ok installed
dpkg -L cargoship and dpkg -V cargoship work as well, which is the main reason to use a Debian-based image.
Its other job is coverage: the .deb is produced by every release, so this image exercises installation of that package format during the image build.
Details worth knowing before editing it:
- The directory is named for the package format, not the base. This image exists to exercise the
.deb; changing Debian to Ubuntu would not change that purpose. dpkg -i, notapt-get install. The package declares no dependencies, so nothing needs resolving and the build remains hermetic.- The binary lands in
/usr/bin, as it does in the RPM-based images. dpkg -idoes not verify package signatures. The_gpgoriginmember is available fordebsig-verifyand repository tooling, but is not checked by this Dockerfile.
Ansible Variant
A third dockers_v2 entry builds containers/ansible/Dockerfile and publishes ghcr.io/colonel-byte/cargoship-ansible:<tag>. It provides a management-node image based on AlmaLinux.
The image installs the signed Cargoship RPM, ansible-core, and the collections listed in ansible/colonel_byte/cargoship/requirements.yml. The collection is installed under Ansible’s system collection path:
/usr/share/ansible/collections/ansible_collections
The container is the management node: Ansible and Cargoship run inside it, while Cargoship opens SSH connections from the container to the external fleet.
Details worth knowing before editing it:
ids: [packagers]supplies the generated package artifacts. The Dockerfile selects the target-platform RPM and ignores the.apkand.debfiles that share the nfpms output.- The RPM signature is verified during the build using
crypto/software@conlon.dev.gpg, which is supplied throughextra_files, imported into the image keyring, and checked before installation. - Package versions for
ansible-coreandbash-completionare pinned via DockerfileARGdefaults (ANSIBLE_CORE_VERSIONandBASH_COMPLETION_VERSION) and explicitly passed in.goreleaser.yamlunderbuild_argsto comply with HadolintDL3041and ensure reproducible image builds. Runmage dev:dnfPinsto query the AlmaLinux repodata and refresh both pins when upstream packages roll forward. - Ansible dependencies come from
requirements.yml. The Dockerfile runsansible-galaxy collection installand places the collections in the system collection path. - There is no
ENTRYPOINT. The image is intended to runansible-playbook,ansible-inventory,ansible-galaxy, or Cargoship as needed. Its default command isansible-playbook --help. - The build validates the installed Cargoship package by running
cargoship version.
docker run --rm \
-v "$PWD:/workspace" \
ghcr.io/colonel-byte/cargoship-ansible:<tag> \
ansible-playbook -i inventory.yml site.yml
Image Runtime Layout
- Runs as uid/gid
65532, which has a real/etc/passwdentry namednonrootwith home/home/nonrootin all four images, soos/userlookups and$HOMEagree with each other and across variants. The static image inherits that account from its base; the package-installing images create it to match. TheUSERline stays numeric (65532:65532) so the image still runs correctly under arunAsUserthat ignores names. HOME=/home/nonroot, which backs viper’s$HOME/.zarfconfig search path (cmd/viper.go) and the default cache path~/.cargoship-cache(config/common.go). Both directories are pre-created and owned by65532, as is~/.sshat 0700.WORKDIR /workspace. Since.is viper’s first config search path, bind-mounting a package directory there picks upcargoship-config.yamlwith no extra flags:
docker run --rm -v "$PWD:/workspace" ghcr.io/colonel-byte/cargoship:<tag> apply ...
OCI Labels
Labels are split by whether they change per release. Static ones (title, description, source, licenses, base image) live in each Dockerfile; per-release ones (version, revision, created) are set in .goreleaser.yaml. See Base Image Pinning for why org.opencontainers.image.base.name deliberately omits the digest. Keep them separate - a --label flag silently overrides a LABEL of the same key, so duplicating a key across a Dockerfile and .goreleaser.yaml creates two sources of truth.
Package Signing
The nfpms packages carry signatures of their own, independent of cosign. That needs two key pairs, because nfpm signs rpm and deb with OpenPGP but signs apk with a bare RSA key:
| Format | Key type | Path variable | Repository secret |
|---|---|---|---|
| rpm, deb | OpenPGP | GPG_KEY_PATH | GPG_PRIVATE_KEY, and GPG_PASSPHRASE if the key is protected |
| apk | RSA, PEM | APK_RSA_KEY_PATH | APK_RSA_PRIVATE_KEY |
Handing the OpenPGP key to the apk signer does not degrade quietly - it fails the release with signing error: no PEM block found.
The public halves live in crypto/, both named after the maintainer address: crypto/software@conlon.dev.gpg (A1D4 DF69 80C8 2444 99C3 D1F9 96F5 5C80 CFB1 EFCB, expires 2046-09-14) and crypto/software@conlon.dev.rsa.pub. The latter’s filename is not a convention - it is the name apk itself looks for.
Generating the Keys
hack/gen-signing-keys.sh produces both pairs in the shapes nfpm accepts, building the OpenPGP key in a throwaway GNUPGHOME so your own keyring is untouched:
hack/gen-signing-keys.sh ~/cargoship-keys
It reads the maintainer address out of .goreleaser.yaml and names the apk public key after it, which is not cosmetic - see below. Set GPG_PASSPHRASE to protect the OpenPGP key. It refuses to overwrite existing files, because replacing a key already in use invalidates every package signed with it.
Constraints worth knowing before doing any of this by hand:
- The apk RSA key cannot be encrypted. nfpm understands only the legacy DEK-Info PEM encryption, so the PKCS#8 output of
openssl genrsa -aes256is rejected withkey type "ENCRYPTED PRIVATE KEY" is not supported, andNFPM_APK_PASSPHRASEcannot help. Its protection is the repository secret and nothing else. - The apk key name is baked into every package. nfpm defaults it to the maintainer address parsed from
nfpms.maintainer, records the signature as the tar member.SIGN.RSA.software@conlon.dev.rsa.pub, and apk then looks for a file of that name under/etc/apk/keys. Changingnfpms.maintainer, or settingapk.signature.key_name, breaks the path every existing consumer deployed. key_fileis a path, not key material. UnlikeCOSIGN_PRIVATE_KEY, which cosign reads straight from the environment, these must exist as files. The workflow writes both secrets under$RUNNER_TEMP- outside the checkout, so no.goreleaser.yamlglob can sweep them up - and removes them in anif: always()step.
Degrading to Unsigned
Both key_file values are guarded:
key_file: '{{ if ne (index .Env "GPG_KEY_PATH") "" }}{{ .Env.GPG_KEY_PATH }}{{ else }}{{ end }}'
The index .Env form is load-bearing. A bare {{ .Env.GPG_KEY_PATH }} on an unset variable aborts the run with map has no entry for key "GPG_KEY_PATH"; the guard turns an unset variable into an unsigned package instead. That is what a local snapshot wants and what a release does not, so the workflow emits a ::warning:: for each missing secret rather than letting the absence pass unremarked.
Verifying
# rpm
sudo rpmkeys --import 'crypto/software@conlon.dev.gpg'
rpm -K cargoship_<version>_linux_amd64.rpm # digests signatures OK
# apk - the filename is the name the signature records, so this is a plain copy
cp 'crypto/software@conlon.dev.rsa.pub' /etc/apk/keys/
apk add ./cargoship_<version>_linux_amd64.apk # no --allow-untrusted
# deb - the signature covers the three ar members concatenated, in order
ar x cargoship_<version>_linux_amd64.deb
cat debian-binary control.tar.gz data.tar.zst > combined
gpg --verify _gpgorigin combined
dpkg -i never checks that signature itself; _gpgorigin exists for debsig-verify and repository tooling.
Image Signing
docker_signs signs the published images with cosign, using the same COSIGN_PRIVATE_KEY / COSIGN_PASSWORD pair as the archive signatures - no extra workflow configuration is needed.
docker_signs:
- id: images
cmd: cosign
args: ["sign", "--key=env://COSIGN_PRIVATE_KEY", "${artifact}@${digest}", "--yes"]
stdin: "{{ .Env.COSIGN_PASSWORD }}"
Two details:
artifactsis deliberately unset. Its default targets the images built bydockers_v2; the named values (images,manifests) address the deprecateddockersanddocker_manifestspipes instead.${artifact}@${digest}, not${artifact}. GoReleaser captures the digest from the push and signs that exact image, so a concurrent push to the same tag cannot end up signed by mistake.
Unlike the archive signatures, this produces no local file - cosign pushes the signature to the registry beside the image, which is why the pipe runs at the very end of publishing. Verify a released image with:
cosign verify --key cosign.pub ghcr.io/colonel-byte/cargoship:<tag>
This pipe only runs on a real publish. A --snapshot run pushes nothing, so it never exercises signing.
Binary Identity
The binary is byte-identical everywhere it ships: the release archive, the apk/deb/rpm packages, the static image, the UBI image (where it arrives via rpm --install), the Debian image (dpkg --install), and the Ansible image (rpm --install). GoReleaser compiles once per target and reuses that one artifact downstream.
This means a cosign verification against the released archive transitively covers the binary shipped in the image. Verify it yourself after a snapshot run:
sha256sum dist/cargoship_linux_amd64_v1/cargoship
cid=$(docker create ghcr.io/colonel-byte/cargoship:<tag>-amd64)
docker cp "$cid:/usr/local/bin/cargoship" /tmp/from-image && docker rm "$cid"
sha256sum /tmp/from-image
The property holds only because no Dockerfile compiles anything.
CI Workflow Steps
Each setup step in .github/workflows/release.yaml exists for a specific reason:
- Syft - generates the archive SBOMs.
- ansible-core - supplies
ansible-galaxy, which builds the collection tarball in thebeforehook. Nothing else in the pipeline needs Ansible. - cosign - signs checksums, archives, and the published images; needs
COSIGN_PRIVATE_KEYandCOSIGN_PASSWORD. - Setup package signing keys - materializes
GPG_PRIVATE_KEYandAPK_RSA_PRIVATE_KEYas files under$RUNNER_TEMPand exports their paths, because nfpm takes a path rather than key material. See Package Signing. A pairedif: always()step deletes them. - QEMU - registers binfmt handlers for cross-platform builds. Required by the package-installing images:
rpm --installanddpkg --installunpack architecture-specific files and must run on the target platform, so theirlinux/arm64builds run under emulation. The static image does not need it - its onlyRUNis pinned to$BUILDPLATFORM. - Buildx - required, not optional; provisions the container driver that can emit a multi-platform manifest.
- GHCR login - required to push. The job’s
packages: writepermission is an authorization, not a credential. fetch-depth: 0- GoReleaser needs full history to resolve tags and changelog range.
Local Testing
Validate the config without building anything:
goreleaser check
Run the whole pipeline locally without publishing. Skip the signing and SBOM stages unless you have cosign keys in your environment:
TMPDIR=/home/$USER/.cache/goreleaser-tmp \
goreleaser release --snapshot --clean --skip=sign,sbom,announce,validate,before
Three gotchas:
- Set
TMPDIRto a path on a real disk. The default/run/user/<uid>/tmpis a small tmpfs, and the Go build cache for the full matrix - ninecargoshiptargets plus three apiece for the two zarf module files - will overrun it. The failure is a flood ofno space left on devicefrom the compiler. - Do not add
--skip=publishif you want images. Underdockers_v2that skips image builds entirely. --skip=signalso disables package signing, not just cosign. GoReleaser’s nfpm pipe zeroes all threeSignaturestructs when that skip is set (internal/pipe/nfpm/nfpm.go), so a run that skips signing produces unsigned packages no matter whatGPG_KEY_PATHandAPK_RSA_KEY_PATHsay. To exercise package signing locally, dropsignfrom the skip list and supply cosign keys too.
Artifacts land in dist/, which is gitignored.
Running the Test Targets Without Mage
Every mage test:* target is a thin wrapper around a go test plus a little bit of setup: an environment variable, a temp directory, a container cleanup pass. This page names exactly what each target does, so you can run the same check by hand – on a box without mage installed, in a script, or just to see what’s actually happening behind the wrapper. For what each suite covers and how to extend it, see e2e-tests, e2e-phase-tests, and fuzz-tests; this page is only about the wrapper.
mage.md’s Running Mage Directly section covers the other way to skip the installed mage binary – go run ./magefiles/core test:endToEndCluster runs the exact same Go function mage test:endToEndCluster would, just compiled and invoked directly. The commands below go one level further: they are what that function itself runs, with mage (and magefiles/core) out of the picture entirely.
The two shared helpers
Every EndToEnd* target funnels through one of two functions in magefiles/pkg/testrunner/testrunner.go:
RunE2E– builds the binary for the host’s OS/arch first, with the same flags and environmentmage build:binaryuses (magefiles/pkg/build/build.go):GOOS,GOARCHandCGO_ENABLED=0in the environment, andgo build -trimpath -gcflags=all="-l" -ldflags "-s -w -X ...CLIVersion -X ...CLICommit" -o build/cargoship_<goos>_<goarch> ./cmd/cargoship. There is no-a; see build-flags for why it was removed. Then it callsRunE2ENoBuild.RunE2ENoBuild– createsbuild/tmp, then runsgo test -timeout=<T> -count=1 -v <pkg>withCARGOSHIP_E2E_TMPDIRandTMPDIRboth pointed at that directory.
The manual commands below spell both steps out for each target, so -mod=vendor is included explicitly even though the mage helper relies on the repo’s go env GOFLAGS to add it.
Test.EndToEnd
$ ids=$(docker ps -aq --filter "label=io.k0sproject.bootloose.owner=bootloose"); [ -n "$ids" ] && docker rm -fv $ids
$ CGO_ENABLED=0 go build -trimpath -gcflags=all="-l" \
-ldflags "-s -w -X github.com/colonel-byte/cargoship/config.CLIVersion=unset-development-only -X github.com/colonel-byte/cargoship/config.CLICommit=$(git rev-parse --short HEAD)" \
-o "build/cargoship_$(go env GOOS)_$(go env GOARCH)" ./cmd/cargoship
$ mkdir -p build/tmp
$ CARGOSHIP_E2E_TMPDIR="$PWD/build/tmp" TMPDIR="$PWD/build/tmp" \
go test -mod=vendor -timeout=1h -count=1 -v github.com/colonel-byte/cargoship/test/e2e/...
Runs both e2e groups, including the example packages that pull real engine artifacts and images. Needs Docker.
Test.EndToEndNonCluster
$ CGO_ENABLED=0 go build -trimpath -gcflags=all="-l" \
-ldflags "-s -w -X github.com/colonel-byte/cargoship/config.CLIVersion=unset-development-only -X github.com/colonel-byte/cargoship/config.CLICommit=$(git rev-parse --short HEAD)" \
-o "build/cargoship_$(go env GOOS)_$(go env GOARCH)" ./cmd/cargoship
$ mkdir -p build/tmp
$ CARGOSHIP_E2E_TMPDIR="$PWD/build/tmp" TMPDIR="$PWD/build/tmp" \
go test -mod=vendor -timeout=30m -count=1 -v -short github.com/colonel-byte/cargoship/test/e2e/noncluster/...
The -short is what the mage target passes; drop it to include TestCargoshipCreateExample, which builds a real example package. Mirrors the e2e-noncluster CI job. No Docker needed.
Test.EndToEndCluster
$ ids=$(docker ps -aq --filter "label=io.k0sproject.bootloose.owner=bootloose"); [ -n "$ids" ] && docker rm -fv $ids
$ mkdir -p build/tmp
$ CARGOSHIP_E2E_TMPDIR="$PWD/build/tmp" TMPDIR="$PWD/build/tmp" \
go test -mod=vendor -timeout=1h -count=1 -v github.com/colonel-byte/cargoship/test/e2e/cluster/...
No binary build step – this suite calls the cargoship packages directly. Needs Docker. This is the full engine-bootstrap walk; CI does not run it, see e2e-phase-tests.
Test.EndToEndClusterStage
Same as EndToEndCluster, plus one env var that makes the suite stop before the engine starts:
$ ids=$(docker ps -aq --filter "label=io.k0sproject.bootloose.owner=bootloose"); [ -n "$ids" ] && docker rm -fv $ids
$ mkdir -p build/tmp
$ CARGOSHIP_E2E_TMPDIR="$PWD/build/tmp" TMPDIR="$PWD/build/tmp" CARGOSHIP_E2E_STAGE_ONLY=1 \
go test -mod=vendor -timeout=30m -count=1 -v github.com/colonel-byte/cargoship/test/e2e/cluster/...
This is the one CI actually runs, on every pull request.
Test.EndToEndClusterDryRun
EndToEndClusterStage’s setup with -run narrowing the suite to the dry-run walk:
$ ids=$(docker ps -aq --filter "label=io.k0sproject.bootloose.owner=bootloose"); [ -n "$ids" ] && docker rm -fv $ids
$ mkdir -p build/tmp
$ CARGOSHIP_E2E_TMPDIR="$PWD/build/tmp" TMPDIR="$PWD/build/tmp" CARGOSHIP_E2E_STAGE_ONLY=1 \
go test -mod=vendor -timeout=30m -count=1 -v -run 'TestClusterPhases/dryrun' github.com/colonel-byte/cargoship/test/e2e/cluster/...
Starts no engine, so it is the fastest way to exercise --dry-run against real hosts.
Test.EndToEndClusterUpgrade
EndToEndCluster with the upgrade walk turned on and a three-hour budget, because it installs a second complete set of engine images on every node:
$ ids=$(docker ps -aq --filter "label=io.k0sproject.bootloose.owner=bootloose"); [ -n "$ids" ] && docker rm -fv $ids
$ mkdir -p build/tmp
$ CARGOSHIP_E2E_TMPDIR="$PWD/build/tmp" TMPDIR="$PWD/build/tmp" CARGOSHIP_E2E_UPGRADE=1 \
go test -mod=vendor -timeout=3h -count=1 -v github.com/colonel-byte/cargoship/test/e2e/cluster/...
No CI job runs this; it is the only way to cover the upgrade phases.
Test.CleanZarfClusters
$ for c in zarf zarf-local-storage; do k3d cluster delete "$c" 2>/dev/null; done
The cleanup EndToEndZarf runs before it starts, exposed on its own. The cluster names are K3dClusters in magefiles/pkg/testrunner/testrunner.go. A cluster that is not there is not an error.
Test.CleanCluster
$ ids=$(docker ps -aq --filter "label=io.k0sproject.bootloose.owner=bootloose"); [ -n "$ids" ] && docker rm -fv $ids
No test runs; this is only the cleanup step the two cluster targets run before they start, exposed on its own for a run that was killed before teardown.
Test.Unit
$ go test -count=1 $(go list ./... | grep -v '^github.com/colonel-byte/cargoship/test/')
Every package except the e2e suites under test/, which have runners of their own. The target builds the list with go list rather than hardcoding it, so a new package is covered the moment it exists. No binary, no Docker, no temp-dir env vars.
Test.Fuzz
$ go test -count=1 ./fuzz/...
No binary, no Docker, no temp-dir env vars – this replays the committed seed corpus in process, which is the whole reason it’s a separate target from the suites above.
Mage Tasks and Automation
This document explains how Cargoship uses Mage to orchestrate build, test, and generation tasks.
Overview
Cargoship uses mage as its primary task runner and automation tool instead of a traditional Makefile. The implementation resides in the magefiles/ directory, which acts as the central repository for:
- Building binaries for the host and for the release target platforms.
- Running end-to-end (e2e) tests.
- Generating documentation from the codebase.
- Generating and publishing JSON schemas from Go types.
The entry point of the automation layer is magefiles/core/core.go, which bootstraps the runtime and exposes exported functions as Makefile-like executable targets.
Namespace Architecture
Mage targets are organized into logical Go namespaces to group related operations together. Every target is invoked as mage <namespace>:<target>, and target names are case-insensitive, so mage generate:engineconfig and mage generate:engineConfig are the same command. Run mage -l for the authoritative list on your checkout.
Build Namespace
The Build namespace is the default target and the build path for the Cargoship binary. It compiles natively with the host’s Go toolchain:
Binary- Compiles the binary for the host’s platform by default, or for a specified OS and architecture via-osand-archflags.All- Compiles all release binaries intobuild/.Examples- Builds a package from every example definition using the cargoship binary onPATH(not a binary built here), exercising the same path a release user would take. Gigabytes per package and hours in total; one failure does not stop the run.
mage build:binary # build for this host's OS/arch
mage build:binary -os=darwin -arch=arm64 # build build/cargoship_darwin_arm64
mage build:all # build every release binary into build/
mage build:examples # build every example package with cargoship on PATH
mage # same as `mage build:all` -- it is the default target
Dev Namespace
The Dev namespace aggregates convenience tasks for day-to-day development:
Clean- Deletes local compilation artifacts and cleans thebuild/directory.Tidy- Runsgo mod tidyinside the workspace.Vendor- RunsTidy, thengo mod vendor, thenWriteOSVOverrides. Use this rather than a barego mod vendorafter changing dependencies:go.mod,go.sum, andvendor/are updated in one step, and the generatedosv-scanner.tomlfiles thatgo mod vendordeletes are put back.WriteOSVOverrides- Writes those generatedosv-scanner.tomlfiles intovendor/.Vendorcalls it, so run it on its own only to repair a tree that was vendored some other way - usually a Dependabot branch, since Dependabot re-vendors without going through mage.VerifyVendor- Checks that every declared override is present and still matches the generator, and thatvendor/holds no non-Go dependency manifest without an entry covering it..github/workflows/check-go-mod.yamlruns it on every pull request. Why vendor/ carries four generated osv-scanner.toml files explains what the files are for.Digest- Resolvesdocker.io/library/alpine:latestthrough the host’s Docker credentials and prints its digest. This is a connectivity and auth smoke test, not part of a build.DnfPins- Queries the AlmaLinux 10 RPM repository metadata over HTTP to find the newest available versions of the packages installed in the container images (ansible-coreandbash-completion), then updatescontainers/ansible/Dockerfileandcontainers/ubi/Dockerfile(viaARGdefaults) and.goreleaser.yaml(viabuild_args). A package is queried only while an image installs it:shadow-utilswas dropped when the ubi image stopped installing it, because a pin nothing consumes makes this target fail looking for anARGthat is not there.TestUpdatersMatchTheRepositoryFilesholds the updaters against the real files for that reason.
mage dev:clean # rm the build/ artifacts
mage dev:tidy # go mod tidy
mage dev:vendor # go mod tidy, go mod vendor, then rewrite the osv overrides
mage dev:digest # print the alpine:latest digest, to check registry auth works
mage dev:writeOSVOverrides # write every osv-scanner.toml override into vendor/
mage dev:verifyVendor # check those overrides are present, current, and complete
mage dev:dnfPins # query AlmaLinux repodata and bump dnf/microdnf package pins
Test Namespace
The Test namespace hosts the integration and validation suites:
Unit- Runsgo testover every package except the e2e suites undertest/, which need Docker and runners of their own. Needs no binary, cluster, or network, and is what theunitCI job runs.EndToEnd- Runs the whole e2e suite: both the cluster and non-cluster groups, including the example packages that pull ~1.5GB of engine artifacts and images. Needs Docker.EndToEndNonCluster- Runs the group that needs no cluster: the misc and package command groups.-shortadditionally skips the example packages, so this finishes in seconds. Mirrors thee2e-nonclusterCI job.EndToEndCluster- Runs only the group that needs a bootloose cluster: the install command group. Needs Docker. It builds nothing; that suite calls the cargoship packages directly rather than driving a binary.EndToEndClusterStage- Runs the same suite asEndToEndCluster, but stops at the boundary phase/60 draws: it stages the files and renders the engine config without starting the engine on any node. It provisions a different five machines rather than fewer:stageOSis one per OS family per role, including a Fedora controller and the upload-only Alpine host, where the full walk’sk3sOSis a single Ubuntu controller and four workers.EndToEndClusterDryRun- Runs only the dry-run walk of the cluster suite: a whole apply, prepare and reset with--dry-runagainst the staging cluster, asserting the hosts are left as they were found and that the read-only phases still gathered their facts. Starts no engine, so it is the fastest way to exercise the flag against real hosts.CleanCluster- Removes the containers a bootloose cluster left behind.EndToEndClusterdoes this before it runs; use this target for a run that was killed partway through, or to inspect what a failed run left before clearing it.Fuzz- Replays the fuzz seed corpus (thef.Addvalues in each target plus anything committed underfuzz/testdata/fuzz/<Target>/). Calls the packages in process; needs no binary, cluster, or network. See fuzz-tests.
mage test:endToEnd # build the binary, then run both e2e groups (needs Docker)
mage test:endToEndNonCluster # run the misc/package command suites, no cluster needed
mage test:endToEndCluster # run the install command suite against a bootloose cluster
mage test:endToEndClusterStage # same, but stop at the pre-engine staging boundary
mage test:endToEndClusterDryRun # run only the --dry-run walk against the staging cluster
mage test:cleanCluster # remove bootloose containers left behind by a killed run
mage test:fuzz # replay the fuzz seed corpus
Generate Namespace
The Generate namespace handles code-generation and repository asset updates:
-
Document- Automatically generates command documentation from Cobra structures, parses cluster operational phase descriptors, rendersdocs/schema/from the same struct reflectionSchemafeeds toschema/*.json, renders the Ansible collection’s module and role reference pages from the action plugins’DOCUMENTATIONblocks and the roles’meta/argument_specs.yml, rebasesREADME.mdintodocs/index.mdand.github/SECURITY.mdintodocs/security.mdfor the book, and formats the mdBookdocs/SUMMARY.mdstructure. -
Schema- Generates YAML-compatible JSON schemas inschema/from Go structs using reflection, facilitating IDE autocomplete and validation for cluster config, distro packages, and runtime configs. -
Completion- Generates shell tab completion scripts for bash, zsh, fish, and powershell intohack/completion/. -
PullEngineSource- Fetches raw k3s/RKE2 source at the tags pinned inthirdparty-src/pins.jsonintothirdparty-src/(see thirdparty-src). Touches the network. -
LatestTag <distro> <vMAJOR.MINOR>- Resolves the newest non-RC upstream tag for that minor line, pins it inthirdparty-src/pins.json, and re-pulls that version’s source if the pin moved. Touches the network. -
UpdatePins- RunsLatestTagover every minor line already pinned inthirdparty-src/pins.json, refreshing each to its newest patch release. Touches the network. -
Examples- Rendersmagefiles/templates/<distro>-distro.yaml.tmplintoexample/<distro>-<cni>/<minor>/v<version>/distro.yaml, one flavor per CNI (example/k3s-flannel/) plus a multi-architecture flavor per CNI (example/rke2-multi-cni-canal/,example/rke2-multi-cni-cilium/,example/k3s-multi/) and single-purpose flavors that vary one setting (example/rke2-cilium-vsphere/), grouped by minor line (v1_35/, matchingthirdparty-src/<distro>/andpkg/engineconfig/gen/<distro>/) and creating those directories as needed. Per flavor it renders once per tag of that distro pinned inthirdparty-src/pins.jsonand once per example directory that flavor already has on disk, so a template edit reaches the older examples instead of leaving them to drift. Everything that varies between versions is derived from the tag, exceptimageConfig.imagesand the digests, which come from that release’s published assets. Touches the network the first time it renders a version: each release’s text assets are cached under<zarf_cache>/examples/, one flat file per asset URL named the way phase 50 names an image tarball from an image reference, so re-rendering a version already on disk fetches nothing. An entry is only used while the URL it was fetched from still matches, andCARGOSHIP_EXAMPLES_NO_CACHE=1refetches everything for a run - needed when Rancher re-cuts a release’s assets in place, since the URL does not change when it does.Distros and their flavors are declared together in
exampleDistrosinmagefiles/pkg/gen/examples/distros.go. Flavors differ by more than their image list: cilium replaces kube-proxy (disable-kube-proxy: true) and is configured through anrke2-ciliumHelmChartConfig manifest, while canal and flannel run alongside kube-proxy and carry no manifest of their own. Adding a CNI means adding a flavor entry there, and a{{ if }}in that distro’s template for anything specific to it.A flavor also names values templates, which are rendered next to
distro.yamlunder the same name minus.tmpl. They are grouped per distro rather than per flavor –magefiles/templates/rke2/andmagefiles/templates/k3s/– because a package carries a single schema, so everything its flavors expose has to be described in one document. The templates are rendered against the sameexampleVersionthe definition is, so a{{ if eq .CNI "cilium" }}in them covers what only the cilium flavors expose. Those values are what let one example stand in for several: which bundled charts the engine installs, and for cilium encryption, L2 announcements, and the Hubble UI, are chosen at install time through the cluster’sspec.config.valuesrather than baked into a separate flavor at render time.The rendered schema suggests
addons.disabledvalues fromexampleVersion.Addons, which is that distro/minor’s packaged-component vocabulary asEngineConfigextracted it – so a v1.37 example offersrke2-gateway-api-crdand a v1.34 one does not. They areexamplesrather than anenum: the vocabulary comes from the engine’s--disablehelp text, which does not name every chart the build bundles, so restricting to it would reject charts the engine would have honored. A version whose source has never been pulled has no vocabulary, and rendering its values files fails rather than writing a values file an editor can offer nothing for. Pull that minor line (Generate.PullEngineSource) and re-runEngineConfigfirst.A flavor also decides which architectures its examples target and which releases it covers.
archeslists the architectures, defaulting to amd64 alone;minorslimits the flavor to named minor lines, defaulting to every line the distro renders; andnameis what follows the distro inmetadata.namewhen the CNI is not the point. The multi-architecture flavors use all three: they target amd64 and arm64, cover the minor lines named inexampleMultiMinors(currentlyv1_32throughv1_37, inmagefiles/pkg/gen/examples/distros.go), and are namedmulti, because they exist to show what a package covering several architectures looks like. Everything a distro publishes once per architecture – RKE2’s versioned RPMs and release tarball, the k3s binary and its digest – hangs offArcheson the rendered version rather than off the version itself, so the templates range over architectures whether there is one or several, and a single-architecture example renders exactly as it did before, without anarchselector on anything.What differs between the distros themselves sits in three hooks on the distro entry, so the rendering itself stays single-copy:
derivebuilds the URLs only that distro has,probenames the URL whose absence retires a build, andfetchpulls anything else only the release can answer. RKE2 installs versionedrke2-common/server/agentRPMs fromrpm.rancher.ioand splits its images across a core manifest plus one per CNI. k3s installs a single binary straight from its GitHub release, ships flannel inside that binary - sok3s-images.txtis the whole list and there is nothing per-CNI to fetch - and publishessha256sum-<arch>.txt, which is where each binary’sshasumcomes from rather than downloading 70 MB per version and architecture to hash it. Both distros install their distro’s selinux policy RPM.k3s examples share one copy of the unit files and killall script, kept in
example/k3s/core/and referenced as../../../k3s/core/…; relativesourcepaths resolve against the directory holdingdistro.yaml.Each RPM entry also carries a
shasum, so cargoship can verify the file it downloads. Hashing an RPM means downloading it, andrke2-commonalone is around 29 MB, so the digests are cached inexample/shasums.json. It is a committed map keyed by file name, each entry holding theurlsit has been fetched from and itssha256, so the file reads as an inventory of what the examples install. Only files missing from it are fetched, which makes a re-render free and a new release cost just its own four RPMs. An entry whose URL is not one already recorded against that digest is re-hashed rather than trusted, so moving a file cannot hand back a stale digest. Delete an entry to force it to be re-hashed.A file extracted out of an archive is keyed by the archive and the path inside it, joined by
!-v1.37.0%2Brke2r1/rke2.linux-amd64.tar.gz!bin/rke2. That is a separate entry from the archive itself, because afilesentry that setsextractPathdeclares the digest of the file it installs, and the archive it came out of hashes to something else. The archive is named by the last two segments of its URL rather than by its file name alone, which a whole-file entry is enough for: every rke2 release publishes its tarball asrke2.linux-amd64.tar.gz, so a key built from that name would be the same for all seventy of them, each render would evict the last, and every run would re-download every tarball. The segment above the file name is the release tag, which is what makes the key specific to a version.RKE2’s binary, its killall and uninstall scripts, and both unit files all come out of
rke2.linux-<arch>.tar.gz, so the template asks for five members of each tarball. One streaming pass hashes every file in the archive and the ones asked for are cached individually, so a tarball is pulled only when the cache is missing at least one of its members, and pulled once when it is. Nothing is written to disk. A new rke2 release costs its two tarballs - around 80 MB - on top of its RPMs; a re-render of what is already cached costs nothing.Before rendering, each build’s
rke2-commonRPM is checked (aHEAD, and only for URLs the cache has never seen). Rancher supersedes anrke2rNwith the next revision and removes the old RPMs, while the git tag and image manifests stay up - so a build can look renderable and still install nothing. Those are skipped, and any example already on disk for one is deleted, along with its minor line directory if that empties it. As of August 2026 that coversv1.35.0+rke2r2andv1.35.3+rke2r2, replaced byrke2r3, andv1.34.3+rke2r2andv1.34.6+rke2r2- use therke2r3release of those patches. Nothing is remembered about a skip, so a build that comes back is rendered again on the next run; a build that goes away is only noticed while it is still pinned or still on disk. Theshasumon an individual file is still allowed to be missing - that path now only covers the odd file a still-published build lost. -
ExampleLine <distro> <vMAJOR.MINOR>- Renders an example for every non-RC release on one minor line of that distro, rather than only the pinned one, sorke2 v1.36backfillsv1.36.0+rke2r1through the newestv1.36release intoexample/rke2-multi-cni-cilium/v1_36/,example/rke2-cilium-vsphere/v1_36/, andexample/rke2-multi-cni-canal/v1_36/. A flavor that names minor lines is only rendered for the lines it names, so asking for a line a multi-architecture flavor does not cover renders the other flavors alone. The leadingvis optional. Once written,Exampleskeeps those files current, since it re-renders every example directory on disk. Touches the network, through the same release-asset cacheExamplesuses.ExampleLine upstream <vMAJOR.MINOR>is a separate path (seemagefiles/pkg/gen/examples/upstream.goand itsupstream_cni.go,upstream_kubeadm.go,upstream_deb.go,upstream_rpm.go, andupstream_repo.gosiblings), since kubeadm does not fitexampleDistroSpec. It renders each line into all three CNI flavors -example/upstream-cilium,example/upstream-canal,example/upstream-flannel. It backfills every stable patch on the line by picking, per package, the closest version published onpkgs.k8s.ioat or before that patch - pkgs.k8s.io keeps every patch’s packages on a minor line, not just the newest, sov1.37.0andv1.37.1render distinctkubelet/kubectl/kubeadm/cri-tools/kubernetes-cniversions. The etcd, coreDNS, and pause versions kubeadm pins come from that same tag’s owncmd/kubeadm/app/constants/constants.go, fetched live fromraw.githubusercontent.com, not from the single per-minor cached copythirdparty-src/upstream/<minor>/holds - that copy only ever reflects whichever tag is currently pinned.containerd.io, pulled fromdownload.docker.com, is the one package still resolved to whatever is newest at render time, since its release cadence is independent of the Kubernetes tag.Upstream ships no CNI, so each flavor’s package carries one and declares it in
spec.config.manifests, which cargoship kubectl-applies from the leader afterkubeadm init(apply). flannel and canal are pinned manifest URLs, hashed intoexample/shasums.jsonlike any other file. Cilium publishes only a Helm chart, so its manifest is rendered withhelm templateintoexample/upstream-cilium/cni/cilium-<chart>.yamland shared by every version of the flavor, referenced as../../cni/…the way k3s examples shareexample/k3s/core/. That render is skipped while the file exists, sohelmis needed only to bump the pinned chart version, not to keep the examples current -refresh-examples.yamlnever needs it. Each flavor’s images are read out of the manifest it applies rather than listed by hand, and a render that produced a private key is refused outright, since committing one would hand every cluster built from the example the same CA. -
EngineConfig- Statically parses raw engine source underthirdparty-src/(see thirdparty-src) to generate typedconfig.yamlstructs per distro/version inpkg/engineconfig/gen/, plus each version’s packaged-component vocabulary (the validdisable:,cni:, andingress-controller:values) in azz_addons.goalongside them.
mage generate:document # regenerate docs/commands, docs/phases, docs/schema, docs/ansible, docs/index.md, docs/security.md, and docs/SUMMARY.md
mage generate:schema # regenerate schema/*.json from the Go API types
mage generate:completion # regenerate hack/completion/ shell completion scripts (bash, zsh, fish, powershell)
mage generate:zarfFlags # re-read the zarf flag lists the Ansible module tests hold
mage generate:pullEngineSource # re-pull every tag already pinned in thirdparty-src/pins.json
mage generate:latestTag rke2 v1.36 # pin rke2's newest v1.36.x, and pull it if the pin moved
mage generate:latestTag k3s v1.31 # same, for a k3s minor line
mage generate:updatePins # bump every pinned minor line, both distros, to its newest patch
mage generate:engineConfig # regenerate pkg/engineconfig/gen/ from what is on disk
mage generate:examples # re-render every example/<distro>-<cni>/<minor>/*/distro.yaml
mage generate:exampleLine rke2 v1.36 # render an example for every rke2 release on the 1.36 line
mage generate:exampleLine k3s 1.36 # same for k3s -- the leading v is optional
mage generate:verifyAssets # re-fetch every image manifest in example/assets.json and report drift
VerifyAssets is the one target that deliberately reads no cache. example/assets.json holds the image manifest each pinned release publishes, so that a fresh clone - or an air-gapped management node - can re-render every example already on disk without reaching upstream. That makes it a pin, and a pin nothing watches goes stale silently: a release whose assets were re-cut in place keeps rendering from the copy taken the day it was first fetched. The target costs one request per manifest, reports rather than rewrites, and exits non-zero on drift; CARGOSHIP_EXAMPLES_NO_CACHE=1 mage generate:examples is what applies the fix.
LatestTag is also how a new minor line is added: pass a prefix that thirdparty-src/pins.json does not yet pin and it appends that line rather than replacing one. Adding an rke2 minor means adding the matching k3s minor too, because rke2’s config is composed against k3s’s flags at the same version:
mage generate:latestTag k3s v1.37
mage generate:latestTag rke2 v1.37
mage generate:engineConfig
mage generate:examples
The usual order after any pin change is updatePins (or latestTag), then engineConfig, then examples - the first two write what the next one reads, and only the pin-moving targets touch the network for source. engineConfig is fully offline.
File-by-File Reference
core/core.go: Configures the bootstrap process and imports distro-specific modules to register Go side-effects before task execution. Lives in its own subpackage (rather than directly inmagefiles/) so it does not collide with thefunc main()that themageCLI generates on the fly - see Running Mage Directly below.build.go: Entrypoint for theBuildnamespace (Build.Binary,Build.All,Build.Examples), delegating tomagefiles/pkg/build.dev.go: Entrypoint for theDevnamespace (Dev.Clean,Dev.Tidy,Dev.Vendor,Dev.Digest,Dev.DnfPins,Dev.WriteOSVOverrides,Dev.VerifyVendor).generate.go: Entrypoint for theGeneratenamespace (Generate.Document,Generate.Schema,Generate.Completion,Generate.EngineConfig,Generate.Examples,Generate.VerifyAssets, etc.), delegating tomagefiles/pkg/gen/....test.go: Entrypoint for theTestnamespace (Test.Unit,Test.EndToEnd,Test.EndToEndNonCluster,Test.EndToEndCluster,Test.EndToEndClusterStage,Test.EndToEndClusterDryRun,Test.EndToEndClusterUpgrade,Test.EndToEndZarf,Test.CleanCluster,Test.CleanZarfClusters,Test.Fuzz), delegating tomagefiles/pkg/testrunner.pkg/build/: Binary compilation logic, flag assembly, and example package builds.pkg/devtools/: Developer tooling packages, includingdnfpins(AlmaLinux repomd XML parser and pin updater) andosv(OpenSSF Scorecard vendor overrides).pkg/gen/: Generator implementations:completion/(shell completion scripts),docs/(Cobra command docs and mdBook pages),engineconfig/(k3s/RKE2 source pins and struct codegen),examples/(Rancher and upstream distro examples), andschema/(JSON schema reflection and docs).pkg/testrunner/: Test runner implementations for e2e suites and fuzz corpus replay.pkg/util/: Shared helpers for aligned table formatting, git commit resolution, and clean build tasks.templates/: Text templates the generation targets render:rke2-distro.yaml.tmpl,k3s-distro.yaml.tmpl, andupstream-distro.yaml.tmpl(rendered once per CNI flavor), plus the per-distro values templates a flavor ships beside itsdistro.yaml(rke2/values.yaml.tmpl,rke2/values.schema.json.tmpl, and thek3s/equivalents).
Managed Outputs
Running various Mage tasks maintains and updates the following filesystem artifacts:
| Output Directory / File | Description | Target |
|---|---|---|
build/cargoship_* | Compiled release binaries | Build.All |
docs/commands/* | Auto-generated CLI documentation | Generate.Document |
docs/phases/* | Auto-generated cluster phase descriptors | Generate.Document |
docs/schema/* | Auto-generated schema field reference, from the same struct reflection as schema/*.json | Generate.Document |
docs/ansible/<collection>/module_*.md | Auto-generated Ansible module reference, from the action plugins | Generate.Document |
docs/ansible/<collection>/role_*.md | Auto-generated Ansible role reference, from each role’s meta/argument_specs.yml | Generate.Document |
docs/index.md | The root README.md, with its links rebased onto docs/ for the book | Generate.Document |
docs/security.md | .github/SECURITY.md, with its links rebased onto docs/ for the book | Generate.Document |
docs/SUMMARY.md | Compiled table of contents for mdBook | Generate.Document |
schema/*.json | JSON schemas for YAML validations | Generate.Schema |
hack/completion/* | Shell completion scripts for bash, zsh, fish, and powershell | Generate.Completion |
pkg/engineconfig/gen/* | Typed engine config.yaml structs per distro/version | Generate.EngineConfig |
thirdparty-src/<distro>/<minor>/* | Raw pinned upstream k3s/RKE2 source | Generate.PullEngineSource / Generate.LatestTag |
thirdparty-src/pins.json | Pinned upstream tags | Generate.LatestTag / Generate.UpdatePins |
example/<distro>-<cni>/<minor>/*/distro.yaml | Rendered rke2 and k3s example packages, one directory per CNI flavor, grouped by minor line | Generate.Examples |
example/shasums.json | Cached sha256 of every remote file the examples hash | Generate.Examples / Generate.ExampleLine |
<zarf_cache>/examples/* | Cached release text assets (image lists), not committed | Generate.Examples / Generate.ExampleLine |
containers/ansible/Dockerfile | Base image package pins for ansible-core and bash-completion | Dev.DnfPins |
containers/ubi/Dockerfile | Base image package pin for bash-completion | Dev.DnfPins |
.goreleaser.yaml | Build args pinning container package versions | Dev.DnfPins |
Running Mage Directly (Without the CLI)
Normally you invoke tasks through the installed mage binary, e.g. mage build:binary. The mage CLI works by scanning magefiles/ for exported functions and namespaces, then generating its own func main() (written to a gitignored mage_output_file.go) that wires those functions up to CLI subcommands before compiling and running the result.
Because that generated file declares package main with its own func main(), it cannot coexist with a second, hand-written func main() in the same package - hence core/core.go (which does exactly that, via mage.Main()) is split out into its own magefiles/core subpackage rather than sitting alongside the task files in magefiles/.
This split means core/core.go can also be run on its own, bypassing the mage CLI entirely:
go run ./magefiles/core
This builds and runs the same mage.Main() entry point that the mage CLI would otherwise generate for you. It’s useful when:
- The
magebinary isn’t installed on the host (e.g. a minimal CI or container image that already has a Go toolchain). This is how CI runs its document checks -go run -mod=vendor ./magefiles/core generate:document- so the workflow needs nothing beyondsetup-goand the vendored tree. - You want a single, explicit
go runinvocation instead of depending on a separately-installed tool.
Task selection still works the same way - pass the namespace:target as an argument, e.g.:
go run ./magefiles/core build:binary
Note that magefiles/ itself remains its own package main for the mage CLI’s benefit; core/core.go is a separate package and binary, not part of that compiled unit.
A local VM fleet
mage dev:vmUp brings up a fleet of Rocky 10 virtual machines on your own machine and writes a ZarfCluster inventory pointing at them, so that cargoship prepare and cargoship apply can be run against hosts with a real kernel. mage dev:vmDown takes it away again.
It exists for the phases the cluster e2e suite cannot reach. Those nodes are containers, and a container reports no SELinux, runs no fapolicyd and has no firewall of its own, so pkg/phase/21_prepare_selinux.go and its neighbours correctly do nothing there. Testing a change to any of them means a virtual machine. choice-microvm-backend records why this is plain qemu rather than Firecracker or libvirt.
Nothing it does touches the host. It needs no root, starts no daemon, and changes no SELinux, firewall or kernel setting on your machine - which is the point, since the configuration under investigation is usually the host’s own.
What you need
qemu-system-x86_64, qemu-img, mkfs.vfat, mcopy, ssh, and a readable /dev/kvm. mage dev:vmUp reports every missing tool at once rather than failing part way through a bring-up.
qemu-setup is the full walkthrough: which package provides each of those on Fedora and Debian, how /dev/kvm access works without joining a group, what to do when your development machine is itself a virtual machine, and how to tell a node is really accelerated rather than emulated.
Bringing a fleet up
mage dev:vmUp -control=1 -worker=2
mage dev:vmUp -control=1 -worker=2 -infra=3 -distro=rke2
Every flag has a default, so mage dev:vmUp on its own gives one control-plane node and two workers running k3s.
| Flag | Default | What it does |
|---|---|---|
-control | 1 | Control-plane nodes. The first is the leader, which is what the inventory’s load balancer names |
-worker | 2 | Plain worker nodes |
-infra | 0 | Infra nodes: workers carrying an infra profile of their own |
-ubuntuControl | 0 | Control-plane nodes running Ubuntu instead of Rocky |
-ubuntuWorker | 0 | Worker nodes running Ubuntu instead of Rocky |
-distro | k3s | The engine the fleet is prepared for, k3s, rke2 or upstream |
-distro changes nothing about the guests. It decides only which of the leader’s ports are forwarded to your machine, since rke2 serves its join endpoint on 9345 while k3s multiplexes onto 6443 and upstream kubeadm has no supervisor port at all.
Infra nodes join as workers – the engine has no third role – and what makes them infra is their profile, which pkg/phase/81_label_nodes.go turns into a node-role.kubernetes.io/infra label and which a cluster config can hang a per-profile concurrency off. They are worth asking for because every other host in the generated inventory has a profile that just restates its role, which leaves both of those mechanisms indistinguishable from the role they came from. Ask for some and the engine phases batch them separately: starting agent (profile: infra).
The first run downloads a pinned Rocky 10 GenericCloud image, roughly 520MiB, into $XDG_CACHE_HOME/cargoship/microvm and verifies its digest. That cache is outside the repository and survives a clean checkout; later runs reuse it.
Each node is a copy-on-write overlay over that image, so a bring-up costs a few megabytes rather than another copy. Bring-up takes a minute or two: a node answers SSH about six seconds in, and the rest is cloud-init installing firewalld and fapolicyd. The fleet is only handed back once cloud-init has finished on every node, because the gates in phases 22 and 26 ask whether those services are running - a fleet returned any earlier looks healthy and has both of those phases silently skipping.
NODE ROLE STATE SSH PRIVATE ADDRESS
kc0 controller running (pid 197656) 127.0.0.1:26241 10.73.156.11
kw0 worker running (pid 197673) 127.0.0.1:26242 10.73.156.12
kw1 worker running (pid 197714) 127.0.0.1:26243 10.73.156.13
Inventory: build/microvm/dev/inventory.yaml
Then drive cargoship at it the way you would any other inventory. prepare and apply take a built package rather than the definition directory it was built from, so create comes first – that is the one step that needs a network, and the one the fleet itself does not:
cargoship create ./example/rke2-multi-cni-cilium/v1_36/v1.36.1-rke2r1 -o ./build/
cargoship prepare ./build/cargoship-rancher-rke2-multi-cni-cilium-multi-1.36.1-rke2r1.tar.zst --config build/microvm/dev/inventory.yaml --confirm
cargoship apply ./build/cargoship-rancher-rke2-multi-cni-cilium-multi-1.36.1-rke2r1.tar.zst --config build/microvm/dev/inventory.yaml --confirm
prepare is optional – an apply runs the same preparation phases itself. Both change every host they are pointed at, which is what --confirm acknowledges; pass --dry-run instead to see what they would change.
What a bring-up is doing while it waits
dev:vmUp blocks until the fleet is usable, and on a six-node fleet that is minutes. It reports what it is waiting on so a long wait does not read as a hung shell:
$ mage dev:vmUp -control=3 -worker=3
Bringing up 3 control-plane, 3 worker node(s) for k3s
waiting for 6 node(s) to finish cloud-init, up to 15m0s
ready: kc0 1m4s
waiting: kc1 cloud-init is still running (30s)
ready: kc1 1m12s
The wait is for cloud-init rather than for sshd, which answers about six seconds in. The packages that open the fapolicyd and firewall gates are installed after that, so a fleet handed back at the earlier moment looks healthy and has both of those phases silently skipping – see “Bringing a fleet up” above.
The 15 minute budget covers the whole fleet, not each node: they are all started before any is waited on, so the wall time is the slowest node rather than the sum.
Interrupting the wait does not stop the fleet. Each node is a detached qemu process, so Ctrl-C leaves them running and the inventory in place – which is deliberate, since a node that is slow to configure is usually one worth logging into. dev:vmList shows what is there and dev:vmDown removes it; a later dev:vmUp refuses rather than adopting a fleet that is already up.
Memory is the limit worth planning. Each node takes 4096MiB and 2 vCPUs by default, so six nodes want 24GiB and a bring-up that would not fit in MemAvailable is refused rather than left to the OOM killer. -memory and -cpus are how to fit a bigger fleet into a smaller machine:
mage dev:vmUp -control=3 -worker=3 -memory=2048
2048MiB is ample for the host-preparation phases. An rke2 control-plane node wants the default: three of them at 2048 will come up and then struggle once etcd and the control plane are both running.
The e2e suite has no flags, so it reads the same setting from CARGOSHIP_E2E_VM_MEMORY. It matters more there than for dev:vmUp, because the suite brings its fleet up at the size the join walk needs rather than the size the apply walk uses - four nodes, so 16GiB at the default:
CARGOSHIP_E2E_BACKEND=microvm CARGOSHIP_E2E_VM_MEMORY=3072 mage test:endToEndCluster
A value that is not a positive number is ignored and leaves the default, since a fleet that does not fit is reported clearly by the bring-up anyway.
A mixed fleet
The fleet runs Rocky 10 unless you ask for something else. Two counts opt in to Ubuntu nodes alongside them:
mage dev:vmUp -control=1 -worker=1 -ubuntuWorker=2 -distro=k3s
mage dev:vmUp -control=0 -ubuntuControl=1 -ubuntuWorker=1 -distro=k3s # Ubuntu only
Rocky nodes are kc* / kw* / ki*; Ubuntu nodes are uc* / uw*. dev:vmList shows which guest each node runs.
This is what a single-distribution fleet cannot cover, and it is the gap choice-microvm-backend records as the reason the container matrix is still the only thing that covers family routing. With Ubuntu nodes in the fleet, apply routes for real:
| On a Rocky node | On an Ubuntu node |
|---|---|
Phase 21 installs container-selinux, because getenforce reports Enforcing | Phase 21 does nothing, because there is no SELinux |
| Phase 22 writes fapolicyd rules and restarts the service | Phase 22 does nothing: fapolicyd is not packaged for Ubuntu |
| Phase 26 selects the firewalld backend | Phase 26 selects ufw |
Phase 57 installs the staged RPMs through dnf | Phase 57 declines it; 59 installs the binary the package ships for hosts it carries no .deb for |
Each of those doing nothing on the other guest is the assertion worth having: the phase tests compare what ran against what the hosts report, so a gate that routes wrongly shows up as a mismatch rather than as a silent skip.
Two things to know before reaching for one:
-
An Ubuntu node is installed from a binary, not from packages, and it does join. The
rke2examples ship RPMs for the Enterprise Linux family and a tarball for everything else, so phase 57 claims the Rocky nodes and phase 59 claims the Ubuntu one. Verified on a mixed fleet: nodpkgentry for rke2, a 133MB/usr/local/bin/rke2,rke2-agentactive, the nodeReady. The glibc constraint that makes the container fleet’s Alpine node upload-only is about musl, and Ubuntu is not musl.What that means for coverage is narrower than it looks. Phase 58 runs and has nothing to install, because no
k3sorrke2example carries a.deb; theupstream-*packages are the only ones that do, so they are what exercises the APT path with real packages. Ak3sorrke2fleet exercises the routing – which phase claims which host – rather than the apt install itself. -
ufw is enabled by cloud-init, which cargoship deliberately never does: bringing up a default-deny firewall over the SSH connection a phase is running on can cut it, so
pkg/firewall/ufw.goapplies only to a host whose ufw is already active. A stock cloud image ships it inactive, so without that the ufw path is not reachable at all.
Updating a pinned image
Both guests are pinned by dated build and digest in test/microvm/image.go, not by a latest alias. The aliases move without warning, and a release that ships a broken build – Rocky has published a GenericCloud image with no cloud-init in it – is then something this package steps over by not moving, rather than something every developer hits at once. TestEveryImageIsPinnedAndVerifiable fails on a URL containing latest, so the pin cannot quietly become a moving one.
Bumping one is three edits and a download. For Rocky, the dated builds and their checksums are published side by side:
curl -s https://dl.rockylinux.org/pub/rocky/10/images/x86_64/ | grep -o 'Rocky-10-GenericCloud-Base-[0-9.]*-[0-9.]*\.x86_64\.qcow2' | sort -u | tail -3
curl -s https://dl.rockylinux.org/pub/rocky/10/images/x86_64/<file>.CHECKSUM
For Ubuntu, the dated directory is what to pin – releases/26.04/release/ is a pointer that moves:
curl -s https://cloud-images.ubuntu.com/releases/26.04/ | grep -o 'release-[0-9]\{8\}' | sort -u | tail -3
curl -s https://cloud-images.ubuntu.com/releases/26.04/release-<date>/SHA256SUMS | grep amd64.img
Then update file, url and sha256 on that image, and run the download so the verification actually sees the new build:
mage dev:vmImage -image=rocky # or ubuntu, or all
Fetch checks the digest and then asks guestfish two things the digest cannot: that cloud-init is in the image at all, and – for a guest declared selinux: true – that it sets SELINUX=enforcing. Both have been wrong in a published build, and both fail late and unrecognisably if nobody looks: without cloud-init the seed is ignored and every node simply times out waiting for SSH, and with SELinux disabled the fleet comes up fine and every phase it exists to test quietly skips.
A cached image whose digest no longer matches the pin is reported rather than silently replaced, naming the file to remove – re-downloading half a gigabyte is not something to do behind somebody’s back. After a bump, bring a fleet up and check the guest is what you think it is:
mage dev:vmUp -control=1 -worker=0
mage dev:vmShell kc0 # then: cat /etc/os-release; getenforce
Looking around, and tearing down
mage dev:vmList # what is there and whether it is running
mage dev:vmShell kc0 # a root shell on one node
mage dev:vmDown # every node stopped, every trace removed
mage dev:vmImage # fetch and verify the base image, start nothing
Each of those takes -fleet as well, and it matters once the e2e suite has been run against VMs: CARGOSHIP_E2E_BACKEND=microvm brings up a fleet named e2e, which the default dev fleet does not cover. A suite run killed before its own teardown leaves that fleet behind, and nothing else will remove it:
mage dev:vmList -fleet=e2e
mage dev:vmDown -fleet=e2e
dev:vmUp refuses a fleet that is already up rather than adopting it, so every bring-up starts from a fresh overlay. Tear the old one down first. That refusal is deliberate: a node still carrying the state of a previous apply is the one thing that makes a phase failure impossible to reason about.
Note that mage dev:clean does not remove a fleet. It removes built binaries, and deleting a live fleet’s directory from under it would leave its qemu processes running with no record of them. dev:vmDown is what takes a fleet away.
How the networking works
Each node has two interfaces, and which is which matters when reading a phase’s output.
enp0s3 is the management interface, a qemu user (slirp) network. It carries the SSH forward your machine connects through, and outbound access to the internet - which phase 21 depends on, since it installs container-selinux by name and fails the run if it cannot. Every node has the same address on it, 10.0.2.15, and no node can reach another there.
enp0s4 is the fleet’s own layer 2 segment, carried between the qemu processes on a multicast socket. This is where the nodes see each other, on 10.73.<fleet>.<node>, and it is what the inventory names as each host’s privateAddress and privateInterface. Those two fields are set explicitly rather than discovered, because the default route is on the management interface: fact gathering would pick 10.0.2.15 for every host, and the firewall and /etc/hosts phases would faithfully distribute an address no peer can reach.
The leader additionally forwards its engine API port to your machine, so cargoship kube-config and kubectl work from here rather than only from inside the fleet.
When a node does not come up
A node that never answers is reported with the tail of its serial console, and the full log is at build/microvm/dev/<node>/console.log. That is the only record: qemu is detached, so nothing else captured its boot.
A node whose cloud-init failed is reported with cloud-init’s own explanation, from cloud-init status --long. Run that on the node yourself for more, and read /var/log/cloud-init.log.
Host keys are regenerated on every bring-up, so host key checking is off and known-hosts is /dev/null for every connection the fleet makes. Set SSH_KNOWN_HOSTS="" when running cargoship against the inventory, for the same reason.
What the management node can and cannot reach
Your machine reaches a node on exactly one path: the forwarded SSH port. That is enough for every phase cargoship drives over SSH, which is all of the host preparation, the uploads, the engine install and the join – and it is all this fleet sets out to cover.
It is not enough for the phases that talk to the Kubernetes API from the management node. Two independent things stop them:
- The inventory’s
loadbalanceris the leader’s address on the fleet’s private segment, because that is the address the other nodes join through. Nothing outside the qemu processes can route to that segment, so a phase that dials the load balancer from here gets a timeout. - Once
--firewallhas run, firewalld’spubliczone – which holds both of a node’s interfaces – allows ssh and nothing else. Node-to-node traffic works because the trusted zone admits the private addresses by source, so the leader’s forwarded API port reaches the guest and is then dropped. This is the firewall behaving correctly.
In practice that means cargoship apply --label-nodes fails at Labeling nodes with their profile group, and a kubeconfig written for this fleet does not work from here. Drive kubectl from inside a node instead:
mage dev:vmShell kc0
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml # or /etc/rancher/k3s/k3s.yaml
/var/lib/rancher/rke2/bin/kubectl get nodes -o wide
Note what that costs in the suite: Test_ZZ1_ClusterHealthy is the only step that asserts the cluster actually works, and it is skipped here because it needs an API route from the management node. A green run against this fleet therefore says the phases did their work, not that the cluster is healthy. Check that from inside a node when it matters.
Giving the management node a routable path onto the private segment would mean a tap device or a bridge, which needs root and puts the fleet back under the host’s firewall – the two things choice-microvm-backend picked a multicast socket to avoid.
The engine’s node IP
Each node reports its own address on the private segment, which is what makes a multi-controller cluster possible here at all. That was not always true, and the story is worth knowing because this fleet is what found it.
Cargoship used to render no node-ip, so the engine chose its own node address from whichever interface held the default route – the management one. Every node then reported 10.0.2.15, the same slirp address, and a three-controller cluster could not form: all three advertised the same etcd peer URL, so the second could never be added and the apply sat on MemberAdd request timed out until it gave up. A single controller hid it completely. The API server also could not reach a kubelet it believed was at an address shared with every other node, so kubectl logs and kubectl exec returned 502 Bad Gateway and a Helm-installed CNI could fail outright, leaving every node NotReady.
ConfigureEngine now writes node-ip from the host’s privateAddress, beside the node-name it already wrote, so the fleet behaves the way any multi-homed fleet should:
$ kubectl get nodes -o wide
NAME STATUS ROLES INTERNAL-IP
kc0 Ready control-plane,etcd 10.73.156.11
kw0 Ready <none> 10.73.156.12
kw1 Ready <none> 10.73.156.13
Driving the OpenTofu provider at a fleet
The provider takes the same hosts, so a fleet is also how cargoship_cluster gets exercised against real machines rather than one container. mage dev:vmUp writes a ZarfCluster inventory, and every field a hosts entry needs is in it:
| Inventory | hosts entry | For the fleet |
|---|---|---|
ssh.address | address | Always 127.0.0.1 – the forwards bind loopback |
ssh.port | port | The per-node forwarded SSH port |
ssh.user | user | root |
ssh.keyPath | key_path | The fleet key, under build/microvm/<fleet>/key |
hostname | hostname | kc0, kw0, … |
privateAddress | private_address | The node’s address on the private segment |
privateInterface | private_interface | enp0s4 |
role / profile | role / profile | Unchanged |
resource "cargoship_cluster" "fleet" {
name = "microvm-dev"
load_balancer = "10.73.156.11" # the leader's private address
package = "./build/cargoship-distro-amd64.tar.zst"
hosts = {
kc0 = {
address = "127.0.0.1"
port = 26241
user = "root"
key_path = "/path/to/build/microvm/dev/key"
hostname = "kc0"
private_address = "10.73.156.11"
private_interface = "enp0s4"
role = "controller"
profile = "controller"
}
kw0 = {
address = "127.0.0.1"
port = 26242
user = "root"
key_path = "/path/to/build/microvm/dev/key"
hostname = "kw0"
private_address = "10.73.156.12"
private_interface = "enp0s4"
role = "worker"
profile = "worker"
}
}
}
Four things to get right, each of which is a property of the fleet rather than of the provider:
key_pathmust be absolute. The inventory already holds an absolute path for this reason; OpenTofu runs from wherever you invoke it.load_balanceris the leader’s private address, not a forwarded loopback port. It is the address the other nodes join through, and they reach each other on the private segment rather than through your machine.- Export
SSH_KNOWN_HOSTS=""beforetofu apply. Host keys are regenerated on every bring-up, so a known-hosts entry is wrong by construction. - Leave
export_kubeconfigandlabel_nodesoff. Both need the management node to reach the API server, which it cannot on a fleet whose load balancer is private – see above. A provider run that sets either will fail there rather than in anything it is testing.
The hosts key orders the run: controllers are acted on first, and the controller whose key sorts first becomes the leader. Keying the map by the fleet’s own hostnames keeps that in step with how mage dev:vmUp numbers them.
Bring the fleet up, generate the block from the inventory, and tofu apply. A removal is the other half: set a host’s state = "absent" and the provider drains it, deletes it from the cluster and uninstalls the engine, which is the same work e2e-external-inventory drives through the CLI. tofu-provider covers building and running the provider itself.
The shipped k3s packages, and what is still unproven
Two separate things used to stop a k3s cluster coming up, and only one of them is fixed.
Fixed: the join URL named the wrong port. ConfigureEngine wrote server: https://<host>:9345 for both engines. 9345 is rke2’s – it runs its supervisor separately – while k3s multiplexes its supervisor onto 6443 and listens on nothing at 9345. K3S embeds the shared Rancher configuration without overriding that, so every k3s cluster was handed a join URL its own engine was not serving, and nothing could join. The port now comes from the distro.
Not fixed, and not retested since: the SELinux policy is staged and never installed. Every example/k3s-* package sets selinux: true and ships k3s-selinux-*.rpm under selector.package: binary. The BIN upload phase moves staged files into place and runs no package manager, so that RPM lands in /var/lib/rancher/k3s/rpm/ and the %post that loads the policy module never runs – leaving a node with SELinux enforcing, an engine told to use it, and no policy for it. The rke2 examples select package: rpm for the equivalent file, which is why semodule -l lists rke2 there.
So a k3s walk on this fleet is worth trying and is not yet known to pass. mage test:endToEndClusterVM still defaults to rke2 for that reason; CARGOSHIP_E2E_DISTRO=k3s is how to find out, and the node’s journal is where the answer will be.
Limits
The fleet runs Rocky and, when asked, Ubuntu. That covers the Enterprise Linux branch against something that really has SELinux, fapolicyd and firewalld, and – with -ubuntuWorker – the Debian branch beside it, which is the pair apply routes between in most of the places it routes at all.
It is still not the whole of what the bootloose cluster covers. That fleet also runs Fedora, which is a second Enterprise Linux opinion with a different dnf and a newer selinux-policy, and Alpine, which belongs to neither family and is the only host that exercises the BIN upload phase as the path it is rather than as a fallback. Adding a guest here is one pinned URL and digest in test/microvm/image.go plus its cloud-init packages, so neither is hard – they are simply not done. Run both suites.
Setting a machine up to run the VM fleet
mage dev:vmUp runs its virtual machines under plain qemu, as your own user, with no daemon and no root. This page is how to get a machine into that state and how to tell that it worked. choice-microvm-backend records why the fleet is built this way rather than on libvirt or Firecracker.
Nothing here installs a service, adds a system unit, or changes a firewall, SELinux or kernel setting. That is deliberate: the fleet exists to test host configuration, so a developer running it should not have to change the configuration of the host they are running it from.
What has to be present
Five executables and one device:
| Needed | Fedora package | Debian/Ubuntu package |
|---|---|---|
qemu-system-x86_64 | qemu-system-x86-core | qemu-system-x86 |
qemu-img | qemu-img | qemu-utils |
mkfs.vfat | dosfstools | dosfstools |
mcopy | mtools | mtools |
ssh, ssh-keygen | openssh | openssh-client |
/dev/kvm | the kernel | the kernel |
sudo dnf install qemu-system-x86-core qemu-img dosfstools mtools openssh
guestfish, from libguestfs/guestfs-tools, is optional. It is used once per image download to check that the base image carries cloud-init and sets SELinux to enforcing; without it the fetch says it is skipping that check and carries on.
dev:vmUp reports every missing tool in one message rather than failing part way through a bring-up, so running it is a reasonable way to find out what you are missing.
Why no libvirt
There is no libvirtd here, and installing one changes nothing. qemu is driven directly, each node is a qemu-system-x86_64 process owned by your user, and the only state is a directory under build/microvm. If you already run libvirt for other work it will not conflict: the fleet defines no libvirt domain, claims no libvirt network, and creates no tap device or bridge.
KVM access, without being in a group
The fleet refuses to start without /dev/kvm, because the alternative is qemu falling back to emulation, which is slow enough that an engine never finishes coming up.
On current Fedora the device is world-accessible and there is nothing to do:
$ ls -l /dev/kvm
crw-rw-rw-. 1 root kvm 10, 232 Oct 10 15:44 /dev/kvm
That crw-rw-rw- is the point – mode 0666, set by a udev rule systemd ships, so an ordinary user can open it without being in the kvm group. On a distribution that still ships 0660 root:kvm, the group is how you get access:
sudo usermod -aG kvm "$USER" # then log out and back in, or: newgrp kvm
Check that virtualisation is actually available to the CPU, not merely that the device file exists:
grep -Eom1 'vmx|svm' /proc/cpuinfo # vmx on Intel, svm on AMD; no output means it is off
No output usually means virtualisation is disabled in firmware rather than absent from the processor.
If your development machine is itself a virtual machine
The fleet then needs nested virtualisation, which the outer hypervisor has to expose:
cat /sys/module/kvm_amd/parameters/nested # or kvm_intel; 1 or Y means available
If that reads 0 or N, enable it on the host running your development machine – on a Linux host, kvm_intel.nested=1 or kvm_amd.nested=1 – rather than on the machine you are reading it from. Without it /dev/kvm may not exist at all inside the guest.
Confirming a node really is accelerated
The surest check is how long a bring-up takes. A node reaches sshd in about six seconds with KVM; under emulation it is minutes, and that is the symptom people usually notice first.
mage dev:vmUp -control=1 -worker=0
To see it directly, look at what the process was given: the command line carries accel=kvm, and the fleet passes nothing else, so qemu either accelerates or fails to start.
pgrep -a qemu-system-x86_64 | tr ' ' '\n' | grep -A1 machine
Memory, disk, and what a fleet costs
Each node defaults to 4GiB and 2 vCPUs, so the usual three-node fleet wants 12GiB available. dev:vmUp refuses a fleet that would not fit in MemAvailable rather than letting the OOM killer choose which process dies, and says what it would have needed.
Disk is cheaper than it looks. The base image is fetched once, about 520MiB, into $XDG_CACHE_HOME/cargoship/microvm – outside the repository, so it survives a clean checkout. Each node is a copy-on-write overlay on top of it, a few megabytes at bring-up, growing to roughly 250MiB after a boot and one dnf transaction. Fleet state lives in build/microvm, which is gitignored and on real storage rather than a tmpfs; see microvm for why that matters here.
Networking, and why there is nothing to configure
Neither interface a node gets needs a privilege or a host-side rule.
The management interface is qemu’s user (slirp) network, implemented inside the qemu process. It provides the forwarded SSH port your machine connects through and outbound access for dnf. Nothing listens on a routable address: every forward binds 127.0.0.1.
The fleet’s own segment is a socket netdev on a multicast group, which joins the fleet’s qemu processes into one broadcast domain over loopback UDP. There is no tap device, no bridge, and nothing for firewalld, nftables or Docker’s rules to have an opinion about – which is exactly why it was chosen. A fleet’s internal networking cannot be broken by the host firewall configuration you are in the middle of changing.
Two consequences worth knowing before debugging: your machine cannot reach that segment at all, and the host’s own firewall needs no rules for any of this. The first is covered under “What the management node can and cannot reach” in microvm.
Host SELinux
A host running SELinux in enforcing mode needs no change. The qemu processes are started from your shell, so they inherit your unconfined context rather than the svirt_t confinement libvirt would apply, and every file they touch – the image cache, the overlays, the seed images – is one your user owns. The fleet never relabels anything on the host.
Note that the guests run SELinux enforcing, and that is the point of the fleet rather than an accident of the host.
When it does not work
| Symptom | Cause |
|---|---|
/dev/kvm is not available | The device is missing or not readable by you. See the KVM section above |
missing qemu-system-x86_64, mkfs.vfat and similar | Install the packages in the table above; the message names everything it could not find |
Could not access KVM kernel module from qemu | /dev/kvm exists but your user cannot open it – check the mode and the kvm group |
| A node never answers ssh, minutes in | Usually emulation rather than KVM. The failure prints the tail of the node’s serial console, and the full log is at build/microvm/<fleet>/<node>/console.log |
a 3 node fleet at 4096MiB each needs ... | The memory guard. Lower -worker, or pass fewer nodes |
cached image ... has digest ..., want ... | An interrupted or corrupted download. Remove the named file and run mage dev:vmImage |
Shell Tab Completion
Two commands complete on the command line in this repository. cargoship generates its own completion scripts, and mage installs its own for targets and flags - but neither knows about the positional arguments the generate: targets take, which is what the overlay in this document adds.
cargoship
Cobra generates these. Write one to wherever your shell keeps completions:
cargoship completion fish > ~/.config/fish/completions/cargoship.fish
cargoship completion bash > ~/.local/share/bash-completion/completions/cargoship
cargoship completion zsh > ~/.zsh/completions/_cargoship
cargoship completion --help covers powershell and the per-shell loading details. Nothing else in this document applies to cargoship.
mage: targets and flags
Mage installs its own completion:
mage -install fish # also: bash, zsh, powershell, pwsh
This completes every target - the same list mage -autocomplete prints - and every mage flag, marking -d, -w and --compile as taking a path. Do this first; the overlay below only adds arguments.
mage: positional arguments
Two targets take arguments mage has no way to know about:
mage generate:exampleLine <distro> <minor>
mage generate:latestTag <distro> <minor>
Both arguments are readable off the tree: <distro> is a directory under thirdparty-src/, and <minor> is a directory under thirdparty-src/<distro>/ with the underscore written as a dot (v1_36 on disk, v1.36 on the command line). Deriving them keeps completion correct as release lines are added and dropped, which a hardcoded list does not.
Three things worth knowing before installing any of these:
mage -autocompleteprints target names lowercased (generate:exampleline). Mage matches targets case-insensitively, so either spelling runs, but completion only ever produces the lowercase one. The scripts below lowercase before matching so a hand-typedgenerate:exampleLinestill completes its arguments.generate:exampleLineaccepts1.36as well asv1.36;generate:latestTagpasses the prefix straight through and needs thev. The scripts offer thevform, which both accept.thirdparty-src/is read relative to the current directory, so the argument lists are empty outside a checkout.
Fish
Fish is the one shell where the overlay coexists with mage -install fish, because it goes in conf.d/ rather than the mage.fish that -install owns and overwrites.
File: ~/.config/fish/conf.d/cargoship-mage.fish
# The mage target being completed, lowercased. Empty until one is typed.
function __cargoship_mage_target
set -l tokens (string replace -r --filter '^([^-].*)' '$1' -- (commandline -pxc))
test (count $tokens) -ge 2; and string lower -- $tokens[2]
end
# The distro argument, once it is on the line.
function __cargoship_mage_distro
set -l tokens (string replace -r --filter '^([^-].*)' '$1' -- (commandline -pxc))
test (count $tokens) -ge 3; and echo -- $tokens[3]
end
function __cargoship_mage_takes_args
set -l target (__cargoship_mage_target)
contains -- "$target" generate:exampleline generate:latesttag
end
# Subdirectory names of $argv[1], with underscores as dots.
function __cargoship_mage_subdirs --argument-names parent
test -d "$parent"; or return
for dir in $parent/*/
string replace -r '^.*/([^/]+)/$' '$1' -- $dir | string replace _ .
end
end
complete -c mage -f -n '__cargoship_mage_takes_args; and __fish_is_nth_token 2' \
-a '(__cargoship_mage_subdirs thirdparty-src)'
complete -c mage -f -n '__cargoship_mage_takes_args; and __fish_is_nth_token 3' \
-a '(__cargoship_mage_subdirs thirdparty-src/(__cargoship_mage_distro))'
Reload with exec fish, or source ~/.config/fish/conf.d/cargoship-mage.fish for the current shell.
One caveat: the completion written by mage -install fish registers its target list with no position condition, so at the distro and minor positions fish offers the full list of targets alongside the values this overlay adds. Typing a prefix (rk, v1.3) narrows it immediately. The alternative is to drop mage’s own completion entirely, which costs you flag completion for -l, -d, -v and the rest, so the noise is the better trade.
Bash
This one supersedes mage -install bash rather than adding to it - bash allows a single complete -F per command, so the overlay has to serve targets too, and mage’s flag completion is lost. Install this file instead of running mage -install bash.
File: ~/.local/share/bash-completion/completions/mage
_cargoship_mage() {
declare -F _init_completion >/dev/null || return
local cur prev words cword
# -n : is load-bearing. Colon is in COMP_WORDBREAKS and _init_completion does not
# exclude it by default, so without this "generate:exampleline" arrives as three
# words and every index below is off.
_init_completion -n : || return
if ((cword == 1)); then
COMPREPLY=($(compgen -W "$(mage -autocomplete 2>/dev/null)" -- "$cur"))
# Readline still replaces only the text after the last colon.
if declare -F _comp_ltrim_colon_completions >/dev/null; then
_comp_ltrim_colon_completions "$cur"
else
__ltrim_colon_completions "$cur"
fi
return
fi
case "${words[1],,}" in
generate:exampleline | generate:latesttag) ;;
*) return ;;
esac
local dir
case $cword in
2) dir=thirdparty-src ;;
3) dir=thirdparty-src/${words[2]} ;;
*) return ;;
esac
[[ -d $dir ]] || return
local entry
local -a values=()
for entry in "$dir"/*/; do
[[ -d $entry ]] || continue
entry=${entry%/}
entry=${entry##*/}
values+=("${entry//_/.}")
done
COMPREPLY=($(compgen -W "${values[*]}" -- "$cur"))
}
complete -F _cargoship_mage mage
Bash-completion autoloads the file on the first mage<TAB>; start a new shell to pick it up.
Zsh
As with bash, this supersedes mage -install zsh and drops mage’s flag completion.
File: ~/.zsh/completions/_mage
#compdef mage
local -a values
local dir
if (( CURRENT == 2 )); then
values=(${(f)"$(mage -autocomplete 2>/dev/null)"})
# compadd, not _describe: target names contain a colon, which _describe reads
# as a value:description separator and would offer "generate" alone.
compadd -a values
return
fi
case "${words[2]:l}" in
generate:exampleline | generate:latesttag) ;;
*) return ;;
esac
case $CURRENT in
3) dir=thirdparty-src ;;
4) dir=thirdparty-src/${words[3]} ;;
*) return ;;
esac
values=($dir/*(N/:t))
values=(${values//_/.})
compadd -a values
The directory has to be on fpath before compinit runs, so in ~/.zshrc:
fpath=(~/.zsh/completions $fpath)
autoload -U compinit && compinit
Signing, and the embedded trusted root
Cargoship signs the distro.yaml blob in a package and verifies it on pull, install and verify. The wrapper around cosign lives in pkg/signing, a fork of zarf’s package of the same name - docs/agent/choice-signing-fork.md records why it is a fork and what it deliberately leaves out.
A local cosign keypair whose password comes from COSIGN_PASSWORD is the only supported signing method. Cloud KMS key references and keyless (Fulcio/OIDC) signing and verification are not available: all of them need to reach a cloud endpoint, and the management node is air-gapped. Whether keyless should return is #848.
cosign generate-key-pair # cosign.key, cosign.pub
COSIGN_PASSWORD=... cargoship sign ./my-package --signing-key cosign.key
cargoship verify ./my-package --key cosign.pub
The end-to-end coverage is in test/e2e/noncluster, and it does real signing against a real generated keypair rather than mocking cosign:
go test -run 'TestCargoshipSign|TestCargoshipSignedRoundTrip|TestCargoshipVerifyCommand' ./test/e2e/noncluster/
There is no embedded trusted root any more
An earlier revision embedded pkg/signing/embedded_trusted_root.json, the Sigstore TrustedRoot, so a keyless verification that supplied no --trusted-root had one to fall back on. Keyless verification is gone and nothing key-based reads a trusted root, so the file and its manual refresh procedure went with it. That is one fewer pin with no automatic owner - see magefiles/AGENTS.md for why that matters.
If keyless verification returns through #848, the trusted root comes back with it, and so does the obligation to refresh it. The procedure that worked was:
TUF_ROOT=$(mktemp -d) sh -c 'cosign initialize && cp "$TUF_ROOT/tuf-repo-cdn.sigstore.dev/targets/trusted_root.json" <destination>'
Third-Party Engine Source (thirdparty-src/)
This document explains what lives under thirdparty-src/ and why it’s handled differently from the rest of the repository.
What it is
thirdparty-src/<distro>/<version>/ holds raw, unmodified source files pulled from an upstream repo at an exact tag - e.g. thirdparty-src/k3s/v1_35/{zz_server.go,zz_agent.go}, pulled from k3s-io/k3s at tag v1.35.3+k3s1. version is truncated to the minor version (v1_35, not the full v1.35.3-k3s1 patch tag): patch releases are assumed not to change the flag set, so one pull/generate covers every patch in that minor line. The zz_ prefix marks these as pulled/tool-managed rather than hand-authored, the same convention used for the generated structs under pkg/engineconfig/gen.
Three repos are pulled, one per <distro> directory, listed in thirdparty-src/pins.json:
k3s- k3s-io/k3s. Pullszz_server.goandzz_agent.gofor the flag set, pluszz_stage.go, which declares the packaged components - see “Packaged components (--disable)” below.rke2- rancher/rke2. The same two, pluszz_root.goandzz_k3sopts.gofor its flag composition - see “RKE2’s flag composition” below - andzz_types.gofor its packaged components.upstream- kubernetes/kubernetes. Not an engine: it supplies the kubeadm API surface theupstreamdistro’s engine config is validated against, aszz_constants.goandzz_types.go. Seepkg/engineconfig/gen/upstream/.
Everything below is about the two engine pulls unless it says otherwise.
These files exist so pkg/engineconfig/extract can statically parse them with go/ast to recover the urfave/cli flag declarations k3s/RKE2 ships for a given version. That, in turn, lets mage generate:engineConfig (see magefiles/pkg/gen/engineconfig/engineconfig.go) generate a typed Go struct describing the valid config.yaml keys for that version - see docs/agent/design-config-codegen.md for the full design rationale.
Why source files instead of a module dependency
k3s and RKE2 are not safe to go get/import as library dependencies: their go.mod files carry replace directives that would leak into (and corrupt) this module’s own dependency graph (see golang/go#30354). Statically parsing their source text with go/parser sidesteps that entirely - the files are never compiled, never imported, and never linked into any Cargoship binary. They exist purely as text for the extractor to read.
Why thirdparty-src/ has its own go.mod
Because these files reference packages (github.com/urfave/cli/v2, github.com/k3s-io/k3s/pkg/version, …) that this repository does not vendor, letting the main module’s tooling discover them causes real breakage: go build ./..., go vet, and IDE tooling (gopls) all try to resolve those imports and fail.
thirdparty-src/go.mod declares a separate, empty module so the Go toolchain treats thirdparty-src/ as outside the main module boundary. go build ./.../go vet from the repo root no longer descend into it. IDEs still open a second module context for it, which can surface unresolved-import diagnostics on the files themselves; the workspace’s .vscode/settings.json sets gopls.directoryFilters to -thirdparty-src to stop gopls from loading that module at all.
The module boundary is not honoured by every tool. CodeQL’s Go autobuilder walks the whole checkout looking for module roots rather than building the main module, finds this one, and tries to resolve the imports above - mis-resolving some of them (github.com/k3s-io/k3s to v1.21.9, whose go.mod declares its path as github.com/rancher/k3s), running go mod tidy -e, and downloading modules this repository deliberately does not depend on. CodeQL’s paths-ignore cannot exclude the directory, because that filter does not apply to languages analyzed by building them, and Go is one. .github/workflows/codeql.yaml therefore uses build-mode: manual and spells the package list out - ./api/... ./cmd/... ./config/... ./fuzz/... ./internal/... ./pkg/... ./types/... . - rather than ./..., which keeps the toolchain inside the main module. Anything else added to CI that walks for go.mod files needs the same treatment.
Layout
thirdparty-src/
<distro>/ # k3s, rke2, or upstream (kubernetes/kubernetes)
<version>/ # e.g. v1_35 (minor version only, see above)
zz_server.go # verbatim upstream source
zz_agent.go
zz_root.go # rke2 only -- commonFlag, shared by server/agent
zz_k3sopts.go # rke2 only -- K3SFlagOption/copyFlag/dropFlag/hideFlag
zz_stage.go # k3s only -- DisableItems, the packaged components
zz_types.go # rke2 -- DisableItems/CNIItems/IngressItems; upstream -- kubeadm v1beta4 types
zz_constants.go # upstream only -- kubeadm constants
SOURCE.txt # repo, tag, resolved commit, and pulled file list
pins.json # the pinned tags and file lists, see below
go.mod # module boundary marker, see above
RKE2’s flag composition
RKE2 doesn’t declare its config.yaml flags outright the way k3s does. zz_server.go/zz_agent.go import k3s’s own command and wrap it at runtime via mustCmdFromK3S(cmd, K3SFlagSet{...}) (defined in zz_k3sopts.go): a map, keyed by k3s flag name, of copyFlag/dropFlag/hideFlag (or an inline {Usage: "...", Hide: true}-style literal) saying whether that k3s flag survives into RKE2 at all, and whether it’s hidden. RKE2 then appends a small literal of its own additions (serverFlag/deprecatedFlags) and a shared commonFlag literal declared in zz_root.go.
mage generate:engineConfig reproduces this: for each RKE2 target it extracts the sibling k3s manifest (thirdparty-src/k3s/<version>/), applies the K3SFlagSet transform parsed from RKE2’s zz_server.go/zz_agent.go, then appends RKE2’s own additions and commonFlag. This is why an RKE2 version’s pull must always be paired with a k3s pull at the same minor version.
Some of commonFlag’s entries have a computed Name (a package constant, e.g. images.KubeAPIServer, or a concatenation like podtemplate.KubeAPIServer + "-extra-mount") rather than a string literal. Those aren’t statically resolvable to a flag name without also pulling and evaluating pkg/images/pkg/podtemplate, so they’re listed in the generated file’s “not resolvable” comment block instead of silently guessed at or dropped.
Packaged components (--disable)
The flag extraction above recovers flag names. disable: is the one key whose values also come from a fixed, version-specific vocabulary - the packaged components (addons) the engine deploys unless told not to - and neither distro declares that vocabulary in the files that hold the []cli.Flag{...} literal, so each pulls one more file for it.
k3s declares it as a comma-separated string const in pkg/cli/cmds/stage.go, pulled as zz_stage.go:
const DisableItems = "coredns, servicelb, traefik, local-storage, metrics-server, runtimes"
There is a second, build-tagged copy of that const in pkg/cli/cmds/nostage.go (//go:build no_stage) listing a strict subset. Cargoship targets the ordinary !no_stage build, so only stage.go is pulled.
RKE2 declares three string slices in pkg/cli/types.go, pulled as zz_types.go:
var (
DisableItems = []string{"rke2-coredns", "rke2-metrics-server", "rke2-snapshot-controller", ...}
CNIItems = []string{"calico", "canal", "cilium", "flannel"}
IngressItems = []string{"traefik", "ingress-nginx"}
)
zz_server.go refers to those as rke2cli.DisableItems / rke2cli.CNIItems when building its usage strings, which is why they can’t be recovered from zz_server.go alone.
RKE2’s list is a computed union, not a copy of DisableItems. disableExceptSelected (in zz_server.go) pushes every un-selected CNI and ingress chart onto --disable at startup - rke2-<item> and rke2-<item>-crd for each entry of CNIItems and IngressItems - and validateCNI adds a literal rke2-multus/rke2-multus-crd pair the same way. So disable: [rke2-ingress-nginx] is legal even though rke2-ingress-nginx appears nowhere in DisableItems; this repo’s own example/rke2-*/distro.yaml relies on exactly that. mage generate:engineConfig reproduces the union: DisableItems, plus the chart-name pair for each CNI and ingress item, plus every string-literal clx.Set("disable", ...) call it finds in zz_server.go - sorted and deduplicated so the generated file is stable across runs.
pkg/engineconfig/extract/components.go does the parsing: StringListDecl handles both the const-string and []string{...} shapes, and DisableSetCalls scans for the Set("disable", ...) calls.
Pulling a new version
thirdparty-src/pins.json is the single source of truth for what gets pulled: one entry per upstream repo, listing the files to copy and every tag to copy them at.
{
"distros": [
{
"name": "k3s",
"repo": "https://github.com/k3s-io/k3s",
"files": ["pkg/cli/cmds/server.go", "pkg/cli/cmds/agent.go", "pkg/cli/cmds/stage.go"],
"tags": ["v1.36.4+k3s1", "v1.35.8+k3s1"]
},
{
"name": "rke2",
"repo": "https://github.com/rancher/rke2",
"files": [
"pkg/cli/cmds/server.go",
"pkg/cli/cmds/agent.go",
"pkg/cli/cmds/root.go",
"pkg/cli/cmds/k3sopts.go",
"pkg/cli/types.go"
],
"tags": ["v1.36.3+rke2r1", "v1.35.7+rke2r1"]
}
]
}
There is no destination path in the manifest: it is derived from name plus the tag’s major.minor, so k3s at v1.35.8+k3s1 always lands in thirdparty-src/k3s/v1_35/. That keeps the truncation rule described above from drifting out of sync with the tag it was truncated from. A tag that does not start with vMAJOR.MINOR.PATCH is rejected rather than guessed at.
mage generate:pullEngineSource reads the manifest and, for each pinned tag, clones repo@tag to a throwaway temp directory, copies out files verbatim into the derived directory (prefixed zz_), and writes SOURCE.txt recording the repo, tag, and resolved commit for traceability. It never runs go get or touches this repo’s module graph.
You rarely need to edit pins.json by hand. mage generate:latestTag <distro> <vMAJOR.MINOR> resolves the newest non-RC tag upstream publishes for that minor line, pins it - replacing whatever that line held, or adding the line newest-first if it is new - and re-pulls that version’s source if the pin moved, so the manifest and the committed files can never disagree. When nothing moved it prints unchanged and writes nothing.
mage generate:latestTag k3s v1.37 # start tracking a new minor line
mage generate:updatePins # refresh every line already pinned
generate:updatePins runs that same cycle over every minor line in the manifest, which is the usual way to answer “are we behind upstream?”. Both targets touch the network, and both leave struct generation to you: follow with mage generate:engineConfig for any version that moved. The version directory name is sanitized into a valid Go package name (dots become underscores) since it becomes the generated package’s directory and name directly.
Consuming it
mage generate:engineConfig walks every thirdparty-src/<distro>/<version>/ directory and writes generated structs to pkg/engineconfig/gen/<distro>/<version>/, alongside a zz_addons.go holding that version’s Addons (valid disable: values), CNIs, and IngressControllers. zz_addons.go is written for every version even when nothing was extracted - the generated registry references those vars from every package, so one package without them would not compile. An empty Addons means no component source was pulled for that version, and consumers treat it as “no data to check against” rather than “nothing is valid”. For k3s (and any distro that declares its flags outright) it parses whichever of zz_server.go/zz_agent.go it finds directly; for RKE2 it composes as described above. See docs/dev/mage.md for the broader Generate namespace.
It also (re)writes pkg/engineconfig/gen/zz_registry.go, wiring every distro/version it just generated into gen.Registry so nothing needs hand-maintaining as new versions are pulled. types/distrocfg (RancherCommon.ConfigureEngine) uses gen.Lookup to find the struct matching a node’s distro and minor version, and drops (with a warning) any config.yaml key that isn’t a real flag for it – falling back to the previous unvalidated, pass-everything-through behavior with a warning if that distro/version was never pulled/generated. On controllers it also checks each disable: value against Entry.Addons via gen.UnknownAddons; an unrecognized component is warned about but kept, since the addon vocabulary is composed rather than read straight off a declaration and dropping the entry would silently deploy something the user asked to be without. See “Consumption: wired into types/distrocfg” in docs/agent/design-config-codegen.md for the full picture.
check-go-mod.yaml
Triggers: pull requests, merge groups.
Two checks that keep the module graph and its vendored security metadata honest:
- Runs
go mod tidyand fails if it produces a diff against the committedgo.mod/go.sum. - Runs
mage dev:verifyVendorto check the four generatedosv-scanner.tomloverride files undervendor/are present and current. These overrides keep the Scorecard Vulnerabilities check reporting on cargoship’s own code rather than on the JS/Python tooling its Go dependencies happen to ship in their own repos.
go mod vendor deletes and regenerates vendor/, so any PR that re-vendors (which Dependabot always does) drops these override files and fails this check on the first push. fix-vendor-osv.yaml regenerates and pushes them back onto Dependabot’s branch so the check passes on rerun. A genuinely dropped module or newly uncovered lockfile still fails here, since fixing that needs a manual edit to osvOverrides.
Uses go run ./magefiles/core rather than an installed mage binary - same targets, but built from vendor/ with no extra tool needed on the runner.
check-pins.yaml
Triggers: scheduled every 2 days at 06:00 UTC, and manual dispatch.
Two independent jobs that look for newer upstream versions and file a GitHub issue when they find one:
- check-dnf-pins - runs
mage dev:dnfPinsto check for newer AlmaLinux 10 DNF package versions used in containers and.goreleaser.yaml. If the run produces a diff, it opens or updates a tracking issue (keyed by an HTML comment marker) with the proposed diff. - check-engine-pins - runs
mage generate:updatePinsto check for newer RKE2/k3s engine releases for pinned minor versions, againstthirdparty-src/pins.json. Same open-or-update-issue pattern.
Neither job pushes a fix itself - both leave a diff in an issue for a human to apply by running the named mage target locally.
codeql.yaml
Triggers: push to main, pull requests, merge groups, weekly schedule (Mondays 05:37 UTC).
Runs GitHub CodeQL static analysis over two matrix targets: Go source (build-mode: manual) and GitHub Actions workflow files (build-mode: none).
For Go, CodeQL analyzes by building the code, so the go build step’s package list - spelled out explicitly rather than ./... - decides what gets extracted. Two directories are deliberately excluded from that list:
thirdparty-src/- a second, intentionally non-compiling module holding verbatim upstream k3s/RKE2 source (seedocs/dev/thirdparty-src.md). Anything that walks the checkout for module roots, including CodeQL’s autobuilder, would otherwise try to resolve its unvendored imports and fetch modules from the network.magefiles/- build-tag-gated (//go:build mage) dev tooling with nomainfunction of its own; it can’t be compiled with plaingo build.
The build runs fully offline (GOPROXY=off, GOSUMDB=off, vendored modules only) so any accidental network dependency fails loudly.
commitlint.yaml
Triggers: pull requests (milestoned, opened, reopened, synchronize, edited), merge groups.
Lints the PR title - not individual commit messages - against conventional-commit rules, since release-please squash-merges PRs and the PR title becomes the final commit message on main. A dedicated lint action was written rather than reusing a commit-message-linting GitHub Action, because those lint the commits, not the title.
The milestoned event type is included as a workaround: release-please doesn’t otherwise trigger pull_request workflows, so its PRs are added to a milestone specifically to fire this check.
title_check is a required status check, so the workflow also runs on merge_group to report one. A merge group carries no pull request and therefore no title, so every step is skipped there and the job reports success - the title was already linted on the pull request that got queued.
dependabot-validate.yaml
Triggers: pull requests touching .github/dependabot.yaml or this workflow file.
Validates .github/dependabot.yaml using marocchino/validate-dependabot, then posts the validation result as a sticky PR comment. Catches malformed Dependabot config before it merges and silently breaks dependency update PRs.
deploy-book.yaml
Triggers: push to main.
Builds the project’s mdBook documentation site and deploys it to GitHub Pages by pushing the built output to the gh-pages branch. This is the only workflow that writes with contents: write permission, and it never touches pull requests - it only runs after a merge to main.
e2e-cluster.yaml
Triggers: manual dispatch, pull requests (opened/synchronize/reopened, excluding doc-only path changes).
Runs the “staging” half of the multi-machine end-to-end suite (test/e2e/cluster): it provisions a bootloose cluster of Docker machines, uploads the cargoship package, and renders the engine config on every OS family - then stops before actually starting the engine anywhere.
This is intentionally split out from e2e.yaml into its own workflow so its trigger, concurrency group, and time budget are independent of the checks required on every push. It calls the cargoship packages directly (distro.Create, action.NewPrepare/action.NewApply) rather than a built binary, so it needs no build step.
The other half - actually installing k3s and bringing the cluster fully up - is not run in CI. It isn’t reliable enough on hosted runners to be a required check (even after switching from a three-controller RKE2/etcd topology to single-controller k3s to remove quorum timeouts). That half still exists and runs locally via mage test:endToEndCluster. See docs/dev/e2e-phase-tests.md and docs/agent/choice-e2e-stage-split.md.
On failure, node diagnostics (disk, memory, container logs) are collected and uploaded as an artifact, and leftover bootloose containers are always cleaned up.
e2e.yaml
Triggers: pull requests, merge groups.
Three jobs, “Basic build”:
- build-e2e - builds the
cargoshipbinary (CGO_ENABLED=0, fully vendored/offline build) and uploads it as an artifact. - e2e-noncluster (needs build-e2e) - downloads that binary and runs the non-cluster e2e test groups (misc and package commands), which need no Docker/bootloose infrastructure. Uses
-shortto additionally skip example packages, which would otherwise pull ~1.5GB of engine artifacts and images. - binary-size (needs build-e2e) - compares the PR’s binary size against the latest release’s binary and posts the diff as a sticky PR comment. Pull requests only; there is no pull request to comment on in a merge group.
The cluster-dependent e2e group is deliberately not run here - it needs Docker and provisions five containers, so it lives in its own workflow, e2e-cluster.yaml, on its own trigger. That workflow runs only the staging half, which starts no engine; the full walk installs single-controller k3s across its five machines and runs locally, not in CI.
build-e2e and e2e-noncluster are required status checks, which is why the merge_group trigger is here and why neither trigger carries a path filter. A path-filtered required check reports nothing at all on a change that only touches the ignored paths, so the check stays pending and the pull request can never be queued. The workflow used to skip CODEOWNERS-only changes; it now builds those too, which costs one build in the rare case and removes the stall.
fix-vendor-osv.yaml
Triggers: pull_request_target (opened/reopened/synchronize) touching vendor/**, dependabot PRs only.
Dependabot re-runs go mod vendor on every dependency bump, which deletes and recreates vendor/ - taking the four generated osv-scanner.toml override files with it (see check-go-mod.yaml). This workflow regenerates those files from the base branch (via mage dev:writeOSVOverrides) and commits them back onto the dependabot branch through the git data API.
Uses pull_request_target deliberately so the job runs against the base branch with a write-capable token, without ever checking out the PR’s own tree - avoiding the classic pull_request_target vulnerability of running a write-privileged job against untrusted code. Overrides are generated purely from magefiles/pkg/devtools/osv/osv.go, which no dependency bump touches, so the base branch produces the exact bytes the check wants.
Only runs for dependabot[bot]-authored PRs, since those branches live in this repo and can be pushed to with an app token; forks can’t be pushed to, and nothing else re-vendors on a schedule.
pre-commit.yaml
Triggers: pull requests, merge groups.
Runs the hooks defined in .pre-commit-config.yaml (via prek-action) against all files, so formatting, licensing, and sorting checks can’t drift from what runs locally.
Three hooks are explicitly skipped in CI:
schema-generator,docs-generator-repo: localhooks that shell out tomageand need the full dev toolchain.golangci-lint- already covered separately byscan-lint.yaml.
prune-example-pr-packages.yaml
Triggers: daily schedule (05:23 UTC), manual dispatch (with retention_days and dry_run inputs).
Deletes throwaway GHCR container packages that publish-example.yaml writes when it runs on a pull request (published under ghcr.io/colonel-byte/cargoship-examples/pr-N, so PR test runs never overwrite the release reference). Nothing consumes these once the PR is done, and each one costs gigabytes of registry storage.
Only pr-N namespaces are ever touched - the release reference (cargoship-examples/<flavor>) is never a candidate. Candidates are enumerated from PRs carrying the publish-example label (the label that gates publish-example.yaml on PRs at all), then probed individually and deleted whole (not version-by-version - GHCR refuses to delete a package’s last version, and whole-package delete also removes orphaned untagged child manifests) if their newest version is older than the retention window (default 30 days).
Package names are constructed from data already in the repo rather than listed via the GitHub API, because GITHUB_TOKEN can delete an individual package but can’t list an organization’s packages (GET /orgs/{org}/packages rejects Actions tokens).
publish-example.yaml
Triggers: manual dispatch (choose a flavor or “changed”), workflow_call (from release.yaml after a release), pull requests carrying the publish-example label.
Builds the example distro definitions shipped in this repo using the released cargoship CLI (not one built in this run) and publishes them to GHCR - exercising what a user actually gets from a release, against the definitions the repo ships.
Two jobs:
- select - decides what to publish before building anything (cheap API probes vs. expensive multi-gigabyte builds). “Changed” mode checks every flavor’s newest example version against what’s already in the registry and only selects ones that moved. A named flavor is always published regardless of registry state. Output is grouped into one matrix entry per distro (RKE2 vs. k3s), because flavors of the same distro/version share most of their cached image pulls.
- publish-example - for each distro, builds and publishes its selected flavors sequentially in one job, so the
~/.cargoship-cachebulk (image lists, RPMs) is shared across flavors instead of pulled once per flavor. Each flavor is published under two tags: the regular version tag and a precise<version>-<job>-<short-sha>tag. Signing uses--signing-key env://COSIGN_PRIVATE_KEY(key material never touches disk).
On a pull request, publishing goes to a per-PR namespace (cargoship-examples/pr-N) instead of the real reference, so test runs of the workflow itself never clobber a release artifact. prune-example-pr-packages.yaml cleans those up later. Fork PRs are excluded since they lack the signing/write secrets needed to publish.
The chain from a release is: release-please cuts the release and pushes a tag → the tag push triggers release.yaml’s goreleaser job → that job calls this workflow via needs, ensuring the release archives exist before setup-cargoship tries to install the just-released CLI.
refresh-examples.yaml
Triggers: weekly schedule (Mondays 06:41 UTC), manual dispatch (with a no_cache input for when upstream re-cuts release assets in place).
Regenerates everything under example/ from whatever upstream RKE2/k3s has released since the last run, and opens (or updates) a single pull request with the result. Nothing else generates example/ on a schedule - the mage targets involved are otherwise run by hand - so this turns “staying current with upstream” into a review task instead of something someone has to remember.
Runs three mage targets in a fixed, dependency-ordered sequence: generate:updatePins (re-pulls source for any moved pin, writes thirdparty-src/pins.json) → generate:engineConfig (reads that source into each version’s component vocabulary) → generate:examples (renders values schemas against that vocabulary). Running them out of order means a schema can render empty instead of failing.
After rendering, it runs go build ./... and the schema validation tests against the result and reports pass/fail in the PR body - a render that doesn’t compile still opens the PR (so it’s visible for debugging) but the job itself fails.
The branch (chore/refresh-examples) is force-pushed each week rather than accumulating one PR per run, so any hand edit to that branch is lost on the next refresh by design - template changes belong in magefiles/templates/ instead. Uses a GitHub App token (same one release-please uses) rather than GITHUB_TOKEN, since a PR opened with the default token wouldn’t trigger pull_request workflows like e2e or pre-commit.
Publishing is a separate concern - this workflow only writes definitions; publish-example.yaml picks them up at the next release or on demand.
release-please.yaml
Triggers: push to main, manual dispatch.
Runs googleapis/release-please-action to manage version bumps, CHANGELOG.md, and release PRs, driven by release-please-config.json and .release-please-manifest.json. It mints a GitHub App token up front and hands it to the release-please action for every write it makes - GITHUB_TOKEN itself is never granted write permission.
release.yaml
Triggers: push of any tag.
The actual release pipeline (“goreleaser” job), granted contents: write (create release, upload archives), packages: write (push images), id-token: write (OIDC token for cosign), and attestations: write.
Sets up Syft (SBOM), cosign, ansible-core, Go, QEMU, and Docker Buildx, then logs into two registries (GHCR, Docker Hub) before running GoReleaser (goreleaser release --clean), which builds and publishes binaries, packages, and container images.
Package signing keys (GPG for rpm/deb, a bare RSA key for apk) are written to RUNNER_TEMP - outside the checkout, so no .goreleaser.yaml glob can accidentally pick them up - and removed at the end regardless of outcome. A missing signing secret disables signing for that artifact type with a warning rather than failing the release.
After GoReleaser runs, the job attests packages, images, and a generated AI-BOM (.github/ai-bom.json, stamped with the release version/commit/timestamp) via actions/attest, and uploads the resulting provenance bundles to the GitHub release.
A second job, publish-example, runs after goreleaser completes (only for v* tags) and calls publish-example.yaml as a reusable workflow with secrets: inherit, so a fresh example package gets published against the release that was just cut. Its permissions are written to exactly match what the called workflow needs, since a caller’s grant is a ceiling that can only be narrowed downstream.
scan-lint.yaml
Triggers: pull requests, merge groups.
Two independent lint jobs:
- lint - runs
golangci-lintover the Go source, at the version.pre-commit-config.yamlpins for the hook of the same name. The action installs the latest release when it is given no version, which once left CI linting with a newer golangci-lint than any contributor ran locally, so a failure here reproduced nowhere. Reading the pinned rev keeps the two in step without adding a second pin – dependabot bumps the one in the hook config, and does not update inputs inside a workflow’swith:block. - containers - runs
hadolintagainst each container Dockerfile individually (ansible,base,deb,ubi), matrix-style. Paths are listed explicitly rather than using hadolint’s recursive glob mode, which would otherwise pull in vendored Dockerfiles undervendor/.
scorecard.yaml
Triggers: branch protection rule changes, weekly schedule (Saturdays 01:30 UTC), push to main.
Runs the OpenSSF Scorecard supply-chain security analysis (ossf/scorecard-action) against the default branch, uploads the SARIF results as a workflow artifact, and publishes them to GitHub’s code-scanning dashboard. Adapted from zarf-dev/zarf’s equivalent workflow. Default permissions are read-all, with security-events: write and id-token: write granted only to the analysis job (for code-scanning upload and Scorecard’s OIDC badge).
test-build-containers.yaml
Triggers: pull requests (opened/reopened/synchronize/labeled) touching containers/**, .goreleaser.yaml, .release-please-manifest.json, or this workflow file - gated on the PR carrying the docker label.
Actually builds all four container images (ansible, base, deb, ubi) via goreleaser release --snapshot --clean --skip=sign,sbom, closing a gap scan-lint.yaml’s hadolint job (syntax only) does not cover: a Dependabot bump that breaks the actual build wasn’t otherwise caught until release.
These Dockerfiles are written for GoReleaser’s dockers_v2 pipe, not a plain docker build - several COPY nfpm-produced .rpm/.deb packages and extra files that only GoReleaser stages. That pipe has no per-image build flag, so a snapshot build always builds all four images together; there’s no cheaper way to build “just the one that changed.”
labeled is included in types because Dependabot’s docker label is applied after the PR opens - without it, the if condition would be evaluated (and fail) on the opened event before the label exists, and nothing would re-trigger once it’s added.
--snapshot skips publish/validate/announce entirely - nothing is pushed anywhere.
unit-tests.yaml
Triggers: pull requests, merge groups.
Runs the fast test suites on a single ubuntu-latest runner, in one unit job with two steps.
The first step is go run ./magefiles/core test:unit, which is everything except the two e2e groups - those have runners of their own in e2e.yaml and e2e-cluster.yaml, drive ./test/..., and need Docker and the better part of an hour. See Test.Unit in magefiles/test.go for the package list that leaves.
The second step replays the fuzz seed corpus with test:fuzz. It takes about a second and needs nothing the first step did not already need, so it rides along here rather than paying for a runner of its own. This is the only job that runs the fuzz/ package - see ../dev/fuzz-tests.md.
Both steps run fully offline (GOFLAGS=-mod=vendor, GOPROXY=off, GOSUMDB=off, GOTOOLCHAIN=local), so accidental network access fails fast rather than passing on a runner and failing on an air-gapped management node. Both invoke the mage target through go run ./magefiles/core rather than a mage binary, which builds from vendor/ and needs nothing installed on the runner.
Concurrency is grouped per ref with cancel-in-progress, so a new push supersedes an in-flight run. Default permissions are contents: read.
AI usage in cargoship’s development
This document discloses how AI coding agents are used to build cargoship. It is not a CycloneDX ML-BOM or SPDX AI profile - those describe AI/ML models and datasets shipped inside a product, and cargoship has none: it’s an Ansible/Kubernetes deployment CLI with no inference or model-serving code of its own. What follows instead is a disclosure of the tools used to write cargoship, in the same spirit as the supply-chain artifacts described in the security policy - an SBOM discloses what’s in a release; this discloses how the release was produced.
Tools in use
Claude Code (Anthropic, Claude models) is used across the repository for code changes, documentation, and the design records under this directory. Its operating rules for this repository live in the various AGENTS.md files scattered through the tree - the repository root’s, and the per-directory ones it points to - which tell an agent how this codebase expects to be worked in, not just what the code does.
Human review
AI-authored and human-authored changes go through the same gate: a reviewed pull request, and the same CI checks - CodeQL, linting, the test suites, OpenSSF Scorecard - regardless of who or what wrote the diff. Nothing here grants an AI-authored change an exemption from review or from CI.
Where the provenance trail lives
Two records carry this forward:
choice-*.mdfiles in this directory record non-obvious design decisions and the tradeoffs behind them, agent-driven or not - seeAGENTS.mdfor why they’re treated as constraints rather than history.- Every tagged release publishes
ai-bom.jsonas a release asset, attested the same way as the release’s SBOM and provenance. It’s a per-release snapshot of this document’s claims - which tools, what they’re used for, and the human-review statement above - rather than a list of what’s in that particular release.
Why pkg/packager/actions is a fork, and why a create-time cluster wait now fails
cargoship runs the onCreate actions in a distro package through zarf’s action executor. runCreateActions in pkg/packager/assemble/actions.go renders each action with cargoship’s template engine and then hands it to actions.Run, which owns everything that makes an action an action: shells, retries, timeouts, output capture, setVariables and setValues. Only the rendering was ever cargoship’s - see choice-v1alpha1-action-conversion.md for why the conversion into zarf’s normalized types happens at the call site.
The executor is now a fork. The reason is one import.
What the import cost
zarf’s src/pkg/packager/actions imports src/pkg/wait, and uses it in exactly one function, runWaitAction. zarf’s wait package serves both kinds of wait from one file: ForNetwork, which probes HTTP or TCP with the standard library, and ForResource, which is kubectl wait’s logic with retries for a resource that does not exist yet. The second one brings:
zarf/src/pkg/wait
-> k8s.io/kubectl/pkg/cmd/wait + cmd/get
-> k8s.io/cli-runtime/pkg/resource
-> sigs.k8s.io/kustomize/api + kyaml
-> zarf/src/pkg/cluster
Measured: 92.98MB to 87.11MB, 5.87MB, and k8s.io/kubectl, sigs.k8s.io/kustomize, k8s.io/cli-runtime, zarf/src/pkg/wait and zarf/src/pkg/cluster all go to zero in go list -deps ./cmd/cargoship. Go cannot drop a transitive import, so the only way to stop linking kubectl is to stop linking the package that imports the package that imports it.
Why a cluster wait was droppable
Actions in the distro spec has one field, OnCreate. There is no deploy-time or remove-time action path in cargoship: actions run while the package is being built, and nothing else runs them. A package is built on a management node that is typically air-gapped.
So wait.cluster in an onCreate action asks cargoship to wait for a Kubernetes resource at a point where there is no cluster in the picture. Upstream would resolve a client from the ambient kubeconfig, which is either absent - and the action fails five minutes later with a connection error - or present and pointing at some cluster that is not the one this package will be applied to. Neither is a thing an operator wants.
runWaitClusterAction therefore fails immediately, naming the action and the resource, and says to use wait.network instead. wait.network is unaffected: the network half of zarf’s wait.go is forked into pkg/packager/actions/wait.go, and it is standard-library HTTP and TCP probing with retry-go around it.
This is a behaviour change, not dead code removal
wait reaches schema/distro-v1alpha1-schema.json because Actions.OnCreate embeds zarf.ZarfComponentActionSet, so a wait.cluster action is something the published schema accepts and an operator could have written. Nothing in the repo uses one - no example, no fixture, no test - and the e2e action fixture exercises cmd actions only. But absence from this repo is not absence from every distro.yaml.
The schema was deliberately left alone. Narrowing it would mean retyping Actions.OnCreate off zarf’s action set, which is the schema break choice-v1alpha1-action-conversion.md exists to avoid, and it would reject the field at parse time instead of explaining the problem at the point of use. A package carrying a create-time cluster wait now fails with a message that says what to do; that is better than the wait it used to get, and better than a schema validation error that cannot explain itself.
The other upstream divergence
Run used to build a VariableConfig when handed nil, out of src/internal/packager/template - an internal package a fork cannot import. Every caller here supplies one and has to: a config built per call loses the variables an earlier action set, so setVariables would not reach the action after it, which is the bug newVariableConfig was written to avoid. The fork returns an error for a nil config rather than quietly running with one that forgets.
Resyncing with upstream
Three places diverge, and nothing else should. When pulling zarf’s actions.go forward, re-apply exactly these:
- the
src/pkg/waitimport is dropped, runWaitClusterActionreturns the refusal rather than callingwait.ForResource,runWaitNetworkActioncalls this package’sForNetwork, andRunerrors on a nilVariableConfiginstead of callingptmpl.GetZarfVariableConfig.
Everything else in the file is zarf’s and should stay byte-for-byte theirs, because the action semantics an operator relies on - retry counts, timeout handling, how mute interacts with output capture - are upstream’s to define.
Why age sits beside Ansible Vault rather than replacing it
Cargoship encrypts four registry credential fields per registry – auth.user, auth.pass, auth.token, tls.ca – and decrypts them at apply time, when the engine’s registry configuration is written. That was Ansible Vault only, built on a single shared passphrase (internal/clustercfg/vault.go). It now also supports age, and the two coexist permanently.
The operator-facing guide is age-encryption. This document records the decisions behind it, and in particular the two things a future change is most likely to get wrong: the probe in EncryptConfig cannot be re-added for age, and the recipient record in metadata must never be read back as key material.
Coexistence rather than migration
Both formats carry a self-identifying header: $ANSIBLE_VAULT and, because age values are armored to live in a YAML scalar, -----BEGIN AGE ENCRYPTED FILE-----. Decryption therefore dispatches on the value itself, and one document can hold both formats with no ambiguity.
That makes coexistence nearly free, so it was chosen over a migration:
- No schema field naming the format, so no field that can disagree with the value beside it. The recipient record added later is the one exception, and it is kept inert for exactly this reason – see The recipient record, and why it is inert.
- No mode flag, so no command whose meaning depends on an invocation the reader of the file cannot see.
- No migration deadline, so a legacy vaulted credential can stay as it is indefinitely.
cluster.IsEncrypted (api/colonel-byte.dev/v1alpha1/cluster/spec.go) is the union of the two checks and is what non-crypto code asks. AgeHeader is spelled out there rather than taken from filippo.io/age/armor, so the package defining the public API types stays free of crypto dependencies; TestAgeHeaderMatchesArmorHeader pins the constant to armor.Header so the two cannot drift.
The vault command group keeps its name despite now covering both formats. Renaming it would break every operator’s scripts to fix a word; the group’s Long string says what it covers instead.
Why the probe cannot be re-added
EncryptConfig probe-decrypts any value that is already ciphertext. The probe answers exactly one question – is this value under the same key I am about to encrypt with? – and it exists because DecryptRegistryAuth applies one vault password to every field in the document. A file encrypted under two vault passwords is a file no apply can read, so the probe turns that state into an error at write time rather than a failure on the cluster later.
For age recipients that probe is impossible, and it is also not protecting anything.
It is impossible for two reasons, and the second is the durable one.
An X25519 stanza is -> X25519 <ephemeral share>: two fields, no key hash, no fingerprint. Ciphertext anonymity is a design goal of the format. This does not generalise to every recipient type – an agessh stanza carries a 32-bit hash of the public key, which is exactly the identifier the probe would want – so it is not on its own a reason the probe cannot exist.
The reason it cannot is that nothing here can read a stanza at all. age.ExtractHeader exists, but the Stanza fields it would yield live behind filippo.io/age/internal/format, and an internal package is not importable. Cargoship would have to parse the header text itself, duplicating a format it does not own, to recover an identifier that only some recipient types carry. A recipient is a public key, which cannot decrypt; there is nothing to compare against, and no supported way to reach the thing that would be compared.
It is unnecessary because the invariant it defends is a property of the vault code path, not of encryption in general. age.Decrypt is variadic over identities, an identity file holds many, and every configured identity is tried. A document whose values are encrypted to different recipient sets is readable by anyone holding a matching key for each, so “several keys in one document” is a supported state for age rather than the broken one it is for vault.
So EncryptConfig probes only when the existing value is Ansible Vault and the format being written is Ansible Vault. Every other combination skips the value on the strength of IsEncrypted alone, which is all idempotency needs. TestEncryptConfigSkipsVaultValuesWhenWritingAge and TestEncryptConfigAgeIsIdempotent hold that shape in place.
A future change that tries to restore a general probe will find that it cannot be written, and a change that tries to approximate one – trial-decrypting with a configured identity, say – would be answering a different question (can I read this?) than the one the probe asks (is this under the key I am writing with?), while making encrypt-file require a private key it otherwise has no use for.
What that leaves, and how it is handled
The one real cost is rotation-by-mistake: running encrypt-file with a new recipient set against an already-encrypted file does nothing, and the operator may believe it did something. This is addressed by reporting the skip rather than by detecting it cryptographically – the command knows perfectly well that it skipped an encrypted value; what it cannot know is whether that was the right outcome, so it says so and lets the operator decide.
rewriteConfig’s wanted callback is the only thing that knows why a value was left alone, so it carries the reason out as a Skip{Path, Reason}. An empty reason means the skip is not worth reporting, which keeps absent paths, empty values, and the vault-under-the-same-password case – where the probe did answer the question – silent without every caller filtering them itself. skipReason names rekey in all three cases it covers, because rekey is what does the thing the operator was probably attempting.
The placement of the warnings matters as much as their text. finishVaultFile emits them before both its --dry-run branch and its len(changed) == 0 early return, since a run that changed nothing is exactly the run the reporting exists for; emitting them after either would leave the loudest failure as the quietest output. They go to stderr through the logger, so a dry run piped to a file still yields a clean document.
Changing which keys a document is encrypted to is rekey’s job, and always was. rekey reads with one key and writes with another, so cross-format migration falls straight out of the shape it already had: rekey --vault-password-file old.txt --age-recipient age1... is the whole vault-to-age path, with no new command and no window in which plaintext touches the disk.
The recipient record, and why it is inert
A document encrypted with age carries metadata.encryption.age.recipients and lastModified, written by internal/clustercfg/vaultmeta.go. It is the answer to issue #392, and it is in direct tension with a bullet three sections above: no schema field naming the format, so no field that can disagree with the value beside it. This is exactly such a field. It can disagree with the ciphertext beside it, nothing can detect that it has, and for the reasons in the probe section nothing ever will be able to.
It was accepted anyway, under one restriction that the entire justification rests on:
Cargoship writes this block and never reads it back as key material. It is read for exactly one purpose – comparing what the document claims against what the operator just named on the command line – and the result of that comparison is a warning and nothing else. No command encrypts to a key that came out of the file being edited.
That restriction is not a matter of taste. If rekey defaulted its recipients to the recorded list when none were given, a one-line edit in a pull request would silently redirect every credential in the file to the author’s key, and the resulting ciphertext would be indistinguishable from a correct rekey – same armor, same length distribution, same absence of any recipient identifier. The reviewer’s only signal would be the YAML line the attacker wrote. So rekey with an identity and no recipients still fails with no age recipients configured, exactly as it did before the block existed, and that failure is a feature. Any future change that proposes “rekey could just use the recipients already in the file” is proposing this attack; the convenience it buys is real and is not worth it.
What the field does buy, given that restriction, is the thing the probe section says is the one real cost: an operator can read the file and see which keys it was encrypted to, and a reviewer can see a recipient set change as a line of YAML rather than as a wall of re-randomized ciphertext. Those are documentation wins, and documentation is a thing a file is allowed to be wrong about in a way ciphertext is not.
Placement, and the shape of the block
It sits in metadata, beside name, rather than in spec. spec is the desired state an apply acts on, and this is a record of something that already happened to the file; putting it there would invite exactly the reading – “this is configuration, so something must consume it” – that the restriction above forbids. The types are in api/colonel-byte.dev/v1alpha1/cluster/spec.go as plain strings, keeping that package crypto-free the same way AgeHeader does.
There is a section per format (encryption.age) rather than one flat list, so a document holding both Ansible Vault and age credentials has somewhere to say so later if that ever becomes worth saying. Only age needs a record today: a vaulted value is read with the one password the operator already supplies by name, so there is nothing about it a file could usefully record.
lastModified is included despite being churn, because “when was this last re-encrypted” is the question that follows “which keys is it encrypted to” often enough to be worth the line. It is RFC 3339 and always double-quoted, since a bare timestamp is a shape YAML is free to read as something other than a string. The clock is var now = time.Now in vaultmeta.go rather than four signature changes; a test concern is not worth reshaping three exported functions.
An SSH recipient keeps its authorized_keys comment, because that is the part that says whose key it is, and whose key it is was the point. Comparison, though, is on the key itself – recipientKey strips options and comments via ssh.ParseAuthorizedKey – so re-labelling a key or reordering a recipients file does not read as drift. The question the comparison answers is who can read this file, and neither of those changes the answer.
When it is written, and why a no-op writes nothing
EncryptConfig writes the record only when it is writing age and len(changed) > 0. A run that encrypted nothing touches nothing, including the timestamp.
That condition is load-bearing in two directions. It keeps encrypt-file byte-for-byte idempotent, which TestEncryptConfigAgeIsIdempotent and TestEncryptConfigLeavesTheRecordAloneWhenNothingChanged both hold. And it keeps finishVaultFile’s len(changed) == 0 early return honest: without it, a run that reported “nothing to encrypt” would nonetheless have a modified document to write, and the command would either lie or write behind its own report.
RekeyConfig rewrites the record when the target writes age and strips it when the target writes vault. DecryptConfig strips it unless age ciphertext remains somewhere in the document – checked with bytes.Contains(doc, []byte(cluster.AgeHeader)) rather than assumed, since encrypt-path can leave age values outside the credentials the whole-file commands walk. A list of age recipients above a file that holds no age ciphertext is worse than no list: it reads as a file that is still protected.
A metadata mapping written in flow style – metadata: {name: e72} – is refused rather than recorded. Flow style has no block collection, so splicing a nested block into one produces a document that no longer parses, with the credentials already encrypted into it. errNoMetadataBlock carries that out, recordRecipients swallows it, and reportRecipientRecord in cmd/misc_vault_encrypt_file.go turns it into a warning. Declining to record a fact about a file is much the smaller loss, and it is the same reasoning inFlowCollection already encodes for the value splice.
Everything in vaultmeta.go is a byte-level splice for the reason spliceScalar is: go-yaml re-indents multi-line literals and truncates ciphertext nested in sequences, so nothing here re-renders the document.
Why skipReason’s wording stays
The age-to-age skip still says the value is encrypted to age recipients already and that cargoship cannot tell whether they are the ones you named. That remains exactly true – it is a statement about the ciphertext, and no record changes what the ciphertext says about itself.
The new warning is a different claim about a different thing: what the document says it was encrypted to. They are emitted together and they are both worth having. A future change that “fixes” the skip message to mention the record would be merging a cryptographic fact with a piece of documentation, which is the confusion this whole section exists to prevent.
reportRecipientRecord lives in cmd rather than in clustercfg, because clustercfg does no terminal I/O and the warning is about the document as a whole rather than about one credential path – so it does not fit Skip{Path, Reason}. It is in encrypt-file specifically, not in finishVaultFile: decrypt-file removes the record and rekey rewrites it, and in both cases that is the operator getting precisely what they asked for. It is called before finishVaultFile for the same reason the skip warnings are emitted where they are – a run that changed nothing is the run it exists for.
It reads the prior claim from the document as it was read, not as it will be written, since a run that encrypted something has already replaced the record. And it fires only when something was skipped: with no skips, every credential in the file is on the keys just named and the record agrees with itself.
Compatibility
Metadata carries additionalProperties: false, so a document holding this block fails cargoship validate on a binary built before it existed. An apply is unaffected – clustercfg.Parse is a non-strict unmarshal, and the generated schema is used only by cargoship validate (cmd/misc_validate.go). Both copies of the schema have to be regenerated together with mage generate:schema, since CI does not run the generator.
Why encryption picks a format but decryption never does
Decryption reads the header. Encryption has to choose, and the rules are in Keyring.EncryptFormat:
- An age recipient beats a vault password that came from the environment.
CARGOSHIP_VAULT_PASSWORDandANSIBLE_VAULT_PASSWORDare typically exported once in a shell profile on a machine already using vault. Treating that as a conflict would make--age-recipientunusable in exactly the place age is most wanted. - An age recipient together with an explicit
--vault-password-fileis an error rather than a third precedence rule. Both flags name the key to write with, a value is written in one format, and choosing on the operator’s behalf means writing a secret under a key they did not pick. That is the one case worth stopping for instead of guessing well. - Otherwise vault, unchanged.
The keyring therefore has to remember where each piece of key material came from, which is what the vaultExplicit and ageExplicit fields are for. A recipient from the config file’s age section counts as explicit: committing a recipient set once is the intended way for a team to work, and it is as deliberate as typing the flag.
RekeyTarget refuses the same collision for the same reason – --new-vault-password-file and --age-recipient each name the key to rekey onto.
Why nothing is discovered
Key material comes from flags, the config file’s age section, CARGOSHIP_AGE_IDENTITY_FILE, and CARGOSHIP_AGE_RECIPIENTS. Nothing else. Cargoship does not read ~/.config/sops/age/keys.txt, does not consult an agent, and does not reach into ~/.ssh for a key it was not pointed at.
The one prompt in the whole path is for the passphrase of an SSH key the operator named, and the one derived path is the .pub file beside such a key; see below for why neither is an exception to this rule.
This matches ResolveVaultPassword, which has never guessed or prompted, and the reason is the same: an apply that succeeds on one operator’s machine because of a file the configuration never mentions is an apply nobody can reason about, and the failure mode shows up on someone else’s machine, mid-cluster-change.
The environment variables differ in shape for a concrete reason. CARGOSHIP_AGE_RECIPIENTS is a whitespace-separated list because a public key contains no whitespace, so the split is unambiguous. CARGOSHIP_AGE_IDENTITY_FILE holds a single path, because a list of paths needs a separator and every separator worth choosing is legal in a file name.
A recipients file that parses to no keys is an error rather than nothing to do. Carrying on would leave a keyring with a vault password and no recipients, and EncryptFormat would then quietly choose Ansible Vault – writing the secret under a key the operator did not ask for, which is precisely the outcome rule 2 above exists to prevent.
Scope deliberately left out
- age passphrases (scrypt). It would duplicate what Ansible Vault already provides here – a shared secret – and age enforces scrypt-as-sole-recipient with a random label, which complicates the recipient path for no gain.
- Plugins. These are a property of the
agebinary. Cargoship links the library so that encrypting and decrypting stays a pure in-process operation in a single static binary, the same reason recorded in choice-vault-library. ssh-agent.agesshneeds the private key itself rather than a signing oracle, so an agent-held key cannot be used even though a passphrase-protected one on disk can.
ParseRecipients may return a *HybridRecipient (post-quantum X-Wing). Passing it through costs nothing, so it works without any code of ours.
SSH keys, and the parsing they force
filippo.io/age/agessh supplies age.Recipient and age.Identity implementations for ssh-ed25519 and ssh-rsa, so an SSH key is invisible above clustercfg: nothing about the ciphertext, the header dispatch, or the skip reporting changes. What it widens is who can use age at all, since operators and CI already hold these keys and already maintain authorized_keys files.
They go in the existing --age-recipient, --age-recipients-file and --age-identity-file rather than in parallel --ssh-* flags. That is what the age CLI does with -r/-R/-i, it keeps nine commands from growing three flags each, and it means an operator does not have to work out which kind of key they hold before choosing a flag.
The cost of that decision is that cargoship has to do its own parsing, in agessh.go:
age.ParseRecipientsfails the whole file on one line it does not recognise, and an SSH line is one. So recipients are scanned line by line here and dispatched on theage1prefix, which keeps a mixed file working. A line that parses as neither kind still fails the file, naming the line: skipping it would encrypt to fewer recipients than the operator listed, and the person who discovers that is the one who cannot decrypt.- Identities cannot be dispatched line by line at all, because an SSH private key is a multi-line PEM block. The file is read whole and routed on a
-----BEGINheader. parseRecipientrefuses anything beginningAGE-SECRET-KEY-or-----BEGINbefore handing it toagessh, becauseagessh.ParseRecipientquotes its argument back in its error where age’s own parser deliberately does not. An identity file passed as a recipients file is an ordinary mistake, and a private key in a log has to be treated as compromised everywhere it was used.
Why the SSH passphrase prompt lives in cmd
A passphrase-protected SSH key becomes an agessh.EncryptedSSHIdentity, which asks for the passphrase only once a stanza matches its public key and caches the decrypted key afterwards. That laziness is worth preserving: an operator holding a key nothing was encrypted to is never asked, and one whose key does match is asked once rather than once per credential. On an apply the ask therefore lands in the VerifyRegistryAuth preflight, before any host is touched.
clustercfg does no terminal I/O, so KeyOptions carries a PassphraseFunc and the commands supply it. Two things fall out of that shape:
- A nil callback is an error, not a prompt. A caller with nowhere to ask fails by construction rather than by luck, which is what CI wants.
- The prompt refuses a non-terminal rather than reading stdin. Reading the passphrase from a pipe would consume input the command wanted for something else and would hang when there is none. Failing with a message naming the key is something a scripted run can act on.
An older PEM key carries no public key beside its encrypted private key, so ssh.PassphraseMissingError.PublicKey is nil and the key has to come from the .pub file. That is the one path cargoship derives rather than being given, and it is derived from a path the operator named explicitly, read only when that key needs a passphrase – which is why it does not contradict “nothing is discovered” above. The age CLI does the same.
AgeRecipientsIn refuses an SSH private key rather than deriving a recipient from it: agessh’s recipient types have no text encoding to print, and ssh-keygen already wrote the public key into the file beside the key. Pointing at that is a better answer than any this could compute.
Generating keys, and the refusals around it
cargoship vault keygen exists because everything else here otherwise assumes the age distribution is installed, when cargoship already links the library that generates a key pair. It writes byte-for-byte what age-keygen writes, deliberately: a file only cargoship could read would strand an operator who later wants age --decrypt, and interoperability is the reason for supporting the format at all.
Three refusals are load-bearing, and none of them should be softened into a prompt or an overwrite:
--outputnever overwrites. Replacing an identity file destroys the only copy of a key, and every value in a committed configuration encrypted to it becomes unreadable by anyone, permanently.O_CREATE|O_EXCLmakes the check part of the create, so there is no window between asking and writing. A--forceflag here would be a flag whose only use is the unrecoverable case.--outputis refused with--public-key. It keeps one meaning – the file holding a private key that was just generated – and so keeps the no-overwrite rule from having an exception attached to it. A public key needs no such care and redirects perfectly well.- An identity that is not X25519 is an error in
AgeRecipientsIn, not a skipped line. The question being asked is “who can read values encrypted to this key,” and a short answer to that is worse than no answer, because the operator acts on it.
The generated identity never passes through logger. Logging is wired to stderr and may be scraped centrally, and a private key that reaches a log aggregator has to be treated as compromised everywhere it was used. Without --output the key goes to stdout so that cargoship vault keygen > key.txt works; the terminal warning covers the case where there is no redirect and the key lands in scrollback.
Dependency
filippo.io/age v1.3.2, BSD-3-Clause, which brings filippo.io/hpke v0.4.0 with it. Everything else it needs – chacha20poly1305, curve25519, hkdf, scrypt – was already vendored under vendor/golang.org/x/crypto/.
agessh is a package of that same module, so SSH support added only filippo.io/edwards25519 v1.2.0, which it uses to convert an Ed25519 key to X25519. golang.org/x/crypto/ssh was already vendored, and moved from an indirect dependency to a direct one because clustercfg now imports it for ssh.PublicKey and ssh.PassphraseMissingError.
ATTRIBUTION.md covers derived code (k0sctl, zarf, bootloose) rather than dependencies, so it needs no entry; the vendored LICENSE is sufficient.
Armoring, and the one thing it costs
age values are armored because they live in YAML scalars and the binary format does not. Two consequences are worth knowing:
- The age writer must be closed before the armor writer, and both must be closed. The first encrypts and flushes the final chunk; the second writes the footer that terminates the block. Closing only the outer one produces a truncated value that looks fine until something reads it.
- An armored value starts with
-----, so passing one as a positional argument tocargoship vault decryptis parsed as a flag. Nothing can be done about that inside the command, since the flag parser runs first, so stdin is the documented path and--after all flags is the alternative.02_vault_encrypt_test.goand08_age_test.goboth exercise the working forms.
Why the Ansible module is the cargoship binary, run on the management node
Cargoship’s cluster lifecycle commands – apply, prepare, reset, engine-config-sync, kube-config – should be drivable from an Ansible playbook, not only from cargoship’s own CLI. Three architectures were considered: the compiled Go binary as the module file itself, run on one node; the same binary copied to every managed node with Ansible sequencing the phases; and a conventional Python AnsibleModule wrapping cargoship as a subprocess. The first was chosen. It preserves how cargoship is operated and shipped today, and the node it runs on is the management node that air-gapped environments already stage onto.
Cargoship runs on one node, and that node is the management node
prepare, apply, and reset are already multi-host SSH orchestrators. They open their own connections to every host in the inventory and run the phase pipeline – connect, detect, validate, prepare, upload, bootstrap, kubeconfig, cleanup – across the fleet from a single process. Nothing puts a cargoship binary on a managed host: what lands there is engine artifacts and configuration. The engine binaries and RPMs (59_bin_install.go, 57_rpm_install.go), image tarballs (50_uploadfiles.go), the upload manifest under /var/lib/cargoship (03_manifest.go), /etc/sysctl.d/99-cargoship.conf, the fapolicyd rules file, and a lock file. All of those are files cargoship writes over SSH, not programs it installs.
The lock phase is the one piece that reads like an on-host agent and is not. Its explanation says it “runs a background task that will touch a file every 30 seconds on each remote node”, but startTicker is a goroutine inside the cargoship process calling Configurer.Touch over the connection it already holds (91_lock.go). The host holds a file; the process holding the lock is on the management node.
This matches how air-gapped environments already work, and how Zarf works: artifacts are staged onto a management VM that has been through the airlock, and distribution to the environment happens from there. Cargoship needs the distro package, the images, and the engine artifacts local to wherever it runs, so that node is the management node whether or not Ansible is involved – which is also how cargoship is operated today, by running the binary there directly. The module executes on exactly one node: connection: local when the Ansible controller is the management node, delegate_to when they are separate.
The inventory comes from Ansible; the connections do not
The module is handed an Ansible inventory and derives the cluster from it, rather than being pointed at a hand-written ZarfCluster document. A team that runs Ansible already has one description of its fleet, and asking for a second one in cargoship’s vocabulary means two files that drift. Group membership is where that fleet already records what a host is for, so it is where the role comes from: by default the group named controller supplies the control-plane nodes and the group named worker supplies the rest, with a role_groups mapping for inventories that use other names. The defaults are cargoship’s own role constants rather than a guess at an Ansible convention, so a site that says k8s_masters says so once, in the mapping.
Nothing in Go parses an inventory file. Resolving an Ansible inventory means group membership, group_vars, host_vars, dynamic inventory plugins, and the precedence rules over all of them; reproducing those in Go would reproduce them wrongly, and the failure would be a silent difference between what the playbook believes and what cargoship installs. Ansible resolves the inventory and hands over the result already merged, as groups and hostvars. internal/ansibleinv does only the typed translation: roles from group membership, connection details from ansible_host, ansible_user, ansible_port, and ansible_ssh_private_key_file, and everything cargoship needs beyond those from variables under a cargoship_ prefix.
Three rules in that translation are worth recording because they are decisions rather than mechanism.
- A host in none of the mapped groups is left out, and a host in two is an error. A play’s inventory legitimately carries hosts that have nothing to do with the cluster, so absence is not a mistake. Two mapped roles for one host is a mistake with no defensible resolution, and picking one silently would install a cluster whose shape nobody asked for. Neither case has a precedence rule, which is consistent with how
validateand the keygen refusals behave elsewhere. - A
cargoship_variable cargoship does not read is an error. Ansible has no notion of a variable belonging to anyone, so a misspelledcargoship_profilwould otherwise sit inhostvarslooking set and do nothing. An unrecognisedansible_variable is ignored: there are hundreds and they are not ours. - Host order is load-bearing.
60_configure_engine.gomakes the first controller in the document the leader, so the generated document emits controllers first, each role’s hosts in inventory order. This is the kind of fact that looks cosmetic in a diff and is not.
The generated document is checked against the inventory schema before anything is done with it, so a translation bug surfaces as a schema error naming the field rather than as a phase failure ten minutes into an apply. It is also written to disk and inspectable: an operator debugging a wrong translation needs to see what cargoship was actually given, and cargoship inventory from-ansible reproduces the same translation outside a playbook run so the result can be diffed and fed to cargoship validate.
What does not change is who connects. Ansible supplies the inventory and nothing else – no ansible_connection to a managed node, no per-host task, no fact gathering on the fleet. Cargoship opens every SSH connection itself, from the management node, exactly as it does when run by hand. That is the whole point of the arrangement described above, and it is the constraint every part of this design is built around.
Why the binary, and not a Python wrapper around it
A conventional AnsibleModule in Python, subprocessing cargoship, is the idiomatic Ansible answer, and it was seriously considered. It was rejected on three grounds.
- It reverses a principle this project has already recorded.
choice-vault-library.mdrejectedgo-ansiblein part because “Cargoship needs neither Ansible nor Python installed on the machine runningapplyorvault encrypt”, explicitly “consistent with Cargoship shipping as a single static binary”. A Python module reintroduces exactly that dependency on exactly that machine. Whatever the merits elsewhere, reversing a stated principle needs a stronger reason than the one the wrapper offers. - The interpreter spread is a liability in the environments this targets, not a convenience. That a management VM has an interpreter is easy –
dnfis written in Python, RHEL 8 and newer shipplatform-python, Ubuntu Server shipspython3. But a Python module has to work across all of them at once, fromplatform-python3.6 on RHEL 8 to 3.12 and later on RHEL 10 and current Ubuntu, in estates that are mixed by nature because they are air-gapped and upgrade slowly. A static Go binary has no interpreter compatibility surface at all. Interpreter availability is what makes the wrapper possible; it is not what makes it better. - A second language costs more than it returns here. The collection has to be packaged, linted, and covered by CI, or it silently holds less rigour than the Go side – which is worse than not having it. What the wrapper actually buys, once examined, is
ansible-docand argument errors that name the offending parameter.
That last point needs one correction to a common argument in its favour. no_log is often cited as something only the Python path provides, for the vault password and age key parameters that apply accepts. It is not: task-level no_log: true in the playbook suppresses a task’s arguments and output regardless of module type. argument_spec’s per-parameter no_log is finer-grained, which is an ergonomics difference rather than a safety one.
A thin action plugin, which is narrower than the wrapper this rejected
That rejection was of a Python module that subprocesses cargoship, and it stands. A Python action plugin is added anyway, and the distinction is the whole of the reasoning.
The collection needs groups and hostvars to project into the module’s inventory argument, and those live in task_vars, which only controller-side code can read. Ansible offers no way to get them without controller-side code – an inventory plugin describes hosts to Ansible rather than the reverse, and a templated argument in the playbook pushes the projection onto every operator instead of doing it once.
An action plugin is materially narrower than the wrapper rejected above. It executes inside the ansible-playbook process, on the interpreter Ansible is already running on, so it adds no dependency Ansible has not already added to that machine. It is bounded by ansible-core’s own controller support matrix rather than the platform-python 3.6 through 3.12 spread that made the wrapper a liability, because a controller is one machine the operator chose, not an estate. And it changes nothing about running cargoship by hand: apply on a management node still needs no Python, which is the principle choice-vault-library.md actually recorded.
The cost that survives is real: a second language in the collection to lint and test. It is bounded by keeping the plugin thin. The plugin reads task_vars, projects groups and hostvars, and delegates. Every translation rule – role derivation, variable mapping, ordering, validation – is in Go, where it is unit-tested without Ansible. A plugin that grows a rule is a plugin that has drifted from this decision.
The revisit trigger is publication. If this collection is ever consumed outside the team, missing ansible-doc and an unhelpful failure on a mistyped parameter are the first two complaints it will receive. The work is not wasted if that happens: the wrapper would subprocess the same JSON entrypoint described below, so the Go side is the part that gets kept either way.
Why not the binary on every managed node
Copying the binary to each target and letting Ansible sequence one task per phase per host is usually pitched as buying native parallelism, retries, handlers, and per-host reporting. It was rejected, and the reason is stronger than the duplicate maintenance it would obviously create.
Cargoship’s phases are not independently runnable. Lock and Unlock are a single paired phase – NewApply builds one lockPhase and appends lockPhase.UnlockPhase() at the end – so the cluster lock is scoped to one Manager.Run. Connect and Disconnect bracket the pipeline, and the SSH sessions in between are reused by every phase that runs. Manager.Run’s deferred cleanup walks the phases that already ran and calls CleanUp on each when a later phase fails. Split those across separate Ansible tasks and the lock is never held across the work it protects, the cleanup-on-failure never fires, and every phase reconnects. Reassembling that in playbook form is not a port of the existing pipeline; it is a second implementation of it with different failure semantics.
The multi-arch problem is real too – Ansible has no arch-aware module selection, so this would need a custom action plugin to choose and copy the right binary per host – but it is the smaller objection. The pipeline is the objection.
One binary, not a second release artifact
The module needs a machine-readable way into pkg/action: arguments in as JSON, exactly one JSON object out, following the WANT_JSON convention. Two things make this cheap here that are usually expensive.
- Dry run already exists, and check mode maps onto it directly.
pkg/phase/05_manager.gocarriesManager.DryRunplus a three-way classifier,ClassifyDryRun, that decides per phase whether a dry run skips it, runs it as itself because it only reads (readOnly), or calls its own dry path (withDryRun). A phase that declares neither is skipped, so dry-run safety is a property a phase has to claim rather than one it is assumed to have._ansible_check_modebecomes--dry-run, not a new subsystem. One gap had to be closed for this:engine-config-syncwas the only lifecycle command that never registered the flag, though it embeds the field and passes it to the manager like the others. It registers it now, on the same terms asapplyandreset– a dry run needs no--confirm, because requiring confirmation to ask what would happen is what pushes someone into running the real thing to find out. - stdout is already clean. A module that prints anything but its one JSON object is a broken module, and in most Go CLIs separating the log stream from the result stream is the bulk of the work. Cargoship’s logging already goes to stderr: the zarf logger’s
DestinationDefaultisos.Stderr, and rig’s own chatter is bridged into that same logger byinternal/rigloggerrather than being written directly. Nothing on the install path writes to stdout.
That entrypoint lives in the existing binary rather than a new cmd/ target producing its own release artifact. Adding a second build to .goreleaser.yaml is a few lines, which is what makes it tempting, but the lines are not the cost. Every release artifact carries a cosign signature and an SBOM, and the nfpms builds produce GPG-signed rpm, deb, and apk packages. A second binary is a second thing to sign, publish, verify, and carry through an airlock, and it introduces a failure mode that is otherwise impossible: a module binary and a staged cargoship on the same node disagreeing about version. One binary means the thing answering Ansible is the thing doing the work, and a management node keeps both entry paths – through a playbook, or by running cargoship by hand as operators do today.
The single binary presents itself as several module files by dispatching on the argv[0] basename Ansible invokes it under: a basename of cargoship_ followed by the name of an action the binary answers as enters module mode, and anything else is the ordinary CLI. The name has to be a real action, not merely carry the prefix. The prefix alone is not rare enough to dispatch on, because the repository builds its own binary as cargoship_linux_amd64 and the whole end-to-end suite runs that file; dispatching on the prefix turned every one of those invocations into a JSON object nobody asked for. What that costs is a misspelled module file: cargoship_engine_confg_sync now runs the CLI and fails where it parses the arguments file as a command, rather than saying it is not a module. Ansible also copies a module to the machine that runs it and prefixes the copy AnsiballZ_, so that prefix comes off before the name is read, and the name the process actually sees is AnsiballZ_cargoship_apply. The module files are symlinks to the binary, created by the same nfpms packages that install it, so there is nothing extra to sign, publish, or carry through an airlock. An environment variable, CARGOSHIP_ANSIBLE_MODULE, provides the same selection for tests, which have no reason to create symlinks to check the contract.
A module does not reimplement the command it stands for. It renders its parameters as the argument vector an operator would have typed and runs that, so package loading, keyring resolution, timeout parsing, and phase construction stay on one path and the two entry points cannot drift apart. What that costs is one indirection: cmd imports internal/ansiblemod to reach the dispatch, so ansiblemod cannot import cmd back, and it spells the flag names out rather than referencing the constants. A test in cmd parses a fully populated argument vector against the real command, so a flag renamed in the CLI fails CI instead of failing a playbook against a live fleet.
The generated ZarfCluster is written before the command runs and is named in the result. A run that fails leaves it there, because a failure that reads like the wrong cluster is one an operator needs the document to diagnose and reproducing the translation afterwards has already lost the run that produced it. A run that succeeds removes it, unless the operator named the path themselves, in which case the file is theirs.
One correction to the reasoning above belongs here, because it is load-bearing for the module contract. “Nothing on the install path writes to stdout” is true of the install path and false of the init path: cmd/viper.go built a bootstrap logger with Destination: os.Stdout, and cmd/root.go reported a failed configuration load and two failed flag-completion registrations with fmt.Printf. All of those are plain bugs for any command that emits a document, Ansible or not, and all are fixed to stderr.
Relying on the fix alone would be relying on an audit staying true, so module mode also duplicates file descriptor 1 at entry, points os.Stdout at stderr, and writes its one JSON object to the duplicate. Anything added later that reaches for os.Stdout reaches stderr instead, which Ansible captures and shows on failure.
The guard covers everything after main begins, which is not everything. cmd builds the root command in a package-level variable, so the writes listed above happen during package initialisation, before main and therefore before any guard could be installed. Those are the audited ones, and they stay audited: the fix to stderr is what holds there, and a test drives the binary in module mode with a broken configuration file so that a regression fails CI rather than a playbook. The init path is also where the binary can exit before answering at all – a configuration file that cannot be parsed calls os.Exit(1) in init() – which Ansible sees as a module failure with the reason on stderr, the same failure the CLI gives for the same file.
One module per action, not one module with an action parameter
The actions do not share an option set. apply carries roughly a dozen flags – host file updates, firewall, fapolicyd, node labelling, unmanaged nodes, kubeconfig update and path, worker concurrency, values files, signature verification, age keys. kube-config carries three, and it does not even load a package: installKubeConfigOptions.run builds a phase.Manager directly instead of going through initManager, because there is no distro package to resolve values against. Collapsing that into one module with an action parameter means a single argument surface where most parameters are invalid for most values of action, and without argument_spec every one of those relationships is hand-checked. Separate modules keep each argument surface small enough to validate by hand.
The changed signal comes from the phases, and says how far it reaches
A module that reports changed: true on every successful run teaches operators to ignore the field, and a handler wired to it fires forever. Reporting it honestly means the phases saying what they did, which is what Manager.Run now collects.
The mechanism is the optional-interface idiom the manager already uses for readOnly and withDryRun: a phase may implement changedReporter, and Run asks it afterwards rather than before, so a phase answers from what it did. A run that failed halfway has still changed the fleet and is expected to say so. Manager.Result holds the per-phase record, and RunResult answers Changed, Outstanding, Complete, and Undeclared over it.
Three decisions are worth recording:
- Silence is not a claim of no change. A phase that does not implement the interface cannot make
changedtrue, and is not counted as having changed nothing either. The overall signal is reported aspartialand the silent phases are named incargoship.changedUndeclared. Reportingchanged: falsewhileKubeConfigandLabelNodeshave said nothing would be a claim cargoship cannot support; naming them lets an operator decide. The alternative considered was assuming no change from silence, which is how a correct-lookingchanged: falsebecomes a missed handler. - Phases filtered out by
ShouldRunare still asked.EngineConfigSyncHostswrites the files the engine re-reads without a restart whilePrepareis deciding which hosts have drifted (except under a dry run, which reports the write instead – see the last paragraph below), and then reports no hosts to sync. A run whose only change landed in chart manifests would otherwise read as a run in which nothing happened.RunResulttherefore records a third disposition,Skipped, besideRanandPlanned. - Under check mode,
changedreadsOutstandingrather thanChanged. A dry run changes nothing, so the useful question is whether a real run would, and a phase only reachesPlannedafter its ownShouldRunfound work against the live hosts. That matches what Ansible’s check mode means bychanged.
The result travels back on the context. The module mode runs cargoship by building an argument vector and calling the ordinary command through cmd.ExecuteArgs, which returns an error and nothing else. Threading a return value back through cobra, the action, and the manager would change five call sites to serve one caller, so phase.WithResultSink puts a sink on the context and Manager.Run publishes to it when one is there. A run that never reached its phases publishes nothing, which the module reports as unknown rather than as no change – the two are different, and conflating them is what makes a changed: false untrustworthy.
Writing the signal turned up one thing worth fixing on its own account: the in-place write in needsUpdate runs from Prepare, and Prepare runs for every phase including the ones a dry run is about to skip. A dry run was therefore writing files to hosts. It no longer does.
Five modules, and the parameters they deliberately do not offer
The binary answers as apply, prepare, engine_config_sync, reset, and kube_config. Those are the actions that converge a fleet, which is what a playbook is for. package create stays out for the reason at the end of this document.
Each module’s parameters follow its own command’s flags, and the surfaces genuinely differ – prepare reads no encrypted value and so takes no vault_password_file, reset installs nothing and so takes no package and no values, kube_config touches the management node rather than the fleet. A parameter offered to the wrong module is an error naming it, rather than a flag quietly dropped. What is shared is shared in code: one embedded common struct carries inventory, inventory_path, log_level, and log_format for every module, so one module cannot spell inventory_path differently from its neighbour.
Two things are deliberately left out.
Keyless signature verification. The modules expose public_key and verify and stop there, which the keyless removal has since made moot: a Cosign key pair is cargoship’s only signing method, so there is no keyless surface left for a module to decline to wrap. The reasoning is kept because it is the argument-surface-size reasoning from the previous section applied one level down, and because it is what should be weighed again if keyless verification returns through #848 – the flags were mutually exclusive in ways cobra enforced and a hand-written module would have had to restate.
A check-mode answer for kube_config. cargoship kube-config has no --dry-run. A module with nothing to answer with has two options: run anyway, or refuse. Running anyway would change the management node during a run that was asked only to describe itself, so the module reports skipped: true and runs nothing. That is what Ansible does with a module that has declared it does not support check mode; a binary module has nowhere to declare it, so it says so in its result instead.
The collection ships the links, and the role hardcodes what must not fail open
Ansible resolves a task name to a module file before any plugin of ours is consulted, so an action plugin alone reaches nothing. Each action therefore needs a file named cargoship_<action>, and each of those is a symlink to the binary. The RPM and deb packages create the links through nfpms contents: entries, installing the collection tree to /usr/share/ansible/collections, which is already on ansible-core’s default collections path; a collection installed from Galaxy or from a checkout has no packaging step to hang that on, so hack/ansible-link-modules.sh does it by hand. Both were chosen over generating a per-action shim file: a shim is a second thing to sign and a second thing that can disagree with the binary.
The action plugin forwards an allowlist of host variables rather than the whole of hostvars. A host’s resolved variables routinely hold credentials for things that are not this cluster, and a module’s parameters are written to a file on disk, so forwarding everything would write unrelated secrets to disk on every run. The allowlist is the four ansible_* connection variables plus every cargoship_* key – all of them, including misspelled ones, because rejecting an unknown parameter by name is ansibleinv’s job and it cannot do that without being shown the misspelling. TestActionPluginVariableAllowlist reads the Python and fails when it drifts from the Go constants, because the symptom of drift is an unset field rather than an error.
The plugin also merges a parameters dict into the arguments. The role has a caller-supplied dict of parameters and no way to know which of them the chosen module takes, and the obvious way to pass it – making the task’s whole argument dict one template – is warned about by Ansible on every run and is genuinely unsafe. Merging in the plugin keeps the task’s keys literal and its values templates, and costs one parameter that the module itself never sees.
The role sets run_once: true on every task. Cargoship converges the whole fleet in one run, so a play over the fleet’s own inventory would otherwise run a full convergence once per host. It defaults no_log: "{{ cargoship_no_log | bool }}" with cargoship_no_log: true by default: a template failure or misconfiguration avoids exposing decrypted credentials and host connection parameters in CI or production logs, while still allowing operators to set cargoship_no_log: false for local troubleshooting. The debug task showing cargoship’s own report carries no parameters.
The interface is documented in the plugins, and nothing validates against it
Each action plugin carries a DOCUMENTATION block naming every parameter the module takes, its type, and the flag it renders. generateModuleDocs in magefiles/pkg/gen/docs/ansible_module.go reads those blocks, generateRoleDocs reads each role’s meta/argument_specs.yml, and together they write docs/ansible/. The reference pages therefore cannot be edited into disagreement with the modules, and a parameter added to the Go struct without being documented shows up as a missing row rather than as nothing at all.
The blocks are inert. An action plugin’s DOCUMENTATION exists for ansible-doc, which is not run against this collection, so nothing in Ansible reads them and no rule moved into Python: there is still no argument_spec, and argument validation is still hand-written in Go. What keeps them honest is TestActionPluginDocsMatchModuleParams, which reads the Python and fails when a documented parameter has no Go field, when a Go field is undocumented, or when a documented cli_flag is not the flag the argument vector actually renders.
cli_flag: is ours rather than a standard Ansible documentation key. Stock validate-modules would flag it; it is not run either, and the parser that reads the key is the one in this repository.
This narrows the first bullet below rather than reversing it. The interface is now documented in a structured place and published, which is part of what publishing the collection was going to require of it anyway. Adding an argument_spec that Ansible itself validates against would be the reversal, and is still declined for the reasons in “Why the binary, and not a Python wrapper around it” above: what it buys is ansible-doc and better-named argument errors, and a plugin that grows a rule has drifted from this decision.
What is being accepted
- No
ansible-docand noansible-test sanity. The module’s interface is documented where we choose to document it – theDOCUMENTATIONblocks and the pages generated from them – and checked by a test of ours rather than by Ansible’s own tooling. - Argument validation is hand-written, so a mistyped parameter fails less helpfully than
argument_specwould manage. - Secret redaction relies on task-level
no_login the playbook rather than per-parameter declarations, which is coarser and has to be documented so operators actually set it. - Ansible sees one task per action, not per-host results. Per-host detail stays in cargoship’s log stream on stderr.
What this leaves open
Most phases still do not report whether they changed anything. Only the engine configuration sync phases declare changedReporter today, so an ordinary run reports its signal as partial. The two undeclared phases that can genuinely change something are KubeConfig, which writes the local kubeconfig, and LabelNodes, which sets node role labels; both are worth declaring next. The rest – Connect, DetectOS, GatherFacts, ValidateHosts, Lock, Disconnect – change nothing a playbook would want a handler for, and declaring them buys only a complete signal.
engine-config-sync came first, and the rest followed it. It is the most Ansible-shaped action cargoship has – converge configuration, report drift, restart services – and pkg/phase/70_engine_config_sync_common.go already computed the per-host idempotency signal in needsUpdate, driftedFiles, and driftReason. Proving the contract and the changed plumbing there first, before taking on apply’s argument surface, is what made the shared parameter and argument-vector machinery fall out rather than be designed up front.
Note that create is not in scope despite sometimes being listed alongside these commands. cargoship package create builds a package; it touches no hosts, runs no phases, and has no dry run. Whatever it needs from Ansible is a different question.
Why the API group moved, and why Zarf names remain afterwards
Cargoship’s own configuration documents used to declare apiVersion: zarf.dev/v1alpha1 with kind: ZarfCluster and kind: ZarfDistro, and the Go types behind them lived in api/zarf.dev/v1alpha1/. They now declare colonel-byte.dev/v1alpha1 with kind: Cluster and kind: Distro, from api/colonel-byte.dev/v1alpha1/.
The move itself is not the interesting part. What this document records is the part that looks like an unfinished job: after a rename whose entire point was to stop saying Zarf, the tree still says Zarf in several places, and each one is deliberate.
What forced the move
zarf.dev belongs to the upstream Zarf project, and upstream already defines kinds under it – ZarfPackageConfig and ZarfInitConfig in vendor/github.com/zarf-dev/zarf/src/api/v1alpha1/package.go. A custom API group is supposed to be an FQDN the authoring project controls, for exactly the reason that two projects defining kinds in one group have no way to tell their documents apart. Cargoship was squatting in a live namespace, and nothing but luck kept upstream from one day shipping a ZarfCluster of its own.
The second reason is a layer boundary. Cargoship manages the host, the OS, and the Kubernetes engine lifecycle; Zarf packages applications on top of the cluster that results. Naming cargoship’s root types ZarfCluster and ZarfDistro put the lower layer under the upper layer’s name.
What did not move, and why
Upstream types keep their names
ZarfComponentAction, ZarfComponentActionSet, ZarfComponentActionDefaults, ZarfComponentActionWait, ZarfPackage, ZarfChart, ZarfDataInjection and the rest are upstream’s types, referenced out of vendor/. Renaming them is not available – they are not ours – and aliasing them locally would buy a cosmetic win at the cost of a second name for one type.
So the distro schema still has Zarf-named $defs
This is the visible consequence, and the one most likely to be mistaken for a missed spot. schema/distro-v1alpha1-schema.json contains:
"$defs": {
"ZarfComponentAction": { ... },
"ZarfComponentActionSet": { ... },
"ZarfComponentActionWaitCluster": { ... }
}
$defs keys are bare Go type names, and Actions.OnCreate is a zarf.ZarfComponentActionSet by deliberate choice – see choice-v1alpha1-action-conversion, which explains why that field is typed on upstream’s model and why retyping it would be a breaking schema change for every distro.yaml an operator has written. Until that field changes type, those keys stay. Renaming the API group was never going to remove them.
The deprecated group is still accepted
v1alpha1.DeprecatedAPIVersion is still a constant, still an entry in internal/cfg’s knownAPIVersions, still a key in pkg/schema’s documentKinds, and still an enum= on both roots. That is the transition window, not residue. Deleting any one of them breaks documents that parse today.
Two details worth knowing before touching them:
- The dispatch table is priority-ordered.
colonel-byte.dev/v1alpha1sits at priority 2 andzarf.dev/v1alpha1at priority 1, soParseMultiDocon a package carrying one document per group reads the newer one.TestHandlerForfails if two entries ever share a priority, because “latest” would stop being well defined. - The combinations are accepted independently. A document may pair either group with either kind. Nothing cross-checks them, and
TestValidatorAcceptsTheDeprecatedSpellingpins that on purpose: an operator half-way through editing a file should not be told the two halves disagree.
What the rename did change, that a reader might not expect
$defs keys and $ref targets moved. They are bare Go type names, so dropping the Zarf prefix from the Go types rewrote both published schemas – #/$defs/ZarfHost became #/$defs/Host – without any document format changing. A rename of an unexported-looking Go type is a change to a published artefact here.
One generator reads a $defs key by name. magefiles/pkg/gen/schema/schema.go looks up defObj["Host"] to stamp required: ["role"] onto the host subschema. It is an ok-checked assertion, so if the key stops matching the type name it skips silently and the constraint disappears from the published schema with no error anywhere. Any future rename of cluster.Host has to change that string in the same commit.
The rename went through gopls, not sed. ZarfFile was declared by both this project and upstream, so the selector v1alpha1.ZarfFile meant different types in different files depending on which v1alpha1 that file imported. Every non-vendor use happened to resolve to ours, so a textual rename would have survived – by luck. A textual rename that retargets an upstream reference produces no compile error, because both types exist. Renaming ours to File removed the ambiguity for good, but the same hazard still applies to the near-miss pairs BuildData/ZarfBuildData and Metadata/ZarfMetadata.
Not done, on purpose
Annotation prefixes were not touched. The original proposal called for migrating zarf.dev/<key> annotations to colonel-byte.dev/<key>. There are none to migrate. Every zarf.dev/ label and annotation in the tree is upstream’s – zarf.dev/agent, zarf.dev/package, zarf.dev/connect-name, zarf.dev/namespace-override – and all of them live in vendor/. Renaming them would break the contract with the Zarf agent rather than assert ownership of anything.
The version stayed at v1alpha1. The schema is still in flux, and a group move is not a reason to spend the version bump.
Schema hosting did not move. The $schema URLs still point at raw.githubusercontent.com. Only the filenames changed, so the 165 generated editor headers kept resolving through the whole migration.
Why Binary-Artifacts and two Pinned-Dependencies findings stay annotated, not fixed
OpenSSF Scorecard’s Binary-Artifacts check scores 9/10 for one finding:
Warn: binary detected: vendor/github.com/open-policy-agent/opa/internal/compiler/wasm/opa/opa.wasm
This document records why that file is there, why it can’t be removed, and why the fix is an annotation rather than a change to vendor/.
The file is not optional
opa.wasm sits next to vendor/github.com/open-policy-agent/opa/internal/compiler/wasm/opa/opa.go, which does:
//go:embed opa.wasm
//go:embed callgraph.csv
go:embed compiles the file into the Go binary. It isn’t test fixture data or an example asset sitting near source by coincidence - the package does not build without it. There is no go mod vendor flag, build tag, or vendor-pruning step that drops an embedded file out of a package that’s actually imported; vendoring copies whatever the package needs to compile.
OPA is a real, reachable dependency
go.mod lists github.com/open-policy-agent/opa v1.20.2 // indirect. go mod why github.com/open-policy-agent/opa/v1/rego traces the chain:
cmd -> zarf/src/pkg/signing -> cosign/cmd/cosign/cli/verify -> cosign/pkg/cosign/rego -> opa/v1/rego
opa/v1/rego is what pulls in opa/internal/compiler/wasm/opa, which embeds the wasm file. This is cargoship’s signature-verification path, not incidental transitive weight - dropping OPA would mean dropping cosign policy verification.
Un-vendoring was already rejected, for the same reason it’s rejected here
choice-osv-vendor-overrides covers the analogous situation for the Vulnerabilities check and reaches the same conclusion for the same reason: the repository is vendored so it builds on the air-gapped management node cargoship runs from, with no route to a module proxy, so vendor/ has to hold everything the build needs, including files that cost Scorecard points on their own. Un-vendoring opa/v1/rego to dodge one file isn’t a candidate any more here than it was there.
Patching the vendored tree by hand to strip the file was also considered and rejected: go mod vendor recreates vendor/ from the module cache on every dependency bump (including Dependabot’s own re-vendoring), so a hand-edit doesn’t survive the next update and there’s no generator target analogous to Dev.Vendor’s osv-scanner overrides to keep re-applying it - the file’s presence is a direct, structural consequence of embedding, not a stray artifact a script can regenerate around.
The fix is an annotation, not a vendor change
Scorecard v5 added maintainer annotations: a scorecard.yml (or .scorecard.yml / .github/scorecard.yml) at the repo root that attaches one of five predefined reasons - test-data, remediated, not-applicable, not-supported, not-detected - to a check. It is informational only. It does not change the numeric score; it surfaces context in Scorecard’s output (and suppresses the matching code-scanning alert) when read with --show-annotations.
.github/scorecard.yml in this repo annotates binary-artifacts with remediated: “a binary is needed but it is signed or has provenance,” which is the documented example that matches this case - the module (wasm file included) is checksum-pinned by go.sum, so its provenance is verified on every build.
What this does not do
It does not raise the check to 10. Binary-Artifacts stays at 9/10, same as choice-osv-vendor-overrides’s Vulnerabilities ceiling stays below 10 for its own dependency-chain reasons. The annotation exists so a human reading the Scorecard result sees why the finding is there and that it was investigated, not to move the number.
Pinned-Dependencies: two more vendored findings, same shape
Pinned-Dependencies scored 9/10 for three findings, not one. Two are the same story as opa.wasm:
goCommand not pinned by hash: vendor/github.com/ChrisTrenkamp/goxpath/coverage.sh:3
goCommand not pinned by hash: vendor/github.com/json-iterator/go/build.sh:10
Both are go mod vendor copying a dependency’s own repository layout, this time its maintainer-facing shell scripts rather than an embedded asset: coverage.sh runs goxpath’s coverage report and opens it in Firefox; build.sh sets up a GOPATH-style tree and runs dep ensure for json-iterator’s old (dep, not modules) vendoring flow. Neither script is on any path cargoship’s build or CI invokes - they exist only because vendoring copies whole package directories, non-Go files included. .github/scorecard.yml annotates pinned-dependencies with not-applicable for the same reason opa.wasm got remediated: nothing here is a candidate for un-vendoring or hand-patching, for the reasons above.
The third finding was different in kind, so it got fixed instead of annotated:
npmCommand not pinned by hash: .github/workflows/commitlint.yaml:35
This one was cargoship’s own workflow, not vendored third-party code, so it was addressed directly rather than annotated. npm install --save-dev @commitlint/{config-conventional,cli} has no hash-pinned form - npm install never does - but npm ci against a committed lockfile counts as pinned (Scorecard’s own isNpmUnpinnedDownload treats ci as verifying all hashes). .github/commitlint/ now holds a small package.json and package-lock.json pinning @commitlint/cli and @commitlint/config-conventional, and the workflow runs npm ci there instead of npm install. .commitlintrc.yaml moved into that same directory (from the repo root) because commitlint resolves an extends entry’s package relative to the config file’s own location, not the working directory it was invoked from - leaving the config at the repository root would have kept failing to resolve @commitlint/config-conventional even with node_modules installed one directory down. Nothing else in the repository referenced the old root-level path. .github/dependabot.yaml gained a matching npm ecosystem entry for /.github/commitlint, since without one nothing would keep the newly pinned versions current, and this manifest is real (installed and run in CI, unlike the vendored ones above), so it’s also a legitimate target for the Vulnerabilities check’s osv-scanner pass rather than something to annotate away.
Why pkg/coci is a fork of zarf’s zoci, and why pkg/oci is not
Cargoship publishes and pulls distro packages over OCI. The client underneath that is zarf’s - pkg/coci is a derived copy of github.com/zarf-dev/zarf/src/pkg/zoci, with the attribution headers on every file - and the package it sits on, github.com/zarf-dev/zarf/src/pkg/oci, is imported rather than copied. Those two different answers for two adjacent packages are the thing this document explains, because from inside pkg/coci the duplication looks like something to clean up and there is nothing in the code to say what cleaning it up would cost.
src/pkg/zoci cannot be imported at all
This is the load-bearing fact, and it is invisible from here.
src/pkg/zoci/fetch.go imports github.com/zarf-dev/zarf/src/internal/pkgcfg, for one call to pkgcfg.ParseMultiDoc. Go’s internal-package rule means any module outside github.com/zarf-dev/zarf/src/... fails to compile if it imports src/pkg/zoci, whichever symbol it wanted. Not “links something unwanted” - fails to build. A package under src/pkg/ that cannot be imported is effectively internal, and neither its path nor its doc comment says so.
So for zoci the choice was never reuse-or-fork. It was fork or reimplement, and a derived copy that keeps upstream’s attribution and tracks upstream’s logic is the better of those two.
What linking it would cost even if it compiled
Worth recording, because “they fixed the internal import” is going to look like the moment to delete this package, and it is not sufficient on its own. zoci reaches the following through three of its imports:
| Via | Pulls |
|---|---|
src/pkg/images | docker/cli/cli/command, docker/cli/cli/flags, moby/moby/client, go-containerregistry (crane, daemon, layout) |
src/pkg/packager/layout | sigstore/cosign/v3/pkg/cosign |
src/pkg/utils | k8s.io/api, k8s.io/apimachinery, sigs.k8s.io/yaml, go-containerregistry, cosign/pkg/oci/remote |
That is the Docker CLI, the Moby client, cosign and the Kubernetes API machinery, none of it reachable from an air-gapped management node, arriving to support a package format cargoship does not write. Against the binary budget in AGENTS.md that is disqualifying on its own, independently of the internal import. No MiB figure is quoted here on purpose: the import closure is the argument, and a number measured against a build that cannot be produced would be fiction.
src/pkg/images is the sharpest case, and it is the one worth asking upstream about. zoci needs exactly two things from it - IsIndex and IsManifest, six-line media-type switches with no dependencies - and gets docker/cli and moby/moby/client with them, because Go links whole packages and src/pkg/images/pull.go imports those.
src/pkg/oci is clean, and is imported
The package underneath zoci is the opposite case. Its entire import closure is oras.land/oras-go/v2, github.com/opencontainers/image-spec, zarf/src/pkg/helpers and the standard library - every one of which cargoship already linked. It carries no custom media types, no zarf annotation keys, no zarf.yaml, and no zarf struct types in any signature; it is a proper functional-options package with Modifier and six With* options covering transport, TLS, plain HTTP, user agent, logger and cache.
So it is imported. Cargoship was on github.com/defenseunicorns/pkg/oci until zarf dropped that dependency and brought the same code in-tree as src/pkg/oci at v0.88.0; staying behind would have left cargoship the sole significant consumer of a library its only other consumer had left. The swap was a one-line import change in ten files - every symbol and method cargoship uses is signature-identical, and setRepository differs by one slog key - and it dropped both defenseunicorns/pkg/oci and the indirect defenseunicorns/pkg/helpers/v2 from go.mod, 2503 vendored lines.
It was not a binary win and should not be reported as one: 39.46 MiB to 39.44 MiB with mage build:binary, both built in the same session. Nothing new is linked and nothing stopped being linked, because the two packages are the same code. The win is one fewer module to track and one fewer orphaned dependency; see the warning about vendor size in AGENTS.md.
Two things in src/pkg/oci are inherited oddities rather than decisions, and both are written up in docs/upstream/: setRepository rewrites the registry names 🦄 and defenseunicorns to ghcr.io/defenseunicorns/packages/..., and credentials are read from the Docker config file with no injection point. Both are fixed on zarf’s main - the shorthand was deleted in #5504 and WithCredentialFunc added in #5502, both 2026-10-21 - but neither is in v0.88.0, so the vendored copy here still has both.
Cargoship tracks zarf’s releases rather than main, so the following is a note for whenever the next bump happens rather than a reason to make one. Two further things in src/pkg/oci moved after v0.88.0. oci.WithCache now takes content.Storage rather than *orasOCI.Store, which GetOCICacheModifier still satisfies, so it is source-compatible for us. The change it came with is the one to read: zarf stopped building the image-layer cache as an oci.Store and now uses oci.NewStorage, because a Store maintains an index and a concurrent oras.Copy writing through it races on that index. Cargoship’s pkg/images/pull.go still constructs the cache the old way and sets copyOpts.Concurrency, so the same exposure is here; oci.NewStorage is already in the vendored oras-go, so taking the fix does not wait on the bump.
What diverges on purpose, and must survive a resync
Everything else in pkg/coci should stay recognisably zarf’s. These four are cargoship’s and are not upstream bugs:
updateIndexForArches(pkg/coci/push.go) replaces theOrasRemote.UpdateIndexit shadows. A cargoship package is fat - one set of blobs, therefore one manifest, valid for every architecture it targets - so the index lists that one manifest once per architecture. Upstream’sUpdateIndexWithDescriptortakes a single platform and would need one call, one index fetch and one index push per architecture. Requested upstream indocs/upstream/zarf-oci-index-multi-arch.md.fetchIndex’s tolerance of a non-index tag (pkg/coci/push.go) returns an empty index for a tag that resolves to a plain manifest, which is what a cargoship old enough to have tagged one directly left behind. Upstream errors instead, so publishing over such a tag would fail rather than migrate it. Same request as above.SoleManifestDigest(pkg/coci/fetch.go) falls back to the one manifest in an index when the host’s architecture matches none of them.layersFromIndexChildrenandlayersFromManifestChildren(pkg/coci/pull.go) walk a multi-arch index when collecting image layers to publish. Seechoice-image-index.mdfor why a package stores several architectures under one index.
The cargoship-specific names are deliberate too: CargoshipConfigMediaType, CargoshipLayerMediaTypeBlob, config.DistroYAML in PackageAlwaysPull, and the layers.LayerType set, which describes cargoship’s layout rather than zarf’s components-and-sboms one.
Do not revert this to an upstream import
There is no import to revert to. src/pkg/zoci does not compile from here, and the fix for that upstream would not be enough by itself - see the table above. If the internal import is ever resolved, the question to ask is still whether linking src/pkg/images, src/pkg/packager/layout and src/pkg/utils is acceptable, and the answer is still no while those pull the Docker CLI and the Kubernetes API machinery.
When resyncing pkg/coci with a newer zoci, re-apply exactly the four divergences listed above and take upstream’s wording for everything else. The upstream asks that would narrow this fork are drafted in docs/upstream/ and have not been filed.
Known drift inside this package
Recorded here rather than fixed, so it is not rediscovered as a surprise:
AssembleLayers’sincludeparameter does nothing.pkg/coci/pull.go:97-98assignsinclude = layers.GetAllLayerTypes()and never reads it, behind a//nolint:ineffassign,staticcheckthat silences the linter which would have caught it. The doc comment above it promises filtering that does not happen, soMetadataLayers,ImageLayers,OSLayersandFileLayersare decorative. Both call sites passGetAllLayerTypes(), so no caller is wrong today.- Three constants are defined twice.
CargoshipConfigMediaTypeandCargoshipLayerMediaTypeBlob(pkg/coci/common.go) andOCITimestampFormat(pkg/coci/push.go) duplicate identical values inpkg/packager/layout/oci.go:51-56. The writer and the reader of the same format can drift apart silently. - Stale zarf wording. Comments in
pkg/coci/layers/common.gostill say “Zarf cache” and “zarf.yaml” for constants describing cargoship’s layout. - Registry-backed paths have no unit coverage.
PullDistro,PushPackageand(*Remote).AssembleLayersare the standing Tier 3 list intodo-pkg-coci-registry-tests.md.
Why crypto11 is ignored by Dependabot rather than pointed at its new home
github.com/ThalesIgnite/crypto11 is pinned at v1.2.5 and will stay there. The upstream repository has been renamed twice and every tag after v1.2.5 declares a different module path, so Dependabot cannot update it and fails its entire daily run trying. .github/dependabot.yaml ignores the dependency outright. This document records why the obvious fix - a replace directive pointing at the renamed module - was built, tested, and rejected anyway, and why the thing this dependency looks like it affects is not actually in the shipped binary.
What forced it
The Dependabot run of 2026-10-02 created six update pull requests and then exited 1 on a single dependency:
| github.com/ThalesIgnite/crypto11 | go_module_path_mismatch | {
| | | "declared-path": "downloading",
| | | "discovered-path": "github.com/eclipse-keypont/crypto11",
| | | "go-mod": "go.mod"
github.com/ThalesIgnite/crypto11 301-redirects to github.com/eclipse-keypont/crypto11, so the module proxy still answers for the old path and still serves the full tag list. What it cannot do is serve a tag whose go.mod agrees with the path it was requested under:
| Tag | module line in its go.mod |
|---|---|
| v1.2.5 | github.com/ThalesIgnite/crypto11 |
| v1.3.0 | github.com/ThalesGroup/crypto11 |
| v1.6.8 (latest) | github.com/eclipse-keypont/crypto11 |
v1.2.5 is the last tag published under the original path. Requiring anything newer under that name produces the error Dependabot reported, which reproduces in four lines in a throwaway module:
module declares its path as: github.com/eclipse-keypont/crypto11
but was required as: github.com/ThalesIgnite/crypto11
Nothing in this repository requires crypto11 directly. go mod why -m traces it to the signing path:
cmd -> zarf/src/pkg/signing -> cosign/cmd/cosign/cli/signcommon -> cosign/pkg/cosign/pkcs11key -> ThalesIgnite/crypto11
sigstore/cosign/v3 v3.1.3 is the newest release and its go.mod still reads github.com/ThalesIgnite/crypto11 v1.2.5, so there is no cosign version to move to that resolves this. The fix has to come from upstream re-importing the renamed module.
The dependency is not in the shipped binary
This is the fact that decides how much any of it matters, and it is not visible from go.mod, where crypto11 looks like an ordinary indirect dependency.
crypto11’s only consumer, cosign/pkg/cosign/pkcs11key, is split across two files by build tag. pkcs11key.go carries //go:build pkcs11key and imports crypto11; disabled.go carries //go:build !pkcs11key and defines the same API as stubs that return errors.New("unimplemented"). Nothing in .github/, magefiles/, or the goreleaser config sets that tag, so every build of cargoship - CI, release, and the e2e suite alike - compiles the stubs:
| Build | crypto11 packages in the graph |
|---|---|
go list -deps ./... | 0 |
with -tags pkcs11key | 1 |
The release build cannot opt in even by accident. .goreleaser.yaml sets CGO_ENABLED=0, and crypto11 reaches the PKCS#11 library through github.com/miekg/pkcs11, which is import "C". Setting the tag under the release environment does not produce a binary with hardware signing in it, it produces a compile error:
$ CGO_ENABLED=0 go build -tags pkcs11key ./...
# github.com/ThalesIgnite/crypto11
vendor/github.com/ThalesIgnite/crypto11/crypto11.go:174:14: undefined: pkcs11.Ctx
So crypto11 is a go.mod, go.sum, and vendor/ artifact and nothing more: no tag enables it, and the release settings could not build it if one did. PKCS#11 hardware-token signing is the feature it backs, and that feature is compiled out. A stale pin here cannot reach a user, and govulncheck -mode=source -tags pkcs11key ./... reports no vulnerabilities either way, so this is Dependabot hygiene and not a security matter.
The corollary for anyone verifying a change to this dependency: go build -tags pkcs11key ./... only compiles it on a host where cgo is available and enabled, which is the default on Linux with a C toolchain but is not what CI or the release build does.
Note that the e2e suite does sign and verify packages for real - TestCargoshipSign, TestCargoshipSignedRoundTrip, and TestCargoshipInstallVerify cover 19 subtests against ephemeral cosign key pairs from cosign.GenerateKeyPair. That coverage is all file-based keys (--signing-key <path>), which never touch crypto11. Good signing coverage is not coverage of this dependency.
The replace directive works, and still does not help
The natural fix is to send the old path at the maintained fork:
go mod edit -replace github.com/ThalesIgnite/crypto11=github.com/eclipse-keypont/crypto11@v1.6.8
This was done in full, not sketched. It resolves, and v1.6.8’s API satisfies cosign’s v1.2.5-era call sites with no shimming: go mod tidy, go mod vendor, go build ./..., go build -tags pkcs11key ./..., go vet, go test -short, Dev.VerifyVendor, pre-commit run --all-files, and Test.EndToEndNonCluster all passed on it. The approach is sound and it does silence the failure.
It was rejected because it buys nothing over the ignore entry. Dependabot does not update replaced dependencies, by explicit design. go_modules/lib/dependabot/go_modules/file_parser.rb filters the dependency set before anything else runs, and skip_dependency? opens with:
# Updating replaced dependencies is not supported
return true if dependency_is_replaced(dep)
The match test treats a directive whose left side carries no version as unconditional, which is this one, so crypto11 is dropped at parse time. Upstream’s comment gives the reason: it “prevents that we change dependency versions without any impact since the actual version that is being imported is defined by the replace directive.” The replacement target does not get picked up instead - the parser only walks require entries, and a replace target is not one. Both names end up invisible.
That leaves the two options doing the same thing by different routes:
| Approach | Dependabot tracks crypto11? | Run stops failing? | Vendor churn |
|---|---|---|---|
ignore entry | No, explicitly | Yes | None |
replace | No, silently | Yes | ~800 lines |
Neither restores automated updates, so the choice is only about which pin goes stale and how loudly. The ignore entry states its own reasoning in a config file someone editing Dependabot behaviour will read. The replace skip is invisible unless you already know the parser behaviour above, and it adds a hand-maintained version pin to go.mod plus a vendored tree jumping four minor versions, for a package that is compiled out. Given that, the smaller and more legible change won.
The replace remains the right move the moment any of this stops being true - if cosign starts calling v1.6.x API, or if something ever sets -tags pkcs11key.
The rest of the log is noise, and chasing it is wasted work
The failing run logged 40 Error while fetching release date info lines at INFO level. Exactly one of the 24,132 log lines was fatal - the go_module_path_mismatch above - and the rest are not this repository’s problem:
| Log entry | Count | Cause |
|---|---|---|
invalid version: unknown revision 0.0.0-... | 39 | Dependabot’s cooldown filter drops the v prefix and queries @0.0.0-..., which can never resolve |
cuelabs.dev/go/oci/ociregistry: ... 403 Forbidden | 1 | That host refuses Dependabot’s egress proxy on ?go-get=1 |
The 39 hit every pseudo-versioned module in the tree - genproto, k8s.io/utils, x/telemetry, digitorus/pkcs7, masterzen/simplexml, and the rest. The only consequence is that cooldown cannot be applied to those modules; Dependabot logs Filtered out 0 version(s) and carries on. Neither row is actionable here, and neither has anything to do with the failure.
When this entry can go away
Delete the ignore entry when sigstore/cosign/v3’s go.mod requires github.com/eclipse-keypont/crypto11 (or whatever path the project is on by then) instead of github.com/ThalesIgnite/crypto11. At that point the old path leaves the module graph on its own and the entry becomes a no-op. Until then, a Dependabot run that does not mention crypto11 is the entry working, not the problem having resolved itself.
Why Dependabot’s pull request titles are exempt from header-max-length
PR Title Check lints a pull request’s title with commitlint, because release-please squash-merges and the title becomes the final commit message. @commitlint/config-conventional caps a header at 100 characters, and ten of the modules in go.mod cannot produce a build(deps): bump <module> from X to Y title that fits. Dependabot has no way to shorten a module path, so every bump of one of those ten failed a required status check until somebody edited the title by hand. .github/commitlint/.commitlintrc.dependabot.yaml turns that one rule off for Dependabot and leaves it on for everybody else. This document records why the exemption is scoped that narrowly, and why the two changes that look like the obvious fix both make things worse.
What forced it
Run 38300320379, on the branch that became #722, reports one problem:
⧗ --- input ---
build(deps): bump github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding from 1.13.19 to 1.13.20
✖ header must not be longer than 100 characters, current length is 103 [header-max-length]
PR Title Check failed four times on that pull request across three head commits, because every synchronize re-lints a title that has not changed. It passed at 20:53 on the same commit it had failed on at 19:50 - an edited event, which is a person renaming the pull request. Four other commits on main carry the same /.../ elision made by hand: #660, #649, #639 and #638.
Ten of the 451 modules in go.mod overflow the cap, and they are not a random ten - they are the transitive trees that bump most often, which is why this came up five times in three weeks. Lengths below are for a 7-character version pair such as 1.13.19, the worst case; the otel bump in #660 measured 115 rather than 117 because its versions are a character shorter.
| Title length | Module |
|---|---|
| 117 | go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc |
| 113 | github.com/AliyunContainerService/ack-ram-tool/pkg/credentials/provider |
| 107 | go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc |
| 106 | github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal |
| 105 | go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc |
| 104 | github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys |
| 104 | github.com/AliyunContainerService/ack-ram-tool/pkg/ecsmetadata |
| 103 | go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp |
| 103 | github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding |
| 101 | github.com/aws/aws-sdk-go-v2/service/internal/presigned-url |
None of them is a direct dependency. All ten arrive through the vendored cosign, Azure and otel trees, so there is no import to rewrite and no module to drop: the names are what they are.
Grouping lengthens the title it was supposed to shorten
The gomod entry in .github/dependabot.yaml had seven groups until 934972b78 removed them, and re-adding one looks like it would collapse these titles to build(deps): bump the opentelemetry group with N updates. It would, on a day when the group has more than one member. On a day when it has exactly one, Dependabot keeps the full module name and appends the group instead:
build(deps): bump k8s.io/streaming from 0.37.0 to 0.37.1 in the k8s group
build(deps): bump github.com/ulikunitz/xz from 0.5.16 to 0.5.17 in the misc group
That is roughly twenty characters added to the form that was already too long. A group containing one long module is strictly worse than no group at all, and a single-module day is the common case for the otel and aws trees - which is exactly when the cap bites. Grouping is a reasonable answer to pull request volume; it is not an answer to title length.
The job cannot be skipped for Dependabot
.github/workflows/commitlint.yaml carried # if: github.event.pull_request.user.login != 'dependabot[bot]' as a commented-out line on the title_check job from 920e51591 until this change. Uncommenting it is the smaller diff and the wrong one: title_check is a required status check on the main ruleset, alongside validate-go-mod, build-e2e, e2e-noncluster, prek and lint. A job that never runs is a result the ruleset and Dependabot’s auto-merge may wait on forever, which trades a red check for a stuck pull request. The merge_group trigger on that workflow exists for the same reason - the job runs and reports success on a ref that has no pull request, rather than not running.
Keeping the job and swapping its config keeps title_check reporting on every event, and keeps Dependabot’s titles linted for everything except the rule it cannot satisfy. type-enum, subject-empty and the rest still apply, so a malformed title from Dependabot fails as loudly as one from a person.
The second config repeats a rule instead of extending the first
.commitlintrc.dependabot.yaml names @commitlint/config-conventional and re-states subject-case, rather than extending its sibling:
extends:
- "./.commitlintrc.yaml"
commitlint resolves an extends entry relative to the config file’s own location, which is why both files live in .github/commitlint/ beside the node_modules that npm ci creates - see choice-binary-artifact-opa-wasm.md for how they got there. But a relative extends goes through @commitlint/resolve-extends, whose loader is built around JS and JSON modules, and a YAML sibling is neither. Duplicating one rule across two short files is cheaper than depending on behaviour that would have to be proven first, and .commitlintrc.yaml holds exactly one rule, so the copy is two lines. Both files carry a comment pointing at the other, because the duplication is the thing that will rot: a rule added to one and not the other means Dependabot and people are linted by different standards without anybody deciding that.
What this does not fix
Check for go mod changes still fails on the first run of every gomod pull request, and that is unrelated and deliberate. go mod vendor deletes the four generated osv-scanner.toml files under vendor/, dev:verifyVendor fails because of it, fix-vendor-osv.yaml pushes them back, and the rerun passes. The cancelled Unit tests, E2E cluster and CodeQL runs on a Dependabot branch’s first commit are that push superseding the in-flight ones. Both are self-healing and need nobody; the header comment on check-go-mod.yaml explains the sequence.
When this entry can go away
Raise the floor rather than delete the file: this stops mattering only if every one of the ten modules leaves the graph, which means cosign, the Azure keyvault SDK and the otel exporters all leaving it. More likely the file outlives the problem quietly. If @commitlint/config-conventional ever gains a rule that needs to differ between the two configs, that is the point to revisit whether a shared JS base is worth the resolution it would buy.
Why the node phases use the dynamic client and a hand-written struct
Three places in cargoship read Kubernetes nodes, and none of them uses a typed client. From the code alone each one looks like the awkward way to do an easy thing - unstructured.NestedBool instead of node.Spec.Unschedulable, a hand-written struct instead of corev1.NodeList. This records why, because the obvious cleanup costs 24MB.
Every typed client registers every API group
k8s.io/client-go/kubernetes/typed/core/v1 imports k8s.io/client-go/kubernetes/scheme, and that package’s init registers every Kubernetes API group - admissionregistration, flowcontrol, resource, all of them. The import is not optional and not per-group: taking the typed core/v1 client takes the scheme, and the scheme takes the rest.
This was measured rather than assumed. Narrowing kubernetes.NewForConfig to corev1client.NewForConfig - the typed client for the one group cargoship touches - moved the binary by 0.01MB and left 56 API groups linked. The typed clients are all one dependency, whichever one you pick.
What worked was avoiding the scheme:
| Client | k8s.io/api groups linked |
|---|---|
client-go/kubernetes (typed clientset) | 59 |
client-go/kubernetes/typed/core/v1 | 56 |
client-go/dynamic | 0 |
client-go/tools/clientcmd | 0 |
k8s.io/api 14MB plus client-go 9.4MB, against five call sites that read a node’s name, its labels, spec.unschedulable and the Ready condition.
What each place does instead
pkg/phase/81_label_nodes.go uses dynamic.Interface against schema.GroupVersionResource{Version: "v1", Resource: "nodes"}. It lists nodes and patches one label, and unstructured.Unstructured serves both directly: GetName and GetLabels are methods on it, and Patch takes the same merge-patch bytes the typed client took. The phase reads no field that needs a typed struct.
pkg/phase/13_detect_removed_hosts.go does not use a client at all. It shells out to kubectl get nodes -o json on the leader over SSH - deliberately, so the check works against a cluster whose firewall admits only SSH - and used corev1.NodeList purely as a JSON decoding target. kubectlNodeList and kubectlNode are that shape in twelve lines. kubectlNodeAddress carries only the address, because the comparison matches any address a node reports against any identifier a host carries and never looks at the type discriminator.
test/kube.go and the cluster suite read the Ready condition and spec.unschedulable through unstructured.NestedSlice and NestedBool. A missing or malformed field reads as not-ready and not-cordoned, which are the right answers while a node is still coming up. NodesResource is exported so the suite and the phase do not disagree about the GVR.
Do not replace these with the typed client
The typed version of any of this is shorter and reads better. It also relinks 24MB for four fields. If a future phase genuinely needs a typed Kubernetes object - a full PodSpec, something with defaulting, anything where the field set matters - that is the moment to reconsider, and the right argument then is that the object is too complex to spell by hand, not that GetLabels is ugly.
clientcmd is unaffected and still typed: it parses and writes kubeconfigs, pulls no API group, and pkg/phase/80_kubeconfig.go and types/distrocfg/admin_credentials.go use it as they always did.
Why the cluster e2e suite splits into a staging half and an engine half
The reasoning below was written against the original ten-machine, three-controller rke2 topology (
mixedOS). The default full walk later switched engines, to a five-machine, single-controller k3s topology (k3sOS) – see the e2e-cluster workflow’s header comment – because rke2’s embedded etcd raft quorum across three controllers was timing out on the hosted runner. The rke2/overlayfs and host-count specifics below are historical; the stage/engine split they justify is still exactly how the suite works today.
The cluster suite (test/e2e/cluster) walks the apply phase list one phase at a time against a bootloose cluster, as choice-phase-e2e-tests describes. That suite is two runs rather than one: with CARGOSHIP_E2E_STAGE_ONLY set it provisions the smaller staging inventory and stops asserting at the boundary phase/60_configure_engine.go draws, and without it, it walks the full inventory end to end. This document records why, because the split looks like an arbitrary line through the middle of a test and the line was not arbitrary.
What forced it
The suite was label-gated into CI on a hosted ubuntu-latest runner and did not pass there. Two things were wrong, and only the first was ours.
phase/55_files_common.go marked a host as claimed from the host list rather than from what it had uploaded. An rke2 package ships no .deb, so the APT phase claimed all five Debian hosts, uploaded nothing to them, and registered an apt-get install with an empty package list – which succeeds. phase/59_bin_install.go, the catch-all that exists precisely for a distro that ships neither RPMs nor debs, then filtered those hosts out as already claimed, and they reached the initialize phases with no engine on them at all. That is a real product bug, it is fixed, and the local run had never caught it because the failure needs a Debian host and a package with no debs to meet.
Underneath it was the blocker that is not fixable in the product. rke2 runs its own containerd, whose default snapshotter is overlayfs, and overlayfs refuses an upperdir that is itself on an overlay mount – EINVAL, reported as "overlayfs" snapshotter cannot be enabled for "/var/lib/rancher/rke2/agent/containerd". A hosted runner’s Docker uses the overlay2 storage driver, so every node container’s root filesystem is an overlay mount and the snapshotter has nowhere to stack. The engine never reaches ready.
The timings from that run are what shaped the rest. Everything through Test_60_ConfigureEngine – fifteen phases, ten machines – took 188 seconds on the runner and passed. Test_61_InitializeControllers alone took 1236 seconds and then failed. The suite is not uniformly expensive; almost all of its cost and all of its environmental risk sit on one side of one line.
The seam was already in the code
Test_60_ConfigureEngine calls harness.dropUploadOnlyHosts(). The Alpine machine that exists to make phase/59 testable as the fallback it is cannot run rke2 – rke2 links against glibc and Alpine is musl – so 60 is where it leaves the walk. The phases up to and including 60 stage files and render configuration; every phase after it installs, starts or queries the engine. That distinction was already written down, already load-bearing, and already the reason one host stops there.
So the split reuses it rather than inventing a second boundary. A stage-only run stops asserting exactly where the Alpine host stops participating, and the sentence that justifies both is the same sentence.
Drawing the line by wall-clock instead – “skip the phases that take more than a minute” – was rejected on the obvious grounds that it encodes a measurement rather than a property, and would have to be re-drawn every time a phase got faster or slower.
An environment variable, not a build tag
The gate is stageOnlyEnvVar, read through strconv.ParseBool, under the CARGOSHIP_E2E_ prefix the suite already uses for its other knobs. Build tags were the alternative and were rejected twice over: a tagged file is invisible to go vet and the linter unless the tag is set, so the gated half rots silently, and the suite would need two go test invocations with different tags to cover what one binary covers now. An environment variable is one build, one binary, and the same code compiled either way.
It is a function rather than a package variable so that it reads the environment when asked rather than at package init, which is what lets a caller set it in process – mage test:endToEndClusterStage does exactly that. CARGOSHIP_E2E_UPGRADE, the opt-in for the upgrade walk, is read the same way for the same reasons; the two gates are independent, and a stage-only run has nothing to upgrade.
Two granularities of skip
ApplyPhaseSuite is skipped per method. Each of Test_61 through Test_81 begins with requireEngine(), which skips that one step. The walk still runs to the end and still tears the cluster down, which is what lets the phases on the far side of the engine half keep running.
The join, upgrade and reset walks are skipped whole, in TestClusterPhases. Each starts from the cluster the apply walk installed – there is nothing to join, upgrade or reset when nothing was started – so gating them method by method would produce three suites of uniformly skipped tests, and a machine provisioned and a phase list walked to reach them. The apply walk is the only one with anything left to say.
The phases after the engine still run
Test_91_Lock, Test_92_Unlock and Test_99_Disconnect carry no requireEngine(), and that is deliberate rather than an oversight. The lock is a file on each host holding the instance ID of the process that took it; unlock removes it; disconnect clears the staged binary paths and drops the SSH connections. All three need a connected host and none needs an engine, so all three run on a stage-only walk and fail it if they regress.
That matters more than it looks. A lock left behind delays the next run by thirty seconds on every node, and a stage-only walk that took the lock and skipped the release would do exactly that to the run after it. The three tests are commented to say they are ungated on purpose, so that a later change adding requireEngine() to everything after 60 has to argue with a comment first.
Nothing else in the 85-95 range can run here. phase/85 through phase/88 and phase/95_daemon_reload.go belong to reset, which tears down an engine that a stage-only run never installed.
Five machines, not three and not ten
The full walk provisions ten machines for reasons choice-phase-e2e-tests sets out: three controllers, six workers, both roles split across Ubuntu and Fedora, plus the Alpine upload-only node. A stage-only run provisions five – one Ubuntu controller, one Fedora controller, one Ubuntu worker, one Fedora worker, and the Alpine node.
Three was the obvious target: one machine per OS family, which is the axis the phase list most visibly routes on. It is one short. APTUploadFiles.Prepare filters both host sets by family and by role, and UploadFilesCommon.Run uploads once per role with that role’s file list:
p.control = p.control.Filter(utils.FilterEngineAlreadyPopulated).Filter(utils.FilterDebianLinux)
p.workers = p.workers.Filter(utils.FilterEngineAlreadyPopulated).Filter(utils.FilterDebianLinux)
With one machine per family each family has to take a single role, so one family’s controller set and the other’s worker set are empty and half the cells the upload phases branch on go unvisited. Those cells are not hypothetical: the host-claiming bug above lives on exactly this axis. Five is the smallest inventory that populates every family-by-role cell, and it is still half the containers, half the uploads and half the disk of the full one.
Nothing else the staging phases do is sensitive to machine count, which was checked rather than assumed. The upload batching in parallelDoUpload is bounded by applyConcurrency, which the suite sets to 300, so all ten hosts already go in a single batch and no count below that changes anything. The WorkerConcurrent batching that the ten-node cluster is partly justified by is reached only in the initialize and upgrade phases, which a stage-only run does not reach.
The host counts the suite asserts on – inventoryHostCount, uploadOnlyCount, clusterControllers, clusterWorkers – were constants pinned to the ten-machine cluster. They are now derived from whichever config the run selected, by countsFor, reading the same kc/kw/kwa name prefixes that renderClusterInventory maps to roles. Writing a second set of constants for the second inventory was the alternative, and it is the arrangement where the two silently disagree after someone edits one.
A volume over the engine’s data directory
Every bootloose machine now mounts an anonymous Docker volume at /var/lib/rancher. A volume is not part of the container’s root filesystem – Docker backs it with a directory on whatever filesystem holds /var/lib/docker – so containerd’s snapshotter gets a plain filesystem and mounts normally. This is the same arrangement kind’s node image makes with its VOLUME declaration, for the same reason.
Anonymous rather than named, on purpose. A named volume would carry one run’s engine state into the next, and every walk in this suite assumes it starts from machines with no engine on them. The cost is that docker rm -f does not reap anonymous volumes, so both StopBootlooseContainers in magefiles/pkg/testrunner/testrunner.go and the workflow’s cleanup steps pass -v.
Configuring rke2 to use the native snapshotter instead was the alternative. It was rejected because it changes what is under test: the suite would then be proving that rke2 installs correctly in a configuration no real deployment uses, and the overlayfs snapshotter is the one that ships.
This fix is unverified. It addresses the exact error in the logs and it is the standard remedy, but confirming it costs a full CI run, and the split is what makes shipping it unconfirmed reasonable – if it does not work, the stage job still covers the phases that do not depend on it.
Two CI jobs rather than one matrix
While both jobs existed, e2e-cluster-stage and e2e-cluster were separate jobs with duplicated steps. A matrix over a mode dimension was the alternative and would have removed the duplication, at the price of turning every field the two jobs disagreed on into an expression over the matrix value. They disagreed on the job timeout, the test timeout, the environment block and the disk-space rationale, and the timeouts differed by a factor of two because that difference was the entire point of having two jobs. Six duplicated boilerplate steps read better than four conditionals.
Only e2e-cluster-stage is left, for the reason the next section gives, so the question is moot unless the full-install job ever comes back.
Both jobs sat behind the same e2e-cluster label at first. The point of the stage job was that it is cheap enough to run unconditionally, and the intent was to drop its if: once a few runs showed what it actually cost and how steady it was. That data came in, and it went the way this section expected: the stage job is cheap and steady. It now runs on every pull request, ungated.
The engine-bootstrap job left CI entirely
The full-install job (e2e-cluster, walking the engine half through bootstrap) did not get the same answer. Even after switching its inventory from the three-controller rke2 topology to the single-controller k3s one to remove the etcd quorum timeouts, the job was still not reliable enough on a free hosted runner to be worth keeping as a CI check – the free tier is not the environment to run a real cluster bootstrap in, independent of which distro is installed.
The choice was between keeping it as a workflow_dispatch-only manual job or removing it from the workflow entirely. It was removed entirely. A manual CI job that nobody runs because the free runner it would run on cannot be trusted is worse than no CI job at all: it looks like a check that exists, and an engineer who runs it and gets a flaky failure has no way to tell a real regression from a runner limitation. The engine-bootstrap walk is still real and still worth running – just on a local box, with mage test:endToEndCluster, documented in e2e-phase-tests. CI’s job is now only the stage half: uploading and rendering files across every OS family, which is exactly what “call that good for the pipeline” asks of it.
What this does not do
A stage-only run does not tell you that rke2 installs. It covers connection, OS detection, fact gathering, host validation, the prepare phases, SELinux, fapolicyd, /etc/hosts, the firewall, all four upload phases, the engine config render, and the lock and disconnect phases at the end – which is most of the phase list and the part a change to pkg/phase is most likely to break. It says nothing about whether the cluster comes up, and the full walk remains the only thing that does.
Why the examples stop at N-2
Every example under example/ is rendered for a set of Kubernetes minor lines, and that set now has a floor: exampleMinorFloor in magefiles/pkg/gen/examples/tags.go, currently v1_35. The newest line and the two before it are rendered; anything older is not, for rke2, k3s, and upstream alike.
Before this, the floor was wherever thirdparty-src/pins.json happened to still pin, plus whatever was already committed – exampleTags adds every version directory it finds on disk to the set it renders, so a line stayed alive by having been rendered once. That is how v1_32 was still being re-rendered on every refresh, long after anybody would install it.
Why a floor at all, rather than letting the pins decide
An example is not just a version string. It is a set of components that have to work together: the distro build, the kubeadm-published OS packages, containerd, and – for the upstream examples – a CNI the package applies itself.
The CNI is what forces the question. A CNI release supports a handful of Kubernetes minors and says which ones: cilium e2e tests exactly four and states outright that “older Kubernetes versions not listed here do not have Cilium support” (Documentation/network/kubernetes/requirements.rst, in the cilium tree). Cilium 1.20 covers 1.33 through 1.36. So an upstream-cilium example rendered for 1.32 would carry a cilium that cilium does not claim works there – a package that builds, publishes, and installs, and then does not route pod traffic.
Pinning a CNI per minor line would answer that differently, and it was considered. It means a version map per CNI, a rendered cilium manifest per chart version, and a judgement call per line about which CNI release to trust – all to keep alive lines that Kubernetes itself no longer supports. N-2 matches what upstream Kubernetes supports, which is the same window the CNIs are tested against, so one floor covers both and there is no map to maintain.
Why moving it is two hand edits
The floor is a constant, and tags below it are dropped rather than rendered. What is already committed below the floor is not deleted by a render.
A generator that deleted a whole example tree because a pin moved would be doing something a reader of mage generate:examples has no reason to expect, and the thing it deleted is the one copy of a definition somebody may be installing from. Retired rke2 builds are pruned automatically (writeExample) and that is a different case: an rke2rN whose RPMs Rancher has already removed cannot be installed from at all, so keeping it only invites someone to try. A 1.34 example still installs fine. It is just outside what this repository carries.
So raising the floor is: edit the constant, and git rm the lines that fall below it. Anything missed shows up as a tree that stops changing, not as a tree that silently keeps claiming to be current.
What this cost
Three minor lines per flavor, deleted: example/*/v1_32, v1_33, v1_34, along with their thirdparty-src/<distro>/ source copies and their generated pkg/engineconfig/gen/<distro>/ packages. That is a breaking change for anyone pointing at one of those paths, and it is recoverable from git history rather than regenerable – the upstream artifacts those examples name are still published, so a checkout of the commit before this one renders them again.
Why a file override refuses a file with no shasum
--file-override redirects the file downloads a distro definition declares to an internal mirror or to a directory of pre-staged assets. If the file it applies to declares no shasum, create fails and names the file instead of downloading it. The restriction looks like an arbitrary inconvenience from the flag’s side, because nothing about a URL rewrite requires a checksum to work, and it will keep looking that way until someone reads why it is there.
The override is the only thing that makes the checksum load-bearing
Without an override, a file source is a URL that whoever wrote the distro definition chose, pointing at a host they chose. A missing shasum there is sloppy, but the trust is the same trust you already placed in the definition.
An override moves that decision to whoever runs create. The bytes now come from somewhere the definition’s author never named and cannot see – an Artifactory instance, a Nexus, a directory somebody populated by hand last quarter. The declared shasum is the one thing that still ties what lands in the package to what upstream published. If it is absent, the override is an unauthenticated substitution, and the resulting package carries no evidence that it contains what it claims to.
That is a bad trade to make silently, and it is worse because it fails open: the package builds, the cluster comes up, and nothing distinguishes it from a package built against upstream.
Why not just warn
A warning is what the code used to do about a checksum mismatch, and it is the reason this restriction had to be added at all. The two file loops in AssembleDistro logged every fileGrabber error and continued, so a mismatch cost one line in a build log and shipped the bad file anyway. Anyone who trusted the “checksums protect you” story was wrong for as long as that was true.
The loops now return the error. Adding an override feature on top of a warning would have recreated the same hole one level up.
Where this bites, and what to do about it
The failure mode you will actually hit is an internal distro definition that never bothered with shasum because it always downloaded from a host the team controlled. Adding an override to it fails immediately, on the first file.
The fix is to add the upstream checksums to the definition, not to relax this. That is work, but it is work that makes the definition better whether or not an override is ever used: the same checksums catch a truncated download, a silently re-tagged release, and an upstream artifact rebuilt in place.
If some future case genuinely needs an unverifiable overridden download, it needs an explicit opt-out flag with a name that says what it gives up, and a line in the build log every time it fires. It does not need this check quietly deleted.
What the record is for
Because the checksum is mandatory, .build.fileSources can record it next to the mirror each file came from, and every entry is then a complete claim: these bytes, from this place, matching this digest. That is only worth writing down because the refusal above guarantees the digest exists. Drop the refusal and the provenance record degrades into a list of URLs nobody can check.
See also
pkg/fileoverride– the matcher, and why it parses URLs instead of comparing string prefixes.docs/guides/file-override.md– the user-facing guide.
Why a multi-arch package stores images under one OCI index
A cargoship package is fat: one tarball carries every architecture it targets, and the host’s architecture selects what gets uploaded at apply time. Files fall out of that arrangement for free, because fileGrabber keys its output directories by the index of the entry in spec.config.files, so an amd64 entry and an arm64 entry are two list entries and land in two directories without anyone arranging it. Images do not, and the reason is not visible from the code that writes them.
The OCI store tags by image reference
images.Pull (pkg/images/pull.go) copies each pulled image into an oci.Store rooted at the package’s images directory, and the store is keyed by the image reference it was pulled as. That reference is registry/name:tag. It says nothing about a platform.
Pull two platforms of registry.k8s.io/pause:3.10 and both copies want that one tag. Whichever finishes last wins, the other is left in the layout as unreferenced blobs, and a later Resolve returns a platform nobody chose. The failure is quiet: the tarball is well formed, the upload succeeds, and the node fails later with an exec format error from a container it cannot run.
What the package does instead
orasSave copies a manifest and returns its descriptor without tagging anything. tagImage then decides what the reference points at:
- One manifest, which is every package that targets a single architecture, is tagged directly. This is byte for byte what cargoship has always written, so nothing about an existing package changes.
- Several manifests are collected into an
application/vnd.oci.image.index.v1+jsonblob, and the index is tagged instead. One reference resolves to every architecture the package targets.
The store is content addressed, so the architectures share every blob they have in common. Two architectures of an image built FROM scratch still duplicate their one binary layer, because those layers genuinely differ, but a shared base image is stored once no matter how many architectures reference it. The pull is faster for the same reason: oras.Copy skips a blob the store already holds.
Each entry in the index carries a platform. A descriptor resolved through a registry’s own index already has one; a registry that serves a bare manifest does not, so ensurePlatform reads the platform out of the image config, which is where the registry would have read it from as well. The manifests are sorted by platform before the index is marshalled, so two builds from the same inputs produce the same index digest and --reproducible keeps working.
Ordering the layout’s index.json
Oras builds images/index.json by ranging over its map of tags, so the entries land in a different order on every run even when the pull fetched exactly the same images. That file ships inside the package and is covered by checksums.txt, so the order alone was enough to make two otherwise identical builds differ. After the last image is tagged, sortIndexFile rewrites the file with its entries ordered by digest, breaking ties on the reference annotation for an image tagged under more than one name.
Entry order carries no meaning to anything that reads the layout, so the sort runs on every build rather than only under --reproducible. The flag stays what it always was: it pins the recorded build timestamp. Note that this only orders the index. Layer order inside a manifest is fixed by whoever built the image, since the manifest digest depends on it.
Selecting a platform when uploading
pkg/phase/50_uploadfiles.go resolves the image reference against the package’s store and exports a tarball for the node. With an index in the store, resolveImageManifest picks the child manifest matching the platform and hands that descriptor to tarball.Write, so the exported tarball holds exactly one platform and looks the same as one exported from a single-architecture package.
The selection is done here rather than left to the tarball writer. A docker-save tarball describes one image per entry, so handing the writer an index would either write blobs for platforms the node will not run or leave it referencing manifests whose blobs are not in the tarball. Picking the child ourselves avoids both.
Reaching that child needs one extra step, because layout.Path.Image resolves a digest against the layout’s index.json and does not recurse into a child index. A multi-arch package tags a synthesized index, so layoutImage looks the parent up first and asks it for the per-arch manifest.
The matcher is imageExportPlatform, which pins the OS to linux and the architecture to the host being uploaded to rather than the machine running cargoship. Each architecture the cluster runs gets its own tarball directory, because the tarball name is derived from the image reference alone and two architectures of one image would otherwise collide.
Alternatives that were considered
A directory per architecture. images/amd64 and images/arm64, each a complete OCI layout with its own tag namespace. It is the smallest change to the writer and it needs no index at all. It was rejected because blobs are not shared between two layouts, so a shared base image is stored once per architecture, and because it gives the package format two shapes: readers have to know whether to look in images or images/<arch>, and every path that names the images directory has to learn the difference.
Encode the platform in the tag. The store would hold registry.k8s.io/pause:3.10-amd64 and ...-arm64 in one layout. This dedupes blobs, but the upload phase would no longer resolve the reference the package definition actually lists, and the mangling scheme would become a compatibility surface between the package format and every future reader. The image reference in distro.yaml should be the thing that resolves.
One tarball per architecture. This was the design alternative rejected at the level of the whole feature. It solves the tag collision by never having two platforms in one package, at the cost of an operator holding several tarballs for one cluster and choosing between them per node, which is precisely the problem a fat package exists to remove.
What it means for publishing
Publishing a package to a registry needed no changes. LayersFromImages in pkg/coci/pull.go reads images/index.json, finds the entry annotated with the image reference, and already dispatches on media type: layersFromIndexChildren walks an index entry and recurses, and the layer list is deduplicated before it is returned. A published multi-arch package therefore shares blobs in the registry the same way it shares them on disk.
Why google/go-containerregistry and not distribution/distribution for the in-memory test registry
Cargoship’s e2e suite needs a real OCI registry to publish/pull against without a network dependency or a live registry container. test/registry.go provides test.SetupInMemoryRegistry(t), which starts one in-process on a random localhost port and tears it down via t.Cleanup.
ORAS (oras.land/oras-go/v2) isn’t a candidate here at all – it’s the client library Cargoship already uses in pkg/coci to talk to registries. It has no server implementation, so it can’t be “the registry” in a test; it’s what the CLI uses to talk to whichever registry the test stands up.
The zarf project’s own src/test/testutil/registry.go (this package is modeled on it) uses github.com/distribution/distribution/v3 – the reference registry server implementation – with its inmemory storage driver. That was the first thing tried here, and it was rejected:
- Dependency weight.
distribution/distribution/v3wasn’t a dependency of Cargoship at all before this (only present transitively ingo.sum, unused). Adding it as a direct import and runninggo mod tidypulled in a large new indirect tree: Redis and Prometheus clients, severalgo.opentelemetry.io/otel/exporters/*packages, logging bridges, etc. – dependencies of the registry’s optional storage/metrics backends that Cargoship will never use for an in-memory test double. - It broke the build. That new tree included
go.opentelemetry.io/otel/log v0.21.0, which is not API-compatible with thego.opentelemetry.io/otel v1.45.0core Cargoship already pins elsewhere.go build ./...failed insidego.opentelemetry.io/otel/exporters/stdout/stdoutlog(undefined: log.Value,v.Kind undefined, etc.) – a version-skew problem with no clean fix short of forcing otel versions across the whole module.
github.com/google/go-containerregistry’s pkg/registry package was used instead. It’s the same kind of thing Cargoship’s own crane-adjacent tooling already depends on transitively (it was in go.sum already, unused), so promoting it to a direct dependency added exactly one new package to vendor/ and nothing else – go mod tidy only moved the existing entry from indirect to direct. pkg/registry.New() returns a plain http.Handler with no storage-backend options to drag in a dependency tree: wrap it in a stock net/http.Server, Listen on :0, done.
Net effect: same job (an in-memory OCI-compliant registry for tests), zero new transitive dependencies, no build breakage.
Why the engine’s manifest directory is pruned through a glob
Cargoship prunes the directories it owns. StaleFiles (types/distrocfg/managed_files.go) lists every file under each of a distro’s ManagedDirs and removes the ones the current configuration no longer asks for, so a registry CA certificate dropped from a cluster inventory does not sit on every node forever. Pruning runs while a node is stopped and about to be restarted, which is the one moment a file can be removed without the engine noticing it go.
That works because a managed directory holds only files cargoship put there. Two of the three qualify outright: /etc/cargoship/tls and the state directory were created by cargoship and are cargoship’s to empty. The third is not cargoship’s at all.
The manifest directory is shared
<data>/server/manifests belongs to the engine. RKE2 and K3s ship their own manifests there – the CNI chart, CoreDNS, metrics-server, the ingress controller – and lay them down again on startup. Cargoship writes into the same directory because that is where a HelmChartConfig has to live for the engine’s helm controller to read it: the values a package’s spec.config.engine.manifest subtree produces become one <chart>-config.yaml per chart (types/distrocfg/rancher_common.go).
Pruning that directory the way the other two are pruned would delete every manifest the engine put there on the first sync, which takes the cluster apart. Not pruning it at all would leave a HelmChartConfig behind for a chart a package no longer configures, and the helm controller would keep reconciling the chart to values nobody asks for any more – the exact failure StaleFiles exists to prevent, on the files where it matters most, since these are the only files cargoship writes that something else then acts on.
The glob
ManagedDir therefore carries an optional Glob, and the manifest directory sets it to *-config.yaml – the same suffix cargoship names its own HelmChartConfig files with. An empty glob means the whole directory is cargoship’s, which is what the other two use.
The suffix is the whole mechanism for telling the two sets of files apart, so it is a single constant, helmChartConfigSuffix, used both by the writer and by the glob. A rename in one place that missed the other would either start pruning the engine’s files or stop pruning cargoship’s.
Alternatives considered
A separate directory for cargoship’s manifests. There is no such thing: the helm controller watches one path, and a HelmChartConfig outside it is inert.
A marker inside each file, read before deleting. Pruning would then have to read every file in the directory over SSH on every sync, on every node, to decide about a handful of them. The name already carries the same information, and it carries it without a read.
Recording what was written, as the ufw backend does. The ufw firewall backend keeps a state file of the rules it applied, because ufw cannot express “replace everything I own”. Here the desired set is already in hand at prune time – filesFor(host) is what drift detection just compared against – so a state file would add a thing to keep in sync for no new information.
A prefix instead of a suffix. A prefix would have to be cargoship’s own, which would rename every HelmChartConfig the project has ever written and orphan the files already on upgraded nodes. The suffix is what the files are already called.
A malformed pattern matches nothing rather than everything (ManagedDir.prunes). Pruning is cleanup, and cleanup is not worth deleting something on a guess.
See package-values for how a package produces the HelmChartConfig files this glob scopes pruning to.
Why the local VM fleet runs on plain qemu, and what it does not replace
test/microvm brings Enterprise Linux virtual machines up and down on a developer’s own machine so that the phases routing on OS family can be exercised against a real kernel. It exists because a container cannot answer the question pkg/phase/21_prepare_selinux.go asks: that phase gates on getenforce reporting Enforcing, no container reports it, and the phase is fatal on failure – so the one branch able to stop an apply was the one branch never run. pkg/phase/22_prepare_fapolicyd.go, the firewalld backend in pkg/firewall/firewalld.go, the semodule scriptlets in k3s-selinux and rke2-selinux, and the reboot in pkg/phase/23_prepare_kernel_modules.go are all in the same position, and selinux: true is set by every shipped package under example/ without anything ever having checked that an engine configured that way starts.
qemu, not Firecracker, and not libvirt
Firecracker is the obvious reach for “microVM”, and it does not fit. It requires an uncompressed kernel binary with every driver statically compiled and no bootloader, and explicitly does not support loadable modules – which is exactly what k3s and rke2 need, and what 23_prepare_kernel_modules writes /etc/modules-load.d entries for. A RHEL-family GenericCloud qcow2 cannot boot on it at all without extracting a kernel out of the image first. qemu’s own microvm machine type carries the same constraint for the same reason, so the name is not the thing that matters here.
libvirt would work, and costs more than it returns. It needs libvirt-daemon-kvm and virt-install installed, a daemon running, and its NAT network managed through the host’s firewalld – host state, on a machine whose SELinux and firewall configuration is frequently the thing under investigation. Lima was ruled out on a narrower point: its default 9p mounts are documented as incompatible with AlmaLinux, CentOS Stream, Oracle Linux and Rocky.
Plain qemu needs nothing beyond qemu-system-x86_64, qemu-img, mkfs.vfat, mcopy and ssh, runs as the developer, and leaves nothing behind. Each node is a copy-on-write overlay over one cached base image, so a bring-up costs a few megabytes and the overlay is recreated every time – which is also why Up refuses a live fleet rather than adopting it. A node carrying state from a previous run is the single thing that makes a phase failure impossible to reason about.
A multicast socket netdev, not a bridge
Every node gets two interfaces. The management one is a slirp user netdev with loopback forwards, which needs no privileges and gives the outbound access phase 21 depends on – it installs container-selinux by name and is fatal if it cannot reach a mirror. The private one is -netdev socket,mcast=, which joins every qemu process in the fleet onto one broadcast domain.
A tap device or a bridge would be the conventional answer and both need root. The deciding point is subtler than that: on a host running firewalld with Docker’s nftables firewall backend, container-to-container traffic is silently dropped, which is enough on its own to stop every multi-node run from forming a cluster. A bridge is subject to the same ruleset. A multicast socket between qemu processes over loopback is not something the host’s firewall has an opinion about, so the fleet’s own networking cannot be broken by the host configuration a developer is in the middle of changing. vde_switch would have been the other rootless option and is not packaged for current Fedora.
The private segment is also why Inventory sets PrivateAddress and PrivateInterface explicitly, where the bootloose equivalent leaves both to fact gathering. A node’s default route is on the management interface, so discovery would give every host the same slirp address – 10.0.2.15, reachable from no peer – and the firewall and /etc/hosts phases would faithfully distribute it.
Infra nodes are workers with a profile of their own
A fleet is counted in three groups – control, worker, infra – and the third is not a third role. The engine has two, so an infra node joins as a worker and is distinguished only by its Profile, which pkg/phase/81_label_nodes.go turns into a node-role.kubernetes.io/infra label and which a cluster config can hang a per-profile Concurrency off.
They exist because every other host in a generated inventory has a profile that restates its role, which makes both of those mechanisms untestable: a label derived from controller on a controller proves nothing about a label derived from a name the role does not supply, and a per-profile concurrency is indistinguishable from the phase default when every profile holds the same hosts as its role. One group whose profile is not its role’s name is enough to tell those apart.
They are named ki*, a prefix of its own rather than an extension of the worker prefix the way the container fleet’s kwf* and kwa* are. The three prefixes then partition the names, so nothing matching on them depends on test order or on matching longest – which the container suite’s prefixes do, since kwa is also kw.
The consequence is that a mapping keyed only on kc and kw claims an infra node as neither. Nothing in test/microvm is such a mapping: Inventory reads a node’s role and profile off the node, not off its name. The container suite’s renderClusterInventory is one, so wiring infra nodes into that fleet means teaching it this prefix rather than relying on the name to carry the role.
The group is created last, after the plain workers, so that asking for infra nodes does not renumber the hosts ahead of them – a fleet brought up with none is addressed exactly as it was before the group existed.
internal/, not magefiles/pkg/
magefiles/AGENTS.md puts implementation logic under magefiles/pkg/, and this package is deliberately not there. It has two consumers – the Dev mage targets and the e2e cluster suite – and a test suite importing out of magefiles/ would invert the dependency. internal/ is what the root CLAUDE.md names for packages private to the module, and the mage targets stay thin wrappers over it, which is what that rule is protecting.
Fleet state lives under build/microvm/ rather than TMPDIR. An overlay reaches a quarter of a gigabyte after one boot and one dnf transaction, and TMPDIR is routinely a tmpfs here – the repository’s own .envrc points it at one. build/ is already the scratch directory, gitignored and on real storage. Note that dev:clean does not touch it – that target removes built binaries, and removing a live fleet’s directory out from under it would orphan the qemu processes holding its overlays. dev:vmDown is what takes a fleet away.
Rocky only, and why that does not retire the bootloose cluster
The fleet runs one distribution. choice-phase-e2e-tests argues that the OS mix is the substantive part of the bootloose cluster – apply routes on OS family in at least five places, and on a single-family cluster the branch that does not match is never taken while the gate assertions still pass, so the suite reports coverage it does not have. That argument applies to this fleet exactly as written.
So the VM backend is additional coverage, not a replacement. It is the only way to run the Enterprise Linux branch against something that actually has SELinux, fapolicyd and firewalld; the bootloose matrix remains the only thing that covers routing between families, and the Alpine upload-only node remains the only host belonging to neither. CARGOSHIP_E2E_BACKEND selects between them and defaults to bootloose.
Adding a second guest image is a later change and a cheap one – the image is one pinned URL and digest in test/microvm/image.go. It was left out of the first version because one distribution with SELinux enforcing closes the gap that motivated the package, and a second would double the bring-up cost of every run before anything had been learned from the first.
Two qemu details that each cost a boot to find
-display none, not -nographic: qemu refuses -nographic together with -daemonize, and reports it only once the process has been launched, so the failure arrives as a node that never appears rather than as a bad flag. And the overlay is opened cache=unsafe, which is correct precisely because these disks are disposable – there is no state on them worth an fsync, and the next bring-up recreates them.
Why the nftables backend shells out to nft and owns a single table
Cargoship’s firewall support (cargoship apply --firewall) renders a backend-neutral firewall.Plan onto whichever firewall a node runs. firewalld and ufw cover Enterprise Linux and Debian-family hosts that use a front end. The nftables backend (pkg/firewall/nftables.go) covers the rest: Debian and Arch nodes configured by hand, and the minimal or immutable images such as CoreOS and Flatcar that ship no firewall front end at all. Several design choices there are not obvious from the code, and each had an alternative that was considered and rejected.
Shelling out to nft rather than talking netlink
github.com/google/nftables is the native Go netlink client for the same subsystem, and it would give typed rule construction instead of string rendering. It was not used:
- It runs in the wrong process. Cargoship configures hosts over SSH; every other backend, and every other host-modification phase, works by executing a command on the remote node through
rig. A netlink library would only configure the local machine, so using it would mean shipping and bootstrapping a helper binary onto every node – a distribution problem cargoship does not otherwise have. - New dependency for no gain.
google/nftablesis not ingo.sumat all today, and it pullsmdlayher/netlinkandvishvananda/netnswith it.nftis already present on any host that qualifies for this backend, sinceDetectrequires it. - The rendered ruleset is the auditable artifact. Cargoship writes
/etc/cargoship/nftables.nftand loads it. An operator can read exactly what cargoship intends to apply, diff it between runs, and load it by hand. A netlink client leaves nothing behind to read.
The cost is that rule construction is string formatting, so rendering bugs are not caught by the compiler. That is why Apply runs nft -c -f before nft -f: -c parses and validates without loading, so a malformed ruleset fails before it can land on a node cargoship may no longer be able to reach.
Replacing one owned table instead of reconciling against a state file
The ufw backend records the rules it applied in /var/lib/cargoship/ufw.rules and, on the next run, deletes the recorded rules the current inventory no longer asks for. That is necessary there because ufw has no way to express “these are all my rules, replace them”.
nftables does. add table inet cargoship, delete table inet cargoship, then the table’s full definition, submitted as one nft -f transaction, atomically replaces everything cargoship owns. A rule dropped from the inventory is gone when the transaction commits, with no record of the previous run needed and no window in which the node has partial rules. The add before the delete is not redundant: delete table on a table that does not exist is an error, and add on one that does is a no-op, so the pair makes the script work on a node cargoship has never configured.
Deleting nothing is also how teardown works. An empty plan renders just the add/delete pair, which removes cargoship’s table and leaves the host’s own rules untouched.
Never flushing the ruleset, and never reading another table
The reflex when writing an nftables integration is flush ruleset. That would be actively harmful here. Kube-proxy in nftables mode keeps its service and endpoint rules in this same subsystem, and so does every CNI that implements network policy. A global flush would break cluster networking until each of those components noticed and resynced – on a node cargoship is in the middle of configuring, quite possibly the node it is reaching the rest of the cluster through.
So cargoship writes one table and reads none. Nothing outside inet cargoship is inspected or modified, which also means an operator’s own tables survive an apply untouched.
policy accept on every base chain
Every base chain cargoship writes has an accept policy. This is the same constraint that stops cargoship from running ufw enable: a default-drop policy applied from a remote phase would cut the SSH connection cargoship is running over, and the node would be unreachable with the phase half-finished.
There is a consequence worth stating plainly, because it makes the nftables backend weaker than the others in one respect. An accept verdict ends cargoship’s chain, not the netfilter hook – other base chains at other priorities still see the packet. So a rule in an operator’s own table can drop traffic that a cargoship rule allowed. Firewalld’s trusted zone is the last word on a packet; cargoship’s nftables rules are not. A deny or reject rule does terminate evaluation, so the restrictive direction behaves as written; only the permissive direction is advisory.
Detecting on the service or a conf file, not on a non-empty ruleset
Detect requires nft to exist, and then either the nftables service to be running or one of /etc/nftables.conf and /etc/sysconfig/nftables.conf to be present. Probing both paths rather than branching on the distro type keeps the check independent of types/os, since the split is Debian/Arch/Alpine versus Enterprise Linux/SUSE and does not line up with anything else cargoship distinguishes hosts by.
The tempting alternative – treat a non-empty ruleset as evidence of a host firewall – was rejected because it is true on every node of a running cluster. Kube-proxy and the CNI populate the ruleset on hosts whose operator never configured a firewall at all, and claiming those hosts would mean applying rules where the inventory’s author expected none.
The gap that leaves is real and deliberate: a CoreOS or Flatcar node with nft present but no nftables service and no conf file does not match, and gets no firewall configuration. Widening the check to close that gap reopens the false-positive problem above, so the conservative direction was chosen.
Matching last
The backend order in pkg/firewall/firewall.go is firewalld, ufw, nftables. firewalld and ufw are both front ends onto nftables, so a host running either would satisfy the nftables Detect as well. The front end is the one an operator expects cargoship to configure – rules written underneath it would be invisible to firewall-cmd and ufw status, and a front-end reload could tear them out. Putting nftables last makes it the fallback it is meant to be.
Persisting with an include line
Cargoship’s rules are loaded into the running ruleset by nft -f, which does not survive a reboot. Persistence is an include "/etc/cargoship/nftables.nft" line appended once to whichever boot-time file the distro reads.
The alternative, dumping nft list ruleset into that file, would capture kube-proxy’s and the CNI’s rules along with cargoship’s and replay them at boot as static rules – stale ones, describing services that may no longer exist. Appending an include leaves the operator’s file otherwise intact, keeps cargoship’s contribution in one identifiable place, and stays correct when cargoship’s ruleset is regenerated. A host with neither conf file gets a warning rather than an error: the rules are loaded for this boot, but nothing will reload them at the next one.
One inet table rather than separate ip and ip6 tables
The inet family covers IPv4 and IPv6 in a single table, so a plan renders one table, one set of chains, and one transaction regardless of what the cluster’s addressing looks like. The cost is that address matches must name their family explicitly – ip saddr versus ip6 saddr – and that trusted addresses split into two sets, cluster_v4 and cluster_v6, since a set has one type. Both are handled in rendering (nftAddrFamily, nftSplitFamilies) and are cheaper than maintaining two parallel tables and deciding which of them a dual-stack plan belongs in.
Why vendor/ carries four generated osv-scanner.toml files
go mod vendor copies four files into vendor/ that have nothing to do with Go: a package-lock.json under golang.org/x/telemetry, and a requirements file each under github.com/theupdateframework/go-tuf, github.com/txn2/txeh and go.opentelemetry.io/otel. Next to each of them now sits a generated osv-scanner.toml that tells a scanner to skip that ecosystem. This document records why those four files exist, why they are generated rather than committed by hand, and what they deliberately do not cover.
What forced it
The OpenSSF Scorecard Vulnerabilities check runs osv-scanner recursively over the whole checkout and scores it in checks/evaluation/vulnerabilities.go as MaxResultScore - numVulnsFound, floored at zero. There is no severity weighting and no reachability analysis: every finding is one point, so ten findings and four hundred score the same. The check read 0 against 42 findings, and vendor/ supplied 40 of them.
| Manifest | Ecosystem | Findings | What it pins |
|---|---|---|---|
vendor/golang.org/x/telemetry/package-lock.json | npm | 34 | The devDependencies of the web UI x/telemetry serves from its repo |
vendor/github.com/theupdateframework/go-tuf/requirements-test.txt | PyPI | 5 | The Python harness go-tuf runs its own conformance tests under |
vendor/github.com/txn2/txeh/requirements-docs.txt | PyPI | 1 | The MkDocs toolchain that builds txeh’s documentation site |
vendor/go.opentelemetry.io/otel/requirements.txt | PyPI | 0 | A codespell pin for otel’s own spelling check |
None of it is a cargoship dependency. Nothing in those files is compiled, executed, installed or shipped; they are in the tree because they happen to sit in the directory of a module cargoship imports Go packages from, and go mod vendor copies directories. govulncheck -mode=source ./... – which resolves what the binaries actually call – reports zero.
So the advisories are true statements about postcss and mkdocs-material, and false statements about cargoship. The check was not measuring this repository.
The otel row is zero findings today and has an override anyway. A requirements.txt with one package in it is one CVE away from costing a point, and the whole purpose of the arrangement is that a new advisory upstream is not an event here.
Deleting the files does not work, and neither does un-vendoring
rm -rf vendor && go mod vendor puts all four back; they are part of what the vendoring step produces. Removing them by hand means removing them again after every dependency bump, and a run of go mod vendor between the removal and the Scorecard run silently undoes it.
Dropping vendor/ entirely would have removed the problem at the root and was never a candidate. The tree is vendored so the repository builds on a management node with no route to a module proxy; that constraint is the reason vendor/ exists at all, and it is not negotiable for a scorecard point.
The config has to sit beside the manifest
osv-scanner resolves configuration relative to the file it is scanning, not to the root of the scan, and it only consults a single root config when the caller passes ConfigPath. Scorecard’s clients/osv.go builds osvscanner.ScannerActions without one. A single osv-scanner.toml in the repository root therefore changes nothing at all – measured, not assumed:
| Arrangement | Score | Findings |
|---|---|---|
| No config | 0 | 42 |
One osv-scanner.toml at the repository root | 0 | 42 |
| One beside each of the four manifests | 8 | 2 |
That is the entire reason the files are scattered through vendor/ instead of sitting in one obvious place at the top of the repository, which is what anyone would try first.
Ecosystem-wide overrides, not lists of advisory IDs
[[IgnoredVulns]] suppresses named advisories and works – it took the count from 42 to 41 when pointed at the one txeh finding. It was rejected because the claim being made here is not “these particular advisories are harmless”. It is “this ecosystem is not part of the build”, which is a property of the manifest and stays true as the advisory list changes. An ID list would need a commit every time a new CVE landed against a lockfile nobody here reads, and each of those commits would look like a security decision while being pure bookkeeping.
[[PackageOverrides]] with only ecosystem and ignore = true says the intended thing once. Each generated file carries a reason and a comment explaining itself to whoever finds it while reading vendored third-party source.
Generated by mage, not committed by hand
magefiles/pkg/devtools/osv/osv.go holds the four entries and the text they render to; Dev.WriteOSVOverrides writes them immediately after go mod vendor recreates the tree, so the normal way of updating dependencies leaves them in place without anybody remembering. Committing them as ordinary files was the alternative, and it fails the first time somebody runs mage dev:vendor and commits the result.
Dev.VerifyVendor is the guard, wired into check-go-mod.yaml. It fails if a declared override is missing or has drifted from the generator, and – the half that matters more – it fails if vendor/ contains a non-Go dependency manifest that no entry covers. Without that second check, a dependency bump that happens to pull in a module carrying a Gemfile.lock would drop the score on some later Scorecard run with nothing in the diff to point at. It fails on the commit that caused it instead.
Dev.VerifyVendor also fails when a manifest an entry names has gone away, which is what turns a stale entry into a build error rather than a file quietly doing nothing.
Dependabot deletes them, so a workflow puts them back
Generating the overrides from Dev.Vendor assumed the normal way of updating a dependency goes through mage. It does not. Dependabot re-runs go mod vendor itself for every gomod update, the recreated tree arrives without the four files, and Dev.VerifyVendor fails the pull request. #464 and #465 each needed a hand-written ci: readd the osv scanner artifacts commit on the Dependabot branch, and #463 exported Dev.WriteOSVOverrides so that commit could be made without a pointless re-vendor.
.github/workflows/fix-vendor-osv.yaml makes that commit instead. It is the one workflow here that runs on pull_request_target, because it needs a token that can push to the pull request branch and Dependabot’s pull_request runs are handed a read-only one. That trigger is also why the job never fetches the pull request at all: it checks out the base branch, runs Dev.WriteOSVOverrides against that tree, and writes the four results onto the pull request branch through the git data API – a blob per changed override, a tree on top of the branch’s own tree, a commit, and a fast-forward update of the ref. Putting a freshly bumped dependency’s tree on a runner that holds a write token is the specific failure pull_request_target is known for, and the API route means it never lands there to begin with. Generating from base is sound because the file contents come from osvOverrides in magefiles/pkg/devtools/osv/osv.go, and a dependency bump never touches that file.
An earlier version checked the pull request out beside the base branch and pushed with git. It never ran anything out of that checkout either, but Scorecard’s Dangerous-Workflow check cannot see that: its validateUntrustedCodeCheckout flags any actions/checkout in a pull_request_target workflow whose ref mentions github.event.pull_request, which caught both the head checkout and the base one that named base.sha – a trusted commit. The check scored 0 and dropped the overall score by about a point. Dropping ref from the base checkout is not a workaround: on pull_request_target the default checkout is already the base branch tip, so naming it bought nothing and claimed something the job does not do.
The commit is built with explicit author and committer fields rather than git commit, so the [bot] identity and the Signed-off-by trailer the DCO check wants are carried the same way they were. The ref update is a plain PATCH with no force: the parent is whatever git/ref/heads/<branch> returned moments earlier, so a Dependabot push that lands mid-run makes the update fail instead of overwriting the newer commit.
The push uses the release-please GitHub App rather than GITHUB_TOKEN, which is not a preference: a push authenticated with GITHUB_TOKEN raises no pull_request event, so validate-go-mod would stay red on the superseded commit and the pull request would still not be mergeable.
Two cases are deliberately left to fail. A bump that drops a module entirely leaves no directory to copy into, and a bump that pulls in a module carrying an uncovered lockfile trips the second half of Dev.VerifyVendor. Both need osvOverrides edited by a person, so the workflow skips them and lets the check report.
What this does not do
It does not raise the check to 10. What remains is Go, and Go is where an override would be wrong: an advisory against a module that is actually in the binary is the thing this check exists to report.
| Advisory | Module | Status |
|---|---|---|
GO-2026-6225 | docker-credential-acr-env | Genuinely linked, via cmd -> zarf signing -> cosign CLI options. Fixed in: N/A |
GO-2026-5932 | golang.org/x/crypto/openpgp | Not in the build: go mod why reports the main module does not need the package. Fixed in: N/A |
CVE-2026-1229 | cloudflare/circl | Already fixed upstream. GO-2026-4550, the same advisory in the Go database, records the fix in v1.6.3 and the dependency is at v1.6.5; the CVE record carries only a commit range, so the match is a defect in the advisory rather than in the dependency |
None of the three has a fixed version to move to, so the ceiling is 7. Expect it to move between 7 and 8 without anything here changing: the circl entry is only matched when the OSV snapshot a given Scorecard run downloaded happens to carry it, and it appeared and disappeared between two scans an hour apart while this was being written. The row has already been rewritten once for that reason: it named CVE-2025-8556 – a different circl advisory, against FourQ rather than secp384r1 – which stopped matching and was replaced by CVE-2026-1229. Both are commit-range-only records against a module whose version range says it is fixed, so expect the identifier in that row to keep changing while the row itself stays true. A score that oscillates by one for that reason is not a signal, and chasing it with an [[IgnoredVulns]] entry for a Go advisory would be the wrong trade – the honest fix is to get the version range filed against the advisory.
More broadly, none of this makes anything safer. govulncheck -mode=source ./... reported zero before the overrides existed and reports zero after; what changed is that a check which was describing x/telemetry’s frontend is now describing cargoship. That is worth the four files and the mage target, and it is worth being clear that it is a measurement fix and not a security fix. The maintenance cost is real and lands on whoever next adds a dependency that carries a lockfile – Dev.VerifyVendor exists so that person gets a clear failure rather than a mystery.
Why the cluster e2e suite steps through the apply phases one at a time
The cluster suite (test/e2e/cluster) used to be a lifecycle test: provision a bootloose cluster, run cargoship apply once, then check that the cluster came up. It proved that apply works end to end and nothing else. When a phase left the wrong artifact behind but the install still succeeded – a /etc/hosts entry written with the wrong address that DNS happened to cover, a lock file that was never cleaned up, a firewall rule that named the wrong peer – the suite stayed green. When it did fail, the failure was “the cluster is not healthy” and the phase responsible had to be found by reading the log.
The suite now drives the apply phase list in-process, one phase at a time, and asserts what each phase left on the hosts before the next one runs. This document records the choices that shape it, because several of them look arbitrary in the code and each had an alternative worth stating.
Replacing the CLI apply rather than adding to it
The obvious way to add phase-level coverage is to leave the existing cargoship apply test alone and add a second suite beside it. That was rejected: a bootloose cluster is the expensive part of the run, and installing rke2 onto it twice costs several minutes and a lot of memory for coverage that overlaps almost entirely.
So the phase walk is the install. Test_00_CreatePackage builds the package and Test_01_Prepare readies the hosts, the phase tests then perform the install one phase at a time, and a whole apply survives only as Test_ZZ2_ApplyIsIdempotent, which runs after the cluster is already up. That step now earns its place twice over: it is the idempotency check and the only test that exercises action.NewApply’s own wiring – the phase list, the flag plumbing, the ordering – which the phase tests deliberately bypass.
None of these steps shell out. Test_00_CreatePackage calls distro.Create, and the four whole-action steps call action.NewPrepare, action.NewApply, action.NewReset and action.NewKubeConfig, each with its own manager built the way the matching command builds one and its own temp directory to remove. The suite was written against the CLI first, and the binary turned out to be doing nothing the packages could not: it cost a build step in CI, a rebuild-before-you-run rule, and a class of confusing failure where a stale binary disagreed with the source being read. What is lost is coverage of the cobra layer – flag parsing, --confirm, the config file resolution – which test/e2e/noncluster covers against the real binary and does so in seconds.
The cost is that the phase tests do not test action.NewApply. If a phase were dropped from the list in pkg/action/apply.go, the phase tests would keep passing, because they build their own list. Test_ZZ2 is what catches that, and the ordering comment in the suite doc is what keeps the two lists in sync by hand. A generated list, read from action and iterated, was considered and rejected: it would make the assertions dynamic, and the whole point is that each phase has a hand-written assertion about the artifact it leaves.
One number per phase, taken from pkg/phase
Testify runs suite methods in lexicographic order of the method name, which is the only ordering mechanism available. So the method name has to carry a number, and there were two candidates: the phase’s position in the apply order, or the number of its source file.
The suite first used the apply order, on the grounds that the point of the suite is to reproduce what apply does. It made the test for phase/25_modify_hosts_file.go a method called Test_12_ModifyHosts living in 25_modify_hosts_file_test.go, and every file in the package carried two numbers that disagreed. Inserting a phase anywhere but the end renumbered every method after it, which is a large diff that says nothing, and cross-references between test comments had to be renumbered with it.
The numbers are now the source file’s, in both places: 25_modify_hosts_file_test.go contains Test_25_ModifyHosts. One number per phase, so the test for a phase is findable from the phase and the ordering needs no separate bookkeeping. Adding a phase adds a file and a method and renumbers nothing.
pkg/phase numbers its files by rough category – 20-26 prepare the host, 50-59 upload, 60-72 install and sync, 80-81 finish, 91-92 lock and unlock – and that ordering happens to agree with apply’s for every phase but one. Apply takes the lock third, right after OS detection, so that it is held for the whole run; the file number puts it after the install instead, with unlock immediately behind it.
That is the cost, and it is worth naming precisely. Test_91_Lock still asserts the thing the phase is responsible for: a lock file on every host holding this process’s instance ID, removed again by Test_92_Unlock. What is no longer covered is the lock being held while the other phases run, so a phase that misbehaved against a locked cluster would not be caught here. Nothing in the current phase list reads the lock, and Test_ZZ2_ApplyIsIdempotent runs a real apply, which takes the lock in its proper place. Special-casing the lock – running it early under a method name that does not match its file – was considered and rejected: it reintroduces the two-number problem for one phase, which is the shape of thing that is forgotten and then misread.
Steps that are not phases have no source file to take a number from. They use the ends of the ordering instead: Test_00_CreatePackage and Test_01_Prepare sort before every phase number, Test_ZZ1 and Test_ZZ2 sort after every phase number because a letter sorts after a digit.
One shared harness, and tests that cannot run alone
Phases communicate through the hosts. DetectOS resolves a Configurer and hangs it on the host; GatherFacts fills in Metadata; Connect leaves a live SSH connection that every later phase reuses; the upload phases record Metadata.Install for the initialize phases to call. Running a phase in isolation means reconstructing all of that, which means reimplementing its predecessors.
So there is one phaseHarness for the whole suite, built once in Test_05_Manager, and the phase tests are ordered and stateful by design. go test -run 'TestClusterPhases/apply/Test_25_ModifyHosts' does not work and is not meant to: the hosts were never connected. This is stated in the suite doc comment because it is the first thing someone debugging a failure will try.
The harness builds its manager from exactly the inputs action.NewApply uses – load.ClusterDefinition for the inventory, distro.Load for the package, registry.GetDistroModuleBuilder for the distro module – so the phases see the state they would see under a real apply. phase.Manager.Run is called once per phase with a single-element Phases, which is safe because Run calls defaults.Set each time and that is idempotent.
The assertion bodies live on phaseWalk, an unexported type ApplyPhaseSuite embeds, with a one-line Test_NN method on the suite calling each one. For a single walk that is more indirection than it needs; it is there because further walks over the same phase list – an upgrade against the installed cluster, a join of a new node – assert the same things and would otherwise have to copy the bodies and keep them in step by hand. Testify collects methods by their Test prefix, so an unexported method on the embedded type is shared rather than run as a test of its own.
Asserting the artifact, not the return code
A phase that returns nil has not necessarily done anything. Every assertion in the suite reads the host back over SSH and checks the thing an operator would check:
- every host’s detected
os-releaseID matching the image its machine was built from, and every family the bootloose config provisions being present - the rendered
/etc/sysctl.d/99-cargoship.confnaming every setting the package carries, at the value it carries - every peer’s private address, long hostname and short hostname in every host’s
/etc/hosts, plusgetent hosts <peer>resolving to the right address – the full matrix, because a broken loop that writes only the first peer passes any spot check - the firewall’s own dump (
firewall-cmd --list-all,ufw status verbose,nft list ruleset) naming each peer, rather than the phase’s idea of what it wrote - every path the upload manifest claims present on the host, or staged beside it under the temporary name the install hook will move
- the engine config on disk containing the host’s name and data directory
- the service actually active and
RunningVersionmatching the package’s version - the lock file’s content on every host, matching
hostname-pid, not merely its existence - the kubeconfig parsed with
clientcmd, checking the context name, the server URL and that the embedded CA and client material are non-empty
Reading through SSH rather than through the phase’s own state is deliberate: a phase that records success in Metadata without touching the host would otherwise pass.
The sysctl test is the one place where the artifact is the file and not the effect. The nodes are containers sharing the host kernel, and the settings the phase writes are not namespaced, so sysctl -n there reports whatever the machine running the tests is set to: asserting on it would fail a correct phase and pass a broken one whenever the host already matched. The phase still runs sysctl --system itself, and a failure there fails the phase.
Phases that legitimately do nothing assert the gate
Several phases are expected to skip on this cluster: SELinux and fapolicyd where neither is active, and later the config-sync phases when nothing has drifted. Asserting nothing there would be a test that cannot fail.
Instead those tests compute what the hosts report – how many have SELinux enabled, how many run fapolicyd, whether the package carries a rule set – and assert that ran(p) matches. That tests the routing logic, which is the part most likely to break, and it keeps working when the cluster changes: adding an image with SELinux enabled turns the same assertion into a check that the phase fires and installs container-selinux.
ran(p) type-asserts interface{ ShouldRun() bool } and is only meaningful after harness.run, because the manager calls Prepare first and most gates are computed there. The manager’s own list of executed phases is unexported, which is why the gate is re-derived rather than read back.
Stopping after the first failure
Twenty-odd ordered, stateful tests mean one broken early phase produces twenty downstream failures that say nothing. TearDownTest records the failure and SetupTest skips the rest, so the output names one phase. Teardown of the containers is unaffected – TestMain deletes the bootloose cluster regardless – so this costs nothing but the coverage that was already lost.
Nine nodes across two OS families
The cluster was six usable nodes on one Ubuntu image (with three further machines provisioned and never wired into the inventory). It is now nine: three controllers and six workers, each role split between Ubuntu and Fedora.
The OS mix is the substantive part. The apply phase list routes on OS family in at least five places – the RPM and APT upload phases, SELinux, fapolicyd, and the firewall backend – and on a single-family cluster the branch that does not match is never taken. Worse, the gate assertions described above pass in that case, so the suite reports coverage it does not have. With both families present, Test_57_RPMUploadFiles and Test_58_APTUploadFiles each assert a real split: the family they claim keeps what was staged, the family they do not comes back byte-identical.
To keep that from silently regressing, Test_09_DetectOS asserts that every family the bootloose config provisions is present, and that each host’s detected os-release ID matches the image its machine was built from. If someone collapses the cluster back to one image, the failure names that cause rather than showing up as a phase test that quietly stopped covering a branch.
Nine nodes rather than six also puts more than one batch through the worker concurrency path: the initialize phases batch workers at WorkerConcurrent, which the suite sets to 5, and six workers is the first count that makes that a real batching decision.
Both images are the project’s own: ghcr.io/colonel-byte/bootloose/ubuntu-26 and ghcr.io/colonel-byte/bootloose/fedora-44. The Fedora replicas are named kcf* and kwf* so that the inventory’s existing kc/kw prefix mapping claims them without the OS family entering the mapping at all.
A tenth machine that receives uploads and never joins
Two of the three upload phases are reached by an OS-family filter. The third, phase/59_bin_install.go, is what a host gets when neither of the other two claims it – which on an all-Ubuntu, all-Fedora cluster meant every host, because rke2 ships as neither an RPM nor a deb. So the phase was being exercised as the path every host happens to take, not as the fallback it is written to be, and nothing in the suite would have noticed if the two family filters had started claiming hosts they should not.
The cluster therefore also provisions one ghcr.io/colonel-byte/bootloose/alpine-3.23 machine. Alpine is neither Enterprise Linux nor Debian, so utils.FilterEnterpriseLinux and utils.FilterDebianLinux both decline it and it reaches the BIN phase as the only host that had nowhere else to go. Test_59_BINUploadFiles asserts against it by name.
It cannot run the engine: rke2 links against glibc and Alpine is musl. Three things follow from that, and each had an alternative.
The host is a worker in the inventory, not a role of its own. The upload phases select by role, so a host with no role would not have been claimed and the whole point would be lost. It is named kwa0, which the inventory’s kw prefix maps to the worker role with no special case, and uploadOnlyPrefix in cluster_inventory.go is what marks it as more than a worker. The name template in main_test.go is built from that same constant so the two cannot drift.
It is removed from the manager after the last upload phase, rather than skipped by each later test. Test_60_ConfigureEngine calls harness.dropUploadOnlyHosts() before it runs anything. Skipping was tried first and is not sufficient: InitializeWorkers.Prepare claims every non-controller host whose agent is not already running, with no OS gate at all, so a test that merely declined to assert on the Alpine host would still have watched the phase spend its retry budget trying to install rke2 on it. Taking it off manager.Config.Spec.Hosts is the only thing that stops the phase from selecting it, and the harness keeps what it dropped so that close can disconnect it, since the Disconnect phase only sees the hosts still on the manager.
There are two generated inventory files. generated-cluster-full.yaml has all ten machines and is what the harness is built from; generated-cluster.yaml has the nine that join and is what e2e.ClusterConfigPath points at. The whole-action steps – Test_01_Prepare, Test_ZZ2_ApplyIsIdempotent, and ResetSuite’s two – load that path, because an action has no notion of an upload-only host and would try to install the engine on Alpine. A single file with an annotation the loader ignores was considered and rejected: it would put a test-only concept into the inventory schema.
Four walks against one cluster, in one parent test
The suite runs four suites rather than one: ApplyPhaseSuite installs the distro, JoinPhaseSuite adds a machine to what it installed, UpgradePhaseSuite walks the same phases again with a newer package, and ResetSuite takes the distro back off. They share one bootloose cluster and one kubeconfig, and they only work in that order.
Go runs top-level test functions in the order they are declared across the package’s files, sorted by file name – which is an ordering nobody declared and that a renamed file silently changes. So there is one top-level test, TestClusterPhases in main_test.go, and the four walks are t.Run subtests of it. The apply subtest’s result is checked: if the install failed there is no cluster for the other three to walk, and the run returns rather than reporting four failures for one cause. The join and upgrade results are not checked, because a half-joined or half-upgraded cluster is still something reset has to be able to tear down.
The join walk runs before the upgrade rather than after it, so that the upgrade has to carry the node that joined late as well as the nodes the install bootstrapped.
That ordering is also why reset is not Test_ZZ3/Test_ZZ4 on the apply suite. Reset has to run last and the other two walks have to run in between, so the two steps moved into ResetSuite. It loads no package – reset and kube-config both build a bare manager – so it needs neither a harness nor a package path, only the distroID constant.
kubeconfigPath moved to TestMain for the same reason. Three of the walks touch the same file: the apply walk writes it, the upgrade walk rewrites it against the upgraded control plane, and the reset walk asserts it survives a teardown that can no longer reach a controller. A suite-owned t.TempDir() would have been deleted between them.
What the join walk covers that the install cannot
Apply against a cluster that already exists routes differently from apply against bare machines, and the install walk cannot exercise that: on a fresh install every host is new. The join walk is what does. It starts one more machine, rewrites both inventories to name it, and walks the same phase list carrying the same package at the same version.
The new machine is a Fedora worker, kwf2. A third replica of an existing template rather than a template of its own, because bootloose names a machine by formatting its template’s name with the replica index and a second template would start counting from zero and collide with kwf0. Fedora rather than Ubuntu so that the node has to come through the SELinux, fapolicyd, firewalld and dnf branches on its own, rather than inheriting anything the apply walk already proved on the nodes beside it. A worker rather than a controller because adding a controller to a running control plane is a different operation with a different failure mode, and the phase list treats it as one.
It runs at the installed version, not a newer one. A join is not an upgrade. Carrying the same package is what makes the established nodes report the packaged version, so the upgrade phases claim nothing and the initialize phases claim exactly one host – which is the routing the walk exists to assert. Test_62 requires that the new machine and only the new machine was claimed.
The phases that assert over the whole matrix earn their place twice. Test_25 and Test_26 assert every peer’s entry on every host, so on the join walk they are what proves the established nodes learned about the new one. A join that configured only the joining node would pass any assertion written from the new node’s point of view.
The lock phases are left out, as they are asserted in the apply walk and holding a lock across a second walk against the same hosts tests the lock file rather than the join.
The machine is provisioned on first use, not in TestMain. A run that never reaches the join walk starts one fewer container. The new cluster object replaces testCluster, because bootloose finds a cluster’s machines by walking the config it was built from rather than by asking Docker: the object built from the five-machine config would leave the sixth container behind on shutdown.
Walking the upgrade, and where it starts
On a fresh install phase/66_upgrade_controller.go and phase/67_upgrade_worker.go claim no hosts, and the config-sync phases find no drift. The apply walk asserts exactly that – correctly, it is the routing that matters – but it means the code inside those phases was never executed by any test. Draining a node, stopping the service, running the install hook, waiting for Ready and uncordoning is the most destructive sequence in the phase list and the only one with no coverage at all.
UpgradePhaseSuite closes that. It builds a second package from exampleUpgradeDefinitions – example/rke2-multi-cni-cilium/v1_37/v1.37.0-rke2r1 for rke2, example/k3s-flannel/v1_37/v1.37.0-k3s1 for k3s – and walks the phases against the cluster the walks before it left running. Both maps are keyed by distroID(), so CARGOSHIP_E2E_DISTRO picks the install target and the upgrade target together.
The target is the next minor line. The install walk takes v1.36.4, which is already the newest patch in its own minor, so there is no next patch to upgrade to and the smallest real upgrade the examples offer is v1.37.0. That makes the walk a minor-line upgrade: it moves the engine tarball, the RPM URLs and the image tags, and it can move the bundled CNI as well, so a failure here has more places to come from than a patch bump would. The upgrade phases route on VersionLess, so what matters to every phase under test is only that the packaged version compares greater than the installed one.
This was a patch-revision upgrade within v1_35 when the suite was written, chosen so the walk moved nothing but the engine. It is a minor-line upgrade now because the examples moved on; if the version floor rises again and a newer patch appears inside the installed minor, prefer it.
It starts at GatherFactsDistro, not at Connect. Connect, DetectOS, GatherFacts and ValidateHosts do the same work against the same hosts they did during the install, and the apply walk already asserts each of them one phase at a time. Repeating that costs runtime and adds no assertion that could fail differently, so they run as a single Test_01_Reconnect step whose only job is to hand the rest of the walk a connected inventory. From Test_12_GatherFactsDistro on, every phase gets its own method, because from there on every phase behaves differently than it did on the install.
The lock phases are left out. They are asserted in the apply walk, and taking the lock again for a third walk against the same hosts would be testing the lock file, not the upgrade.
The assertions that differ are the point. Test_12 is where the divergence starts: the hosts now report the installed version, and the test asserts it is VersionLess than the packaged one by calling the same comparison the phases’ own Prepare calls, so the test cannot disagree with the routing it is asserting. Test_61 and Test_62 must claim nothing, which is the inverse of the apply walk’s assertion and the check that an initialize phase never re-bootstraps a live node. Test_66 and Test_67 are the only place the upgrade phases are asserted to have run, and they check all three outcomes of the sequence: the service came back up, RunningVersion is now the packaged version, and the node is not left cordoned. That last one is worth having on its own – a node left Unschedulable looks healthy from the host side, service running and version correct, and only shows up later as a cluster that will not schedule.
Test_00_UpgradePackage also requires the host count the join walk left behind rather than the one the install started with. An upgrade that saw only the machines the install saw would leave the node that joined last on the old engine, and nothing else in the walk would notice.
The upgrade is opt-in
The upgrade walk runs only when CARGOSHIP_E2E_UPGRADE is set. Like the rest of the engine-bootstrap half, it does not run in CI at all – see choice-e2e-stage-split – so the variable is a local-run switch, set by mage test:endToEndClusterUpgrade, rather than something a workflow input or a pull request label ever needs to thread through.
The install walk already installs and starts k3s on five nodes; the upgrade imports a second complete set of engine images into every containerd store on top of that, so it roughly doubles the disk and adds tens of minutes. Making it unconditional would mean every run of mage test:endToEndCluster pays for it, including the runs that only want to know the install still works. A single environment variable, read in SetupSuite, is checked in one place and skips with a message naming the variable.
Profiles on the generated inventory
The generated bootloose inventory set no Profile, which makes LabelNodes a no-op: it iterates hosts, skips those with an empty profile, and would have labelled nothing. Each host now gets a profile equal to its role.
This was checked against every other consumer of Profile before making the change. GatherFacts.setupProfileOverrides looks the profile up in Spec.Config.Profiles and does nothing when it is absent. The per-profile concurrency grouping resolves through Profiles.ResolveConcurrency, which falls back to the caller’s value for an unmapped profile. The file selectors in 55_files_common.go compare against the role constants passed by the RPM/APT/BIN phases, never against h.Profile. So the profile is inert everywhere except the phase under test, which is what makes it safe to set purely to make that phase observable.
Why the firewall backend is chosen by the node’s OS, not by what is running
Cargoship’s firewall support (cargoship apply --firewall) renders a backend-neutral firewall.Plan onto whichever firewall a node runs. Three backends exist: firewalld, ufw, and nftables. Selection lives in firewall.Select (pkg/firewall/firewall.go).
The first implementation matched a host against the backends in a fixed order – firewalld, then ufw, then nftables – taking the first whose Detect was true. Ordering firewalld and ufw ahead of nftables was already deliberate, since both are front ends onto nftables and a host running either would match the nftables backend as well. What that ordering could not express is the case the current implementation is about: a node whose front end is installed but stopped.
The OS decides, and Detect only confirms
Each OS module now names the front end its distribution ships, through PreferredFirewall on the Configurer interface (types/os/interface.go): firewalld on Enterprise Linux and SUSE, ufw on Debian and Ubuntu, nothing on Alpine, Arch, CoreOS, Flatcar, and Slackware. Select looks for that backend first and uses it when it is running. Only a node whose preferred front end is absent, or whose distribution ships none, falls through to the ordered Detect match.
The alternative was to keep selection entirely inside the firewall package and infer the front end from what Detect finds. That is what the ordered match already did, and it cannot distinguish “this distribution has no front end” from “this distribution’s front end is not up”. Cargoship already resolves an OS module per host and already asks it distribution-specific questions, so the distribution’s own answer is both available and more trustworthy than a probe.
An inventory field to override the choice was considered and rejected. It would be a second source of truth for something the OS already answers, and the failure mode – an operator naming a backend the node does not run – produces a worse outcome than any case it would fix.
An installed but stopped front end means cargoship configures nothing
When the preferred backend is installed and not running, Select returns it as Selection.Skipped and cargoship makes no change to that node. Two other options were available and both were rejected:
- Start the service. Bringing up a firewall from a remote phase is the same hazard that stops cargoship from running
ufw enable: a default-deny policy applied over the SSH connection cargoship is running on can cut it, leaving the node unreachable with the phase half-finished. - Fall through to nftables. The front end owns the nftables ruleset on that node. Rules written underneath a stopped firewalld or ufw would be overwritten the moment the operator started it, so the node would silently lose the configuration cargoship reported as applied.
An operator who installed a front end and left it down has made a decision about the node’s firewall posture. Cargoship logs the node it skipped and leaves that decision with them.
The cost is that a stock Ubuntu node, which ships ufw installed and inactive, gets no firewall configuration at all. That is a real change in behaviour: the ordered match would have configured such a node through the nftables backend if nft was present and the nftables service or an /etc/nftables.conf existed. The rules it wrote there were the ones ufw would have discarded on its next start, so what is lost is a configuration that was never durable.
Installed is a separate method rather than a mode of Detect
Detect answers “does this backend manage the firewall on this node”, which requires the firewall to be running – the nftables backend deliberately reads it as “the operator configured a persistent ruleset”, not “nft is present”. Overloading it would have changed what every existing caller means. Installed is a plain presence check (firewall-cmd, ufw, nft on PATH), and only the skip decision uses it.
PreferredFirewall is declared on each leaf OS module
The obvious place for the method is a shared base: EnterpriseLinux for the seven Enterprise Linux modules, Debian for Ubuntu, SLES for OpenSUSE. It works for the second and third, whose modules embed only their base, but not for the first. RockyLinux and its siblings embed both linux.EnterpriseLinux and configurer.Linux, so a method on EnterpriseLinux and the default on configurer.Linux are promoted at the same depth, and Go leaves the selector ambiguous – the modules stop satisfying Configurer. Declaring the method on each leaf keeps it unambiguous, at the price of seven one-line methods.
Why apply refuses a removed host instead of reconciling it
Deleting a host block from a cargoship config does not remove that machine from the cluster. Apply detects the leftover node and stops (pkg/phase/13_detect_removed_hosts.go); it does not drain it, delete it, or uninstall its engine. Reconciling the removal was the obvious alternative, and it is what every other declarative tool in this space does, so the choice needs an argument.
Apply cannot finish the job
Removing a node is two pieces of work. Deleting the Node object needs the node’s name, which the API server will give us. Uninstalling the engine – stopping the service, removing the packages, deleting the data and config directories – needs an SSH connection, and the address, user and key for that lived in the host block that was just deleted. Apply holds no memory of previous runs, so it has no way to reach a machine the config no longer describes. UninstallEngine is not hard to call here; it is impossible to call, because there is nothing to connect to.
Doing only the first half is worse than doing nothing. A node deleted from the API but left running keeps its certificates and keeps trying to rejoin, and for a controller kubectl delete node does not remove the etcd member: the cluster looks smaller while quorum is still counted against a machine nobody can reach. The failure mode of stopping is a visible leftover node and a message naming it. The failure mode of doing half is an apply that reports success and a cluster that loses quorum on the next controller restart.
Absence is the wrong trigger for a destructive operation
Adding a host that is already there is idempotent. Removing a host that should have stayed is not. A typo in a hostname, a bad merge, a templating bug that drops a list entry – under reconcile each of those becomes a drained and destroyed node, and under refuse each becomes a stopped apply with a message. The blast radius of a false positive is not symmetric, so the default should not be either.
The old state is not a substitute for a connection
The tempting way to supply the missing connection details is to take them from the provider’s prior state and generate a small config to run reset against. It should not be built.
A state file records what was true after the last successful apply, and reset is destructive. Between then and now the machine may have been rebuilt, re-addressed, or returned to a DHCP pool, and none of that produces a diff the provider would see, because the resource is being removed and there is nothing left to refresh against. The address is the least stable identifier a host has and the one reset would connect through, so the target becomes whatever machine currently answers there. Other kinds of state drift surface as a plan diff somebody reads; this one surfaces as a wiped machine.
The generated config would also splice two points in time. The host comes from the old state, but the paths an uninstall removes come from the distro module: CleanupPaths() returns the data and config directories as the current configuration defines them. Upgrade the cluster or move the data directory and reset cleans today’s paths on a machine running yesterday’s engine, leaving the real files behind and removing whatever sits at the new ones.
Making removals depend on state would also settle the separate question of keeping SSH key material out of tofu state (#309) the wrong way, by making it unsolvable.
The phases cannot express a subset anyway
Independent of where the contents come from, there is no config that makes reset remove one host from a live cluster. DeleteCommon.Prepare finds its leader by scanning Spec.Hosts for a host already running the controller service, so a config holding only the removed host has no leader: both delete phases report ShouldRun() == false and the node is never deleted, while UninstallEngine runs anyway. Adding a live controller so a leader can be found is worse, because UninstallEngine.Prepare sets p.hosts = p.manager.Config.Spec.Hosts with no filter and uninstalls the engine from it too.
Neither is a bug in reset. The unfiltered list is exactly right when the instruction is “tear down this whole cluster”. The gap is that no action separates the hosts it targets from the hosts that supply cluster access, which is tracked as #339.
What the check actually does
DetectRemovedHosts runs after fact gathering and before anything is changed, lists the nodes joined to the cluster, and stops the apply when the cluster holds a node no host accounts for, naming each one with its role.
The list is read by running kubectl get nodes -o json on the leader over the existing SSH connection, which is how every other cluster command in the phases works. LabelNodes is the exception: it builds a Kubernetes client dialled at https://<loadBalancer>:6443 from the machine running cargoship. That was not copied here. A cluster whose firewall admits only SSH from outside is an ordinary deployment for cargoship, and a check that runs on every apply must not be the one thing that needs the API port open. Going through the leader also drops the need for the load balancer address and for reading the admin certificates.
Two guards keep it from becoming a nuisance, and both exist because a false positive would stop an apply on a working cluster:
- Nodes are matched to hosts by any shared identifier: the node’s name or any address it reports, against the host’s hostname, private address, or connection address. Names alone are not enough, because an operator can set
node-namein the engine config, and matching on names would then report every node in the cluster as removed. - Only positive evidence stops the run. Failing to reach the cluster says nothing about whether a host was removed, so a failure to list the nodes warns and continues.
The phase is read-only, so a dry run executes it. That is the run where being told about a host you deleted from the config costs nothing.
--allow-unmanaged-nodes downgrades the refusal to a warning, for clusters holding nodes cargoship never joined. Cargoship leaves those nodes alone either way; the flag only chooses whether their presence is an error or a log line.
What would justify revisiting
Not a state file, and not inferring removal from absence. The variant that survives every argument above is a tombstone: the host stays in the config for one apply, marked for removal, so the connection details are present and current rather than remembered, and the operator is editing a file that says in the present tense which machine is to be destroyed. OpenTofu’s own removed block is precedent for stating a removal explicitly rather than inferring it.
What that needs from the phases is exactly what #339 builds – act on this subset, use those other hosts for access – and a subset reset does both halves on the target, so the orphaned etcd member stops being a hazard. If #339 lands, the remaining work is a per-host config marker, a quorum guard before a controller is removed, and a decision on when the marker is cleaned up. This check stays either way, as the backstop for a host deleted outright rather than tombstoned.
Why a targeted reset names its hosts separately from the hosts that provide access
cargoship reset --target-hosts removes named hosts from a live cluster and leaves the rest alone. The gap it closes is recorded in #339: reset was all-or-nothing by construction, and the two shapes available to an operator who wanted one node gone were both wrong.
A configuration holding only the host being removed has no controller, so DeleteCommon.Prepare finds no leader, DeleteWorkers and DeleteControllers both report ShouldRun() == false, and the node is never deleted from the cluster – while UninstallEngine runs regardless and takes the engine off anyway. That is an orphaned NotReady node object. Adding a live controller to supply the leader fixes the deletion and uninstalls the engine from the controller too, because UninstallEngine.Prepare set p.hosts to the whole host list with no filter. That is a destroyed control plane.
Neither was a defect. The unfiltered list is exactly right when the instruction is “tear this cluster down”, which is what reset had always meant. What was missing was a way to say which hosts an action acts on, separately from which hosts it reaches the cluster through.
Two host lists, not one
TargetHosts is that second list. The delete phases and UninstallEngine act on it; leader selection keeps scanning the full configuration. So a controller can supply cluster access without being torn down, which is the one thing neither earlier shape could express. An empty target list means every host, so reset with no flag behaves exactly as it did.
Hosts are matched generously – hostname, private address, or connection address – for the same reason the removed-host check in apply is generous: an operator naming a host by the address they SSH to it on, when the configuration calls it by hostname, has not made a mistake worth failing over.
Three refusals, each replacing a silent half-change
The natural behaviour in each of these cases is a no-op or a partial teardown that reports success, which is worse than stopping.
| Refused | What would otherwise happen |
|---|---|
A --target-hosts entry matching no host | Every phase filters down to nothing and the reset reports success having removed nothing. A typo is the ordinary route into this, so it is checked in NewReset – before a single connection is opened – rather than in a phase |
| Every running controller targeted, some host kept | Nodes drained and deleted through a controller the same run is about to uninstall, with the outcome depending on the order the hosts happened to be in. Removing all of them is a teardown, and plain reset is what does that |
| More than one running controller targeted in one run | The second removal starts from a membership with a member already stopped, and hangs with the cluster unavailable. One run per controller |
The second and third are about etcd membership rather than about cargoship, which is why they are refusals and not retries: there is no ordering of the work that makes either safe.
The engine stops before the node is deleted, except once
k3s and RKE2 drop the etcd member when the Node object is deleted, but only once the engine behind it has stopped – an engine still running re-registers its node seconds later and the deletion does not stick. So the engine comes down first.
The exception is a removal that leaves a single controller behind. There the member being removed has to still be up to form the majority of two that commits its own removal, so the engine stays running through the deletion and is uninstalled immediately after. keepEngineForDelete is that case, and it is the reason the ordering is decided in Prepare from the controller membership rather than being fixed.
What is not covered
The refusals count controllers whose engine answers, not etcd’s own membership, so a cluster that is already one member down looks whole to them. A quorum check against etcd would be the stronger test and is not here.
Choice: what a run receipt records, and what it refuses to
cargoship apply --receipt-path writes a structured record of a run, and cargoship redact-receipt projects it into something safe to attach to a public issue. Five decisions in that design look like oversights from the code alone. They are not.
Host records are an allowlist, never a denylist
internal/receipt.Host copies in a fixed set of fields by name, and nothing anywhere marshals cluster.ZarfHost into a receipt.
That looks like needless transcription – the host object already has every field, and a json:"-" tag on the ones that must not appear would be less code. It is deliberate. ZarfHost embeds a live rig.ClientWithConfig and a Metadata struct full of connection state, and a receipt built by excluding what must not appear leaks the next field somebody adds to either. An allowlist fails closed: a new fact is absent from receipts until a person adds it to Host, which is the safe direction to be wrong in.
TestHostFieldsAreAnAllowlist pins the field set so that adding one fails the build until it is classified. The precedent is the OpenTofu provider’s factsOf, which does the same thing for the provider’s attributes and for the same reason; it moved to colonel-byte/terraform-provider-cargoship.
If you are adding a host fact to a receipt, add it to Host and update that test. Do not reach for reflection over ZarfHost.
Diffs exist only where a phase already read the file
Four phases carry a real unified diff – 25_modify_hosts_file, 24_prepare_selinux_policy, 70_engine_config_sync_common and 89_reconfigure_survivors. Four do not: 20_prepare_host (sysctl), 22_prepare_fapolicyd, 23_prepare_kernel_modules, and the engine config.yaml written through types/distrocfg.
The split is not an unfinished job. It is exactly the line between phases that fetch the prior content anyway and phases that do not. driftedFiles in the sync phase holds the host’s copy and the desired copy in the same scope, so a diff there costs nothing. The other four write without reading, and ZarfHost.FileChanged compares by sha256 without ever pulling the remote file down – so adding a diff to them means a new ReadFile per file per host, every run, on every host in the fleet. On a large fleet that is a real cost paid by every operator to benefit the few who read a diff.
Those phases still report that they changed a host, with the file path and a reason. Only the body is missing. HostChangeDetail.Diff is documented to mean “not compared” when empty, never “no difference”.
If you add a diff to one of the blind-write phases, that is a deliberate trade, not a cleanup – say so, and measure it against a fleet rather than a two-node test cluster.
Redaction happens at share time, not capture time
There is no --receipt-redact flag, and --receipt-path never writes a second, safe file beside the real one.
Both were considered. A pair of files one infix apart – apply.json and apply.redacted.json, one carrying join tokens and one not – is how the wrong one ends up attached to a public issue, and the failure is silent and unrecoverable once it is there. Redaction is also lossy and one way: the operator’s copy is the audit record and has to stay complete, and a reporter must not have to re-run a failed apply to get something shareable.
So it is a post-processor. That also means it works on receipts already on disk, and can be tightened later without anybody re-running anything.
Hosts are pseudonymised, not hashed
Redact replaces each address with host-01, host-02 in the order the receipt lists them.
A digest of the real address would be the obvious choice and is the wrong one. Hostnames and RFC-1918 addresses come from a small enough space to enumerate exhaustively, so a hash of one is recoverable by anybody willing to spend an afternoon on it – while looking like a one-way function to everyone who reads the code. A sequential counter carries no information at all.
The numbering follows the cluster configuration rather than the order phases reached hosts, so the alias a host gets in the host list is the alias it gets inside every phase, and a maintainer can still follow one machine through the ledger.
The receipt path is templated, and only two placeholders exist
expandReceiptPath substitutes {command} and {timestamp} into a receipt path, and rejects anything else. A flag value that is just a filename would be simpler, and for a flag it would be right.
The templating exists for the config file, not the flag. --receipt-path names one file for one run; .distro.receipt_path is read by every apply, prepare, reset and engine-config-sync the operator ever runs, so a literal path configured there has each run truncate the record of the one before it. That failure is silent and leaves exactly one receipt on disk – the audit trail an operator thought they had turned on is instead a single file whose contents are whichever run happened last. The two placeholders together make the name unique per run, which is what somebody setting the key meant by setting it.
An unrecognised placeholder is an error returned from startReceipt, before the run touches a host, rather than a literal written into the filename. Both halves are deliberate: a run that succeeds and records itself in apply-{date}.json is not discovered until somebody goes looking for a receipt that was written somewhere nobody expects, and failing after a fleet-wide change – when the run is cheap to repeat before it, not after – would be the wrong end to fail at.
There is deliberately no {phase} alias for {command}. “Phase” in this repository means a numbered step inside a run (25_modify_hosts_file and its siblings), and a receipt spans all of them, so a {phase} that expanded to apply would name one thing after another. Adding more placeholders is cheap – a map entry in expandReceiptPath – but each one is a name an operator has to learn, so it should answer a question the two existing ones cannot.
Why a package’s SELinux policy is CIL, applied fatally, and removed from a record on the host
spec.config.os.selinux lets a Cargo package carry the SELinux policy its workload needs: CIL policy modules, booleans, and file context mappings. pkg/phase/24_prepare_selinux_policy.go applies it, and pkg/phase/89_remove_selinux_policy.go takes it back off. Four decisions in there are not recoverable from the code, and this is the record of them.
CIL, not Type Enforcement source and not a compiled module
A policy can reach a host in three shapes, and only one of them fits a management node driving an air-gapped fleet.
Type Enforcement source – the familiar .te and .fc pair – has to be compiled on the host, with checkmodule and semodule_package out of policycoreutils-devel and selinux-policy-devel. Neither is on a minimal Enterprise Linux install, and neither is staged inside the package. Cargoship installs nothing on a host beyond what a phase uploads, so a feature whose first requirement is two development packages from a repository the host cannot reach is a feature that does not work on the fleet it was written for.
A compiled .pp module avoids the toolchain by moving the build off-host, and pays for it in portability: a binary module is bound to the policy version it was built against, so a package carrying one works on the hosts whose policy matches and fails on the rest. For a package that is built once and applied to whatever an operator has, that is the wrong trade.
CIL is what semodule -i accepts directly. No build tooling, no version binding, and it is plain text, so it lives in distro.yaml the way the fapolicyd rules already do and shows up in a review as the policy it is. The cost is that CIL is lower-level than .te and less pleasant to write by hand. That cost falls on the few packages that need a module at all, and booleans and file contexts – which need no policy language whatsoever – cover the common cases without it.
Applying is fatal, removing is tolerant
The rule set in choice-unpinnable-hosts.md is prerequisite against convenience: a failure that leaves the install impossible stops the run, and a failure that leaves it merely weaker warns. A policy written into distro.yaml by hand is the first kind. Nobody adds a CIL module or a container_var_lib_t mapping speculatively; it is there because the workload is denied without it. A host that quietly skipped it is an enforcing host running a workload whose policy is absent, and the operator finds out from an AVC denial somewhere downstream rather than from the phase that could have told them. So semodule, setsebool and semanage failures return, and the phase stops the run.
A host that cannot take a policy at all is a different question, and is filtered rather than failed – selinuxPolicyCapable requires both SELinux enforcing and semodule present, the same shape as PrepareFapolicy filtering on the service running. A Debian host with no SELinux is not a failure to apply a policy; it is a host the policy was never about.
Removal inverts both. RemoveSelinuxPolicy warns and carries on through every step. A reset has to be re-runnable, and the state it is undoing can legitimately be half gone – a module an operator removed by hand, a context already deleted, a boolean already back at its default. Returning an error on the first absent thing would strand the host with the state file still claiming the policy is installed, which is worse than the inconsistency it was reporting.
Reset reads a record on the host, because it has no package to read
A reset loads no distro package. cmd/install_reset.go builds its phase.Manager by hand precisely because there is nothing to load, and Manager.Distro is nil for the whole run. So the removal phase cannot ask spec.config.os.selinux what to remove: whatever it undoes, it has to learn from the host.
That is what /var/lib/cargoship/selinux/cargoship-state.json is for. The apply phase writes it before it changes anything, recording the modules and their priorities, the file context mappings, and – the part that cannot be reconstructed any other way – the value each boolean held beforehand. getsebool after the fact reports cargoship’s own value, so without the record “unwind” could only mean guessing at a distribution default, and guessing wrong on a boolean is a security change made silently in the wrong direction. A boolean already present in the record keeps the value written the first time, so a second apply does not overwrite the host’s original with cargoship’s.
A host with no record is skipped. It has nothing cargoship applied, or it has a policy applied by something else, and neither is the removal phase’s to touch.
No command-line flag
The phase gates on two things only: what the hosts can take, and what the package asks for. There is no --selinux-policy flag, and this is deliberate rather than unfinished.
apply and prepare used to bind -f/--fapolicyd, and no phase read it – PrepareFapolicy has no Enabled field and PrepareOptions has no fapolicyd member. That flag has since been removed rather than wired up, along with its config key, schema entry and Ansible parameter, for the reason this paragraph gives. CHANGELOG.md records the matching cleanup on the phase side, drop the unused InstallFapolicy phase. A flag is not one flag: it is types/config.go, cmd/root.go, cmd/viper.go, both cmd/install_*.go files, internal/ansiblemod/params.go, and the DOCUMENTATION block in both Ansible action plugins. Adding that surface for a toggle nothing threads through is how the dead -f happened. An operator who does not want the policy applied removes it from the package, which is where it is declared.
Dry run
The phase is classified DryRunSkip, like PrepareSelinux and PrepareFapolicy beside it: a dry run reports it and does not run it. Its mutations are still wrapped in GenericPhase.Wet, which under that classification never executes, and the wrapping is there so the phase does not have to be rewritten if it ever declares a dry-run path. Wet now takes only that function: the host and message it also used to accept were never read, and the second function it would have run during a dry run was never passed by anybody, so the per-operation account they promised did not exist. A phase that wants to say what it would do says it through ClassifyDryRun and DryRunNote, once for the phase. Claiming readOnly would be wrong – the phase changes hosts – and a DryRun() that reported the commands it would run would have to re-derive the host filtering against live hosts to say anything true, which is work with no caller asking for it yet.
Why pkg/signing stopped being a wrapper around cosign’s CLI
Cargoship signs and verifies the distro.yaml blob with cosign, and until this change it did so through github.com/zarf-dev/zarf/src/pkg/signing. That package is a thin wrapper over cosign’s CLI command structs, which is exactly what cargoship wants, so importing it rather than reimplementing it was the right call. The reason it is now a fork is not a disagreement about the wrapper. It is five blank imports’ worth of binary, and then the command structs behind them.
What the import cost
zarf/src/pkg/signing registers four cloud KMS plugins and cosign’s ambient OIDC credential providers by side effect:
_ "github.com/sigstore/cosign/v3/pkg/providers/all"
_ "github.com/sigstore/sigstore/pkg/signature/kms/aws"
_ "github.com/sigstore/sigstore/pkg/signature/kms/azure"
_ "github.com/sigstore/sigstore/pkg/signature/kms/gcp"
_ "github.com/sigstore/sigstore/pkg/signature/kms/hashivault"
Every one of those resolves a key by reaching a cloud endpoint - KMS for the key material, Fulcio and an OIDC issuer for the keyless flow. The management node cargoship runs from is air-gapped, so none of them can be used from where cargoship actually runs. aws-sdk-go-v2/service/kms and its three siblings were reachable from nothing else in the tree. Removing them measured 100.39MiB to 92.98MiB, -7.41MiB, with mage build:binary.
Go has no way to drop a transitive blank import. The package that writes _ "…/kms/aws" registers it for every binary that links the package, however the linker is invoked. Dropping the registration means not linking that package, which means owning a copy.
The second step: the CLI structs went too
Dropping the KMS registrations left the cosign CLI command structs in place, and they carry their own weight: cmd/cosign/cli/verify pulls Open Policy Agent through pkg/cosign/rego and cuelang through pkg/cosign/cue, both for policy evaluation cargoship never used, and cmd/cosign/cli/options pulls go-github and the GitLab client through pkg/cosign/signature for git-based key storage cargoship never used.
Taking them out, with the keyless surfaces that needed them, measured 87.11MiB to 56.79MiB, -30.32MiB. Signing and verification are now sigstore-go, in signkey.go and verifykey.go: sign.Bundle with a Keypair adapter over a cosign-format private key, and verify.NewVerifier with WithKey() and WithNoObserverTimestamps(). The bundle format is unchanged, and the interoperability was tested in both directions - a cargoship-signed bundle verifies under cosign verify-blob --key, and a cosign sign-blob bundle verifies under cargoship. TestSignBlobWithKeyRoundTrip and TestVerifyBlobWithKeyAgainstCosignCLI pin both.
One seam onto cosign’s library is kept deliberately: cosign.LoadPrivateKey reads the encrypted private-key envelope, which is cosign’s own scrypt and secretbox format. Re-deriving it here would be a way to get it subtly wrong, and the cost of keeping it is sigstore/rekor and goleveldb arriving through cosign/pkg/cosign - about 1.3MiB, which is worth paying for not reimplementing a key-decryption format.
What no longer works
Keyless (Fulcio/OIDC) signing and verification, and cloud KMS key references (awskms://, azurekms://, gcpkms://, hashivault://). All of them need network egress to a cloud the air-gapped management node cannot reach. A COSIGN_PASSWORD-protected local keypair is the only signing method, and is what the e2e suite exercises.
Whether keyless should return, and in what form, is #848. sigstore-go/pkg/verify supports WithCertificateIdentity, so keyless verification is reachable without relinking the CLI structs - which is the option that issue argues for.
A package already signed keylessly cannot be verified. VerifyPackage detects that from the bundle via ReadBundleInfo and fails naming the method and the issue, rather than letting the key verifier report a mismatch it cannot explain.
What was dropped as unused
SignManifest and VerifyManifest, their options structs and defaults. Cargoship signs the distro.yaml blob, not an OCI manifest referrer, and nothing in the tree called them. They were the only users of go-containerregistry/pkg/v1/remote, pkg/name and zarf/src/types inside the package, so those imports went with them.
A build tag would be the better fix, upstream
The KMS half of this did not have to be a fork. A //go:build cloud file in zarf’s own src/pkg/signing, holding nothing but those five blank imports, would gate the registrations on a tag the consumer chooses:
//go:build cloud
package signing
import (
_ "github.com/sigstore/cosign/v3/pkg/providers/all"
_ "github.com/sigstore/sigstore/pkg/signature/kms/aws"
// ...
)
Measured on a throwaway module importing exactly those packages, release flags, linux/amd64: 4.58MiB without the tag against 23.07MiB with it. The linker never sees the KMS providers unless somebody asks for them, and go build -tags cloud restores the capability for a consumer that can reach a cloud. That is strictly better than a fork, because it tracks upstream for free.
Two things it does not do, which is why the fork is still here.
It does not shrink vendor/. go mod tidy cannot know which tags a consumer will set, so it keeps the requirement for every build configuration, and go mod vendor copies the SDKs in. The same throwaway module vendors 34MiB including a full aws-sdk-go-v2 tree with the tag unset. The binary is the win; the dependency surface is not.
It does not touch the second half of the fork. The cosign CLI command structs are called directly – verify.VerifyBlobCmd, sign.SignBlobCmd – not blank-imported, so no tag excludes them, and they are what pull Open Policy Agent, cuelang, go-github and the GitLab client. Cargoship does not want those code paths at any build setting: keyless is removed and the rego and cue policy evaluation was never used.
If this is raised upstream, the polarity matters: the tag has to be opt-in (cloud adds the providers) rather than opt-out, or every consumer stays fat unless they know to disable something. The honest argument against it is that a consumer who forgets the tag loses awskms:// support at runtime rather than at build time, so the “unsupported provider” error needs to name the tag.
Do not revert this to the upstream import
The fork looks like duplication, and from inside pkg/signing there is nothing to show what reverting would cost: the option structs have the same field names, and CosignSignBlobWithOptions and CosignVerifyBlobWithOptions have the same signatures they always had. Going back to zarf/src/pkg/signing relinks four cloud SDKs and the whole cosign CLI tree – 37.73MiB across the two steps, on a binary that is now 39.48MiB – none of it reachable from an air-gapped host.
If the fork needs resyncing with upstream zarf, resync bundle.go and leave the rest alone. cosign.go is no longer zarf’s code in any meaningful sense: the option structs are cargoship’s, and the bodies delegate to signkey.go and verifykey.go rather than to cosign’s commands. The attribution header stays because the structs and CheckOverwrite began as zarf’s.
The OpenTofu provider was removed, and where it went
cmd/terraform-provider-cargoship, internal/tofuprovider, test/e2e/tofu, example/terragrunt, docs/tofu and the Release mage namespace used to live here. choice-tofu-provider-layout records why: one module rather than two, because the linker never loads a package the import graph does not reach, so the CLI paid nothing for a provider it did not import. That page also named the one trigger that would make a separate repository right – a registry listing – because the OpenTofu registry validates a repository against {owner}/terraform-provider-{name} and no tag or directory inside colonel-byte/cargoship substitutes for it.
That trigger was pulled. The provider now lives in colonel-byte/terraform-provider-cargoship, which consumes this module as a library – the phases an apply runs are the same ones the CLI runs, which was always the premise – and released v0.1.1 carrying the zips, SHA256SUMS and per-platform manifest the registry protocol parses, signed with both the GPG key that signs cargoship’s rpm and deb and the cosign key that signs its archives.
This file is kept, at the name the removal plan had, for two reasons. The page was published, so the name is a URL somebody may still hold; and a reader who finds tofu in go.mod and no provider should not have to work out whether something was lost.
What stayed, and is not the provider’s
zarf-dev/zarfingo.modhas nothing to do with any of this. It is a library dependency for package formats, architecture detection and OCI handling, and predates both the provider and the collection that left before it.pkg/action.NewRefreshstays. It is the read-only half of an apply – connect, detect, gather facts, disconnect – and it exists so that “which phases are safe to run without changing a host” is decided once. The CLI does not expose it; the provider consumes it from here, and refresh is still generated from it.cluster.Host.ConfiguredAddresswas added for the provider’saddressattribute and is now also what a run receipt identifies a host by.pkg/action.KubeConfigOptions.NoWrite,KubeConfig.BytesandKubeConfig.Configwere added so the provider could have the credentials without writing~/.kube/config.cargoship kube-config --no-writeis not the same thing.test/microvmand the roottestpackage stay, and are importable rather than internal, because the provider’s end-to-end suite consumes both from the other repository. That export was #920;CARGOSHIP_E2E_EXAMPLE_ROOT, which lets that suite read the generated distro definitions out of a cargoship checkout rather than carrying a copy, was #941.cmd/cargoship/deps_test.gokeeps its guard onhashicorp/terraform-plugin. The two entries naming the provider’s own packages went with them, and what the test protects did not: a provider-shaped dependency reaching the CLI costs an air-gapped operator megabytes they carry by hand, and the only symptom is a larger binary, which no check reads.
The decision records went with it
choice-tofu-provider-layout, on why it was a package of this module and what would move it; choice-tofu-secrets, on why the kubeconfig is withheld unless asked for; choice-tofu-host-removal, on why a removal is a marker rather than a deletion; choice-tofu-e2e, on why the suite drives the shipped module; and design-tofu-provider-install, the install path’s design notes.
What was given up on purpose
The ghcr.io OCI mirror went with the Release namespace. mage release:tofuProvider assembled an image index an OpenTofu provider mirror serves and pushed it to ghcr.io, and release-tofu-provider.yaml ran it on the CLI’s v* tag. The new repository releases the same artifact shape on its own tags, so keeping the target here would have published a second copy of a provider this repository no longer builds. An operator whose .tofurc points an oci_mirror at ghcr.io/colonel-byte/tofu-providers needs to repoint it at the new repository’s releases; nothing here serves that path any more.
Why a missing dnf versionlock plugin warns instead of failing the install
After installing the engine packages, cargoship pins them at the installed version so a host-level package manager update cannot move the engine out from under it (pkg/phase/51_package_pin.go). On Debian-family hosts that is apt-mark hold; on RHEL-family hosts it is dnf versionlock add. The two are not equally available, and that asymmetry is the whole of this decision.
apt-mark is always there; versionlock is not
apt-mark ships inside the apt package itself. Any Debian-family host that just installed a .deb has it, so a failing apt-mark hold means something is genuinely wrong and holdAPTPackages returns the error.
dnf versionlock is a plugin in a separate package – python3-dnf-plugin-versionlock on dnf4, dnf5-plugin-versionlock on dnf5 – and a minimal RHEL-family install does not include it. On such a host dnf versionlock add exits non-zero with No such command: versionlock, which under fail-fast turned a perfectly good install into a failed apply.
Cargoship cannot fix it, and cannot fix it later
The obvious repair is to install the plugin first, and cargoship cannot. It runs from a management node against an air-gapped fleet, so a package that is not staged inside the package being applied has to come from a repository the host can reach, and the versionlock plugin is not staged.
Cargoship does install container-selinux by name, in pkg/phase/21_prepare_selinux.go, and that phase is fatal on failure. It looks like a counterexample and is in fact the rule this decision follows. The staged RPM set carries k3s-selinux / rke2-selinux, and those declare Requires: container-selinux. With no repository to resolve it from, dnf installing the staged file fails on the unmet dependency unless container-selinux is already on the host, which is why that phase runs at 21, ahead of the install, and why it stops the run when it cannot succeed.
The distinction is not by-name against staged, it is prerequisite against convenience. Without container-selinux there is no engine install to pin. Without versionlock the install completes and is merely unpinned. Failing one and warning the other is the same rule applied to two different stakes.
So the choice on an unpinnable host is between installing unpinned and not installing at all.
Failing is also badly timed. By the time pinning runs, InstallPackage has already succeeded and the engine packages are on disk. Returning an error there does not undo the install; it reports a failure for a host that is in fact installed, and the operator’s only route forward is to stop the run from trying to pin. An unpinned host is a weaker guarantee than a pinned one, but it is a working host, and the warning names it and the packages it left unpinned.
What is still fatal
The tolerance is deliberately narrow, and it lives in the rpm-specific holdRPMPackages, unholdRPMPackages, and unholdKubeadm rather than in the shared installAndPinPackagesFor, so it cannot spread to the apt path:
- A host that has versionlock and still fails to pin is a real error. The probe (
dnf versionlock --help, chosen because it reads no repository metadata and touches no existing lock) is what separates “this tooling is absent” from “this tooling failed”. - A staged file whose package name cannot be read fails the phase on either package manager. That is a property of the artifact, not of the host’s pinning tooling, and
packageNamesgives up on the first file it cannot read – so continuing would leave every package on the host unpinned while reporting success.
The same rule now covers the unhold
The unhold this once deferred now exists. installAndPinPackagesFor releases the prior pin before installing, because that is what lets apt or dnf move a held package to a new version at all, and the same closure runs on every upgrade. That extends this decision rather than changing it: unholdRPMPackages skips a host with no versionlock plugin for the same reason holdRPMPackages does, and it matters more there, because unhold runs before the install – dnf versionlock delete failing with No such command: versionlock would fail the phase outright on exactly the hosts this decision keeps working. A host with no plugin has nothing locked, so there is nothing to release. It skips at debug rather than warn, since holdRPMPackages warns about the same host later in the same phase and that warning is the one naming a real consequence.
The kubeadm upgrade unholds separately
Upstream.PreStartUpgrade installs a new kubeadm before the shared upgrade phase touches the rest of the package set, so it releases kubeadm’s own pin rather than going through installAndPinPackagesFor. It is subject to this decision for the same reason and in the same shape: unholdKubeadm probes with dnf versionlock --help and skips at debug when the plugin is absent, leaving the apt path unconditional.
It cannot reuse pkg/phase’s helpers, because pkg/phase imports types/distrocfg and the import cannot run the other way. The probe is therefore duplicated, and the two copies have to agree – a change to one is a change to both.
One consequence is worth naming: the probe cannot tell an absent plugin from a host that answers nothing at all, so an unreachable rpm host skips the unhold and surfaces its failure on the install that follows instead. That is the same trade holdRPMPackages already makes, and the phase still fails; only the message moves.
What would justify revisiting
Staging the versionlock plugin as part of the package, so a RHEL-family host can be given the plugin before the pin instead of excused from it. That turns an unpinnable host back into a fatal error, which is the stronger behavior, and it is the only change that should reverse this.
Why the upstream examples carry a CNI, and how each one gets its manifest
Upstream (kubeadm) installs no CNI. A cluster built from an upstream example without one forms, passes every check cargoship makes, and leaves every pod at ContainerCreating forever. spec.config.manifests exists for exactly this: cargoship kubectl-applies what a package names there from the leader once it has bootstrapped (pkg/phase/64_apply_manifests.go), and only the upstream distro implements ManifestPaths, because rke2 and k3s ship their CNI with the release and configure it through a HelmChartConfig instead.
So the question is not whether an upstream example declares a CNI. It is which one, and where the manifest comes from – and the three CNIs answer that differently enough that each gets its own flavor directory rather than one example with a knob.
One flavor per CNI
example/upstream-cilium, example/upstream-canal, example/upstream-flannel. The CNI decides more than the image list:
- the manifest that gets applied,
- the pod CIDR kubeadm has to be told about, since flannel and canal hardcode
10.244.0.0/16in the manifest they publish and cilium reads the cluster’s, - whether kube-proxy stays.
A single example with a values-driven CNI could not express the second: the pod CIDR goes into kubeadm init’s ClusterConfiguration, which is read once at bootstrap, so it has to be in the definition rather than chosen later. The rke2 examples already split per CNI for the same reason.
flannel and canal: a pinned URL
Both publish a manifest per release. The flavor pins the URL with its version in it, cargoship verifies the download against the digest in example/shasums.json, and the generator reads the same file to pull out the images.
The URL names a version rather than using releases/latest/download. An alias serves different bytes from one stable URL, and example/shasums.json keys entries by file name – so kube-flannel.yml would flip digests on every upstream release, which is precisely the “same name, different content” case the cache treats as a cache miss worth re-hashing.
Cilium: rendered, because there is nothing to download
Cilium stopped publishing an applyable manifest around v1.10; install/kubernetes/quick-install.yaml 404s. What exists is the Helm chart. Cargoship has no Helm dependency and the apply phase takes kubectl apply -f paths, so the manifest is produced at generate time with helm template and committed.
Rendered once per chart version, not once per example. It lands in example/upstream-cilium/cni/cilium-<chart>.yaml and every version directory refers to it as ../../cni/…, the way k3s examples share example/k3s/core/. Nineteen committed copies of a 65 KB manifest would otherwise say the same thing nineteen times.
The render is skipped while the file exists. This is what keeps helm out of the loop that keeps the examples current: only a chart pin bump produces a path that is not already in the repository, so mage generate:examples on a committed tree never shells out to helm, and refresh-examples.yaml – which runs weekly and never touches the pin – needs nothing but the Go toolchain. The cost of a new pinned tool falls on whoever moves the pin, locally.
hubble.tls.auto.method=cronJob is not optional
The chart’s default (helm) generates Hubble’s CA keypair while templating and writes it into a Secret in the output – it labels that object cilium.io/helm-template-non-idempotent itself. Two consequences, both disqualifying:
- the render stops being reproducible, so every regeneration produces a diff,
- the private key would be committed, handing every cluster installed from the example the same Hubble CA.
cronJob has cilium’s own certgen job mint the certificates in the cluster instead. The manifest then carries no key and renders byte-identically every time; the certgen image it adds is picked up by the image scrape like any other.
refuseEmbeddedKey fails the render if a key shows up anyway. A chart’s defaults can change under a version bump, and this failure is silent by nature – a committed key looks like any other base64 blob in a diff of a file marked linguist-generated.
Images come out of the manifest
Each flavor’s imageConfig.images is the kubeadm-pinned control plane list plus whatever image: lines the manifest it applies carries. A second list maintained by hand would agree with the manifest until the CNI added a sidecar, and the failure shows up only on an air-gapped host at apply time – the one place it is most expensive to find.
What was not done
Kube-proxy-free cilium. upstream-cilium runs cilium alongside kube-proxy, unlike example/rke2-multi-cni-cilium which replaces it. Replacing it needs InitConfiguration.skipPhases: [addon/kube-proxy], which types/distrocfg/upstream_kubeadm.go does not render, plus a k8sServiceHost that only runtime cluster state knows. That is a feature in the kubeadm config path, not an examples change.
Helm chart support in the apply phase. It would let the package ship the chart instead of a rendered manifest, and would be the more faithful way to install cilium. It is also a new surface in api/ and pkg/phase/ for a problem helm template solves at generate time.
Why onCreate actions go through zarf’s package converter
runCreateActions in pkg/packager/assemble/actions.go hands each action to zarf, and since zarf v0.87.0 that means converting it first. The conversion is one function, toAPIActionSet, and it does something that looks wasteful: it wraps a single action set in a throwaway ZarfPackage carrying one unnamed component, runs convert.PackageFromV1alpha1 over the whole thing, and reads the converted action set back out of Components[0]. This document records why that is the shape, and the behavior change it brought with it, because both look like accidents from the code alone.
What forced it
zarf v0.87.0 moved its canonical action model out of src/api/v1alpha1 and into a normalized src/api, and retyped the executor against it:
// v0.86.0
func Run(ctx context.Context, basePath string, defaultCfg v1alpha1.ZarfComponentActionDefaults, actions []v1alpha1.ZarfComponentAction, variableConfig *variables.VariableConfig, values value.Values, stateAccess template.StateAccess) error
// v0.87.0
func Run(ctx context.Context, basePath string, actions []api.Action, opts RunOptions) error
The v1alpha1 types still exist, so nothing in cargoship stopped compiling for lack of a type. What broke is the handoff: cargoship holds v1alpha1 actions and Run now takes api.Action.
cargoship holds v1alpha1 actions because its own package schema embeds them. Actions.OnCreate in api/colonel-byte.dev/v1alpha1/distro/spec.go is a zarf.ZarfComponentActionSet, which is what puts mute, maxRetries, setVariable and the rest into schema/distro-v1alpha1-schema.json. Retyping that field onto api.Action would convert the problem into a schema change – mute becomes silent, maxRetries becomes retries, wait.cluster.condition stops being a string – and every distro.yaml an operator has written would need rewriting. A dependency bump is not where that happens.
So the conversion has to live at the call site, and it has to be the same conversion zarf performs on a package it loaded itself. Anything else means cargoship’s actions behave differently from zarf’s for the same YAML.
Why a whole package, for one action set
zarf’s action converters are in src/internal/api/v1alpha1. actionSetToGeneric, actionToGeneric and waitToGeneric are exactly what is needed here and none of them are reachable: Go’s internal-package rule is enforced by the compiler, not by convention. The only exported door through that boundary is src/api/convert, and it converts packages.
The synthetic package is therefore the price of using zarf’s mapping instead of writing a second copy of it. The alternative – a hand-rolled v1alpha1 to api converter in cargoship – was rejected on how it fails rather than on how it reads. A hand-rolled converter compiles forever. When upstream adds an action field, or changes what an empty wait condition defaults to, cargoship keeps converting the old way and the action quietly runs with the old meaning. Going through convert fails at compile time if the function is removed or retyped, and tracks upstream silently when it is not.
The field mapping is not a rename list
Three of the mappings would be easy to get wrong by hand, which is the concrete argument for not doing it by hand:
| v1alpha1 field | api field | Not a rename because |
|---|---|---|
mute | Silent | The sense is preserved but the word inverts in ActionDefaults, where Mute: true became Silent: true |
template | EnableTemplating | *bool with a false default collapses to bool, through ShouldTemplate’s opt-in semantics |
wait.cluster.condition | Condition.Expression | A string became a struct, and the converter stamps a Default of existence that the executor never reads |
TestToAPIActionSetMapsEveryField pins all of them. A field that stops crossing is not a build failure – the action still runs, just without whatever stopped crossing – so the test is the only thing standing between a dropped mute and a package build that starts printing a command’s output.
What else runs, because the door is a package-level one
convert.PackageFromV1alpha1 calls migrateDeprecated before it converts anything. That is the side effect of entering through the package door, and it is load-bearing in one case and inert in the rest.
Inert: migrateScriptsToActions reads component.DeprecatedScripts. The synthetic component has none and cargoship’s schema has no scripts field at all, so it cannot fire.
Load-bearing: migrateSetVariableToSetVariables rewrites an action’s deprecated singular setVariable into the setVariables list, when the list is empty. Under v0.86 nothing on this path read that field: runAction looked only at SetVariables. The field is in cargoship’s schema, marked deprecated rather than rejected, so a distro.yaml could carry it and validate.
What that cost, measured against test/e2e/noncluster/testdata/actions/distro.yaml on the commit before the bump:
ERR failed to create distro package: unable to run component before action: unable to render cmd of the "read both variables back" action: template "echo '{{ .Variables.FROM_LIST }}' > before-list.txt\necho '{{ .Variables.FROM_DEPRECATED }}' > before-deprecated.txt\n" resolved a missing key to "<no value>"
Not a silently unset variable. zarf ignored the field, so the variable was never set, and cargoship’s own renderer – which errors on a missing key rather than rendering <no value> into a shell command – failed the build on the next action that read it. The three routes a variable can take to a later action, before and after:
| Route | v0.86 | v0.87 |
|---|---|---|
setVariables list | works | works |
setVariable, deprecated singular | variable never set, build fails on the read | works |
| A value out of the values file | works | works |
So the bump turns a hard failure into a working build for a definition the schema already accepted. That is the right direction, and it is still a behavior change: a package that failed to build now builds, and the file an action writes changes from nothing to a value.
One thing is lost on the way in. migrateDeprecated returns the warnings it accumulated, including “please migrate to the list form of setVariables”, and convert.PackageFromV1alpha1 discards them. The migration is silent. The wrapper that logs them, ApplyMigrations, is in the same internal package as the converters, so there is no exported way to get the warning without reimplementing the migration to produce it.
What a reversal has to account for
Replacing toAPIActionSet with a direct converter is the obvious cleanup, and it costs three things that are not visible at the call site:
- the field mapping stops tracking upstream, in the silent direction described above;
- the deprecated
setVariablestops working again, failing builds that work today, unless the migration is reimplemented alongside the mapping; TestToAPIActionSetMigratesTheDeprecatedSetVariable,TestRunCreateActionsHonorsTheDeprecatedSetVariableandTestCargoshipCreateRunsOnCreateActionsfail, which is intended – they exist so that the second point cannot be reverted by accident.
TestRunCreateActionsDoesNotRewriteItsInput covers the other direction. migrateComponentActions rewrites the slice it is handed in place, so toAPIActionSet is called with a fresh one-element slice per action rather than with the caller’s own list. Converting the whole set in one call would hand zarf’s migration the list that is written back out to distro.yaml, and the package would ship carrying migrations its author never made.
Why sosedoff/ansible-vault-go and not apenella/go-ansible
Cargoship’s Ansible Vault support (cargoship vault encrypt, and automatic decryption of vault-encrypted registry credentials at apply time) is built on github.com/sosedoff/ansible-vault-go. github.com/apenella/go-ansible was also considered, since it’s a more widely known Go/Ansible integration, but it isn’t a fit for this use case:
- No decrypt support.
go-ansible’svault/encryptpackage only exports anEncryptStringtype – there is noDecrypt/DecryptStringanywhere in the module. Decryption is Cargoship’s actual requirement:applyhas to transparently decrypt registry credentials on every run. - It wraps the same library anyway.
go-ansible’sEncryptStringusessosedoff/ansible-vault-gointernally for the crypto. Depending ongo-ansiblefor encryption would still pull insosedoff/ansible-vault-goas a transitive dependency, plusspf13/aferoandgo-ansible’s own password-reader abstraction – more surface area to replace logic Cargoship already has inResolveVaultPassword(internal/clustercfg/vault.go), for no functional gain. - Wrong shape of library.
go-ansible’s primary purpose is wrappingansible-playbook/ansible/ansible-galaxyCLI execution as subprocesses. Cargoship needs neither Ansible nor Python installed on the machine runningapplyorvault encrypt–sosedoff/ansible-vault-gois a native Go implementation of the vault cipher, so encrypting and decrypting single string values stays a pure in-process operation, consistent with Cargoship shipping as a single static binary.
Net effect: switching would add a heavier, partially-redundant dependency, lose decrypt support outright, and reimplement functionality sosedoff/ansible-vault-go already provides directly.
The zarf collection was removed, and where it went
ansible/colonel_byte/zarf, internal/zarfmod, cmd/zarf/ and test/e2e/zarf used to live here. They were a proof of ZEP-0072, which proposes native Ansible module support for zarf, and they were built in this repository because the pattern the proposal describes is the one colonel_byte.cargoship already runs – see choice-ansible-module – so the cheapest place to find out whether the pattern held for zarf was beside the implementation it copies. That reason expired when the proof outgrew being a guest here. It now lives in colonel-byte/ansible-collection-zarf, which released v0.87.0 carrying the same signed collection tarball, module binaries and Linux packages that cargoship’s release used to, and the copy here was deleted in #822 rather than maintained as a second, worse one.
This file is kept, at the name the removal plan had, for two reasons. The page was published, so the name is a URL somebody may still hold; and a reader who finds zarf all over go.mod, pkg/action and pkg/coci and no zarf collection should not have to work out whether something was lost. Nothing was: cargoship depends on zarf-dev/zarf as a library, for package formats, architecture detection and OCI handling, and has since long before the collection existed. go mod tidy was run on the removal and changed nothing. The collection was never reachable from any cargoship code path – nothing outside internal/zarfmod and cmd/zarf/ imported it, and the cargoship collection never referred to it – which is what made it safe to remove on short notice.
The decision records that explained the proof went with it: choice-zarf-ansible-module, on why the modules are wrapper binaries rather than zarf itself and what that costs, and choice-zarf-info-modules, on why the read-only modules are plain action plugins and why zarf_state_info withholds credentials by default.
Design: Versioned Config Struct Generation for k3s / RKE2
Goal
Programmatically generate Go structs (and derived config.yaml files) that represent valid k3s server, k3s agent, rke2 server, and rke2 agent CLI flags - per minor version (e.g. 1.34, 1.35, 1.36) - so we can produce config files we know a given version will accept, without hand-transcribing flags from docs.
Hard requirement: the whole pipeline must be buildable and runnable in a disconnected/air-gapped network. No step may require live access to the Go module proxy, GitHub, or any other network resource at build time.
Why this isn’t a simple “unmarshal into the upstream struct” problem
Neither project unmarshals config.yaml into its internal Agent/Server structs via yaml struct tags. Instead:
- Flags are declared as
urfave/cliFlagvalues (cli.StringFlag,cli.BoolSliceFlag, etc.), each with aDestination: &SomeStruct.Fieldpointer. config.yamlis parsed generically and translated into equivalent CLI args, then fed through the same flag-parsing path as real CLI input.- The flag name (kebab-case, e.g.
cluster-cidr) is the authoritative YAML/CLI vocabulary - not the Go struct field name, which has noyamltag and doesn’t always match the flag name 1:1.
Conclusion: the flag list, not the struct, is the source of truth for what a valid config.yaml key looks like.
Key constraint: k3s cannot be safely imported as a library dependency
k3s’s own go.mod contains extensive replace directives pinning forked versions of k8s.io/api, k8s.io/apiserver, containerd, etc. Go’s module system does not honor a dependency’s replace directives - they only apply when that module is the main module being built (see golang/go#30354). This means:
go get github.com/k3s-io/k3s/pkg/cli/cmdsinto our own module, followed bygo mod vendor, is fragile: it may fail to resolve, pull incompatible transitive dependencies, or silently vendor different code than what k3s actually ships.- This risk is version-dependent and not something we can fully verify up front - a future k3s release could introduce an import that trips this even if earlier versions didn’t.
Decision: do not build or import k3s/RKE2 code at all. Extract flag metadata via static source parsing (go/parser / go/ast) instead of compiling or reflecting over the real package. This sidesteps the replace-directive problem entirely - we only ever read text, never resolve a dependency graph.
Tradeoff accepted: we lose compiler-verified type information. Acceptable, because k3s/RKE2 flag types are plain (string, bool, int, cli.StringSlice) - no exotic cli.Generic implementations expected.
Architecture
Two phases that never touch Go’s “one version per build” limitation, because they operate on different things:
- Extraction (per upstream version) - operates on committed source text, not compiled code. No third-party Go dependency ever appears in our own
go.mod. - Consumption - our own generated packages, one per version, distinct Go package paths. Ordinary Go, no conflict, since they’re not “the same package at two versions” - they’re just different packages we generated.
/thirdparty-src/
k3s/
v1_34/agent.go, server.go, version.go (raw files, pinned to v1.34.x+k3s1)
v1_35/...
v1_36/...
rke2/
v1_34/...
v1_35/...
/extractor/ # our tool: go/ast walker, no third-party deps
/gen/
v1_34/zz_server_config.go, zz_agent_config.go # generated, zero deps
v1_35/...
v1_36/...
/thirdparty-src/pins.json # manifest: distro -> repo, files, upstream tags
Workflow
Adding / refreshing a version (the only step requiring network access)
- Resolve the exact upstream tag (e.g.
v1.36.0+k3s1,v1.36.0+rke2r1). - Pull only the needed raw source files at that tag (
git show <tag>:pkg/cli/cmds/server.go, or a sparse/shallow checkout scoped topkg/cli/cmds) - not the whole repo, not a buildable module. - Commit those files under
thirdparty-src/<project>/<version>/. - Append the tag to that distro’s entry in
thirdparty-src/pins.json.
This is a small, auditable diff each time (a handful of files), not a dependency tree. git diff between two version directories is a reasonable first-pass changelog of flag changes between releases.
Offline steps (extraction → codegen → build)
All pure Go, no network:
- Extract:
go/astwalks each committedserver.go/agent.go, locates the[]cli.Flag{...}literal(s) (including flags referenced indirectly via package-level vars, e.g.&SELinuxFlag), and pulls out name, type (scalar vs. slice), default, hidden/deprecated status, and theDestinationfield name where present. Emits a JSON manifest per version. - Generate: a
text/templatepass turns each JSON manifest into a Go struct withyaml:"<flag-name>,omitempty"tags, in its own version-namespaced package (gen/v1_36, etc.). - Build: our main module builds normally. It has no dependency on k3s/RKE2 at all - at most one small, ordinary vendored dependency (e.g. a YAML marshal library) if we want marshal helpers, which is trivial to
go mod vendorsince it’s a real, un-forked dependency.
Known edge cases to handle explicitly
- Slice-typed flags (
cli.StringSlice, e.g.tls-san,node-label) must generate as YAML lists, not scalars. - Hidden/deprecated flags (e.g.
disable-agent,cluster-secretare markedHidden: truein k3s) exist in the flag list but are often aliases for newer flags - decide per-flag whether to emit them into generated configs or just track them for reference. - Cross-field validation (e.g.
--disable-apiserveris incompatible with--datastore-endpoint) is enforced only in k3s’s runtime code, not expressed in the flag list. A generated config can be structurally valid YAML but still semantically invalid. Out of scope for the generator itself; worth a follow-on validation layer if needed. - RKE2 depends on a specific k3s version, not a 1:1 Kubernetes version. RKE2’s generator needs to resolve both the RKE2 tag and the k3s tag it vendors for shared flags.
- Renames without an explicit
Aliases:entry won’t be caught automatically by a diff - flag as “needs manual review” rather than silently treating as added+removed.
Rough effort estimate
| Piece | Effort |
|---|---|
go/ast flag-slice extractor (incl. indirect var refs, Destination unpacking) | 3–5 days |
| JSON manifest → struct templating | ~1 day |
| Slice/hidden/deprecated-flag edge case handling | ~1 day |
| Per-version diffing / changelog view | ~half day (mostly free once manifests exist) |
| Sparse source-pull tooling per version | ~1 day |
Consumption: wired into types/distrocfg
RancherCommon.ConfigureEngine (types/distrocfg/rancher_common.go) looks up the generated struct for the node’s distro (d.ID, i.e. k3s/rke2) and dis.Spec.Version, truncated to its minor release, via gen.Lookup (pkg/engineconfig/gen/lookup.go). If a match exists, every key in the resolved config.yaml map is checked against the matching ServerConfig/AgentConfig struct’s yaml tags (gen.Keys, reflection-based) - server vs. agent selected by host.IsController() - and any key that isn’t a real flag for that distro/version/target is logged as a warning (a likely typo or version mismatch) and deleted from the map before it’s written. If no generated version matches (nobody has pulled/generated that minor line yet), it logs a warning once and leaves the config completely untouched - falling back to blindly writing every key exactly as before this check existed. So: known distro/version -> unrecognized keys are dropped; unknown distro/version -> nothing is validated or removed. See “Open questions” below for what’s still deferred.
gen.Registry (pkg/engineconfig/gen/zz_registry.go) is generated alongside the per-version structs by mage generate:engineConfig - every distro/version pull gets wired in automatically, with no hand-maintained import list to keep in sync.
Packaged components: value-level validation for disable:
disable: is the one key whose values also come from a closed, version-specific vocabulary, so it gets checked too. Each generated package carries a zz_addons.go with Addons (valid disable: values), CNIs, and IngressControllers, extracted from k3s’s pkg/cli/cmds/stage.go const and RKE2’s pkg/cli/types.go slices; Entry carries all three. See docs/dev/thirdparty-src.md for how RKE2’s list is composed - it is the union of DisableItems, the chart-name pair per CNI and ingress item, and the literal clx.Set("disable", ...) calls, because RKE2 accepts far more than it advertises.
Both consumption points use gen.UnknownAddons: RancherCommon.validateEngineConfig warns on controllers (agents deploy nothing, and the key check has already removed disable there), and assemble.logUnknownEngineConfig logs at debug when a package is built. Neither drops the value. The key check can afford to delete, because a key is either in the extracted flag list or it isn’t. The addon list is derived, so a false positive is likelier - and the cost of one is inverted: dropping an unrecognized disable: entry would deploy a component the user explicitly asked to be without, which is worse than writing through a name the engine will simply ignore. An empty Addons (a version whose component source was never pulled) means the check is skipped entirely.
Open questions / follow-ups
- Do we want a validation layer that encodes the known cross-field constraints (disable-apiserver vs datastore-endpoint, etc.), or leave that to whoever consumes the generated config? (Partly answered for value-level checks:
disable:values are now validated against the extracted packaged-component list, warn-only.cni:andingress-controller:have their vocabularies extracted intoEntry.CNIs/Entry.IngressControllersbut nothing checks them yet.) - Minor-version granularity only (1.34/1.35/1.36), or also track patch releases when a flag changes mid-minor?
- Should hidden/deprecated flags be emitted into generated structs at all, or tracked separately as metadata only?
- The current consumer (
types/distrocfg) drops unrecognized keys (with a warning) for known distro/versions, but doesn’t fail the build. Worth revisiting once there’s confidence false positives (e.g. a flag added in extraction but not yet in a resolvable form) are rare enough to make a hard failure safe instead of a silent drop.
Ideas: other sources for cargoship inventory from-*
This is a shortlist, not a decision. cargoship inventory from-ansible established a command group whose members all answer the same question – where does the list of machines come from – and the obvious next question is which other sources belong in it. Nothing here is committed to; the point is to record the criteria and the ranking while the reasoning is fresh, so that the next adapter is argued against a list rather than added because someone asked.
What every adapter has to do, and nothing more
An adapter builds the ansibleinv.Input shape – groups, hostvars, roleGroups – and hands it to one translator. Role derivation, controllers-first ordering so that the leader is a controller, the refusal of an unknown cargoship_ variable, and validation against the embedded inventory schema are written once and are already fuzzed (fuzz/ansible_inventory_fuzz_test.go). An adapter is a decoder. When one starts wanting rules of its own – its own notion of a role, its own precedence between two variables that disagree – that is the signal it does not belong in the binary, and should be a script that emits the JSON contract described below.
One naming consequence, worth settling before a second adapter exists rather than after: the shared translator lives in a package called ansibleinv, and the moment anything but Ansible calls it the name is wrong. Rename it to something like clusterinv when the second caller lands.
The four criteria
A source is worth first-party support when it meets all four.
It works offline. The management node is inside the airlock and reaches nothing. A source that needs an API call from the management node is a source that needs a script on the outside anyway, and if a script is being written the script may as well emit the contract.
The owner of the truth resolves it. Cargoship should read a resolved answer, not reproduce someone else’s resolution. This is the same argument choice-ansible-module makes for letting Ansible resolve its own inventory rather than parsing inventory files in Go: reproducing group_vars, children and precedence would reproduce them wrongly, and the symptom is a node installed in the wrong plane with nothing in between to say so.
The format is documented and stable. A format that changes under us turns a silent misread into an install against the wrong machines.
Roles are stated, not guessed. A host’s role has to come from something the operator wrote – a group, a column, a tag, a flag. Inferring it from a hostname pattern is guessing, and Host.Role is read verbatim everywhere else in this repository precisely because it is not inferable.
The list, ranked
1. from-json – the contract rather than a source
Publish {groups, hostvars, roleGroups, cluster} as a documented, versioned input shape, read from a file or from stdin. Every other item on this list collapses into “emit this JSON”, which is what keeps the command group at three or four members instead of fifteen. It is also the cheapest thing here: the shape already exists as ansibleinv.Request, and the work is documenting it as an interface other people may target, plus a version field so it can change later without breaking the scripts that targeted it.
Ship this first, whatever else happens. It is the thing that bounds the size of the rest of the list.
2. from-csv
The rack sheet. Columns for hostname, address, role, user, port and key path, out of the spreadsheet a data center hands over, with a flag to map a header that disagrees (--column role=Function). No network, no dependency, no format drift, and it is the one source that exists in every air-gapped shop regardless of what tooling they otherwise run. It is unglamorous enough to be overlooked and is probably the highest-value first-party adapter on the list.
3. from-ssh-config
An SSH client configuration maps almost exactly onto rig.SSH: HostName onto the address, User, Port, IdentityFile onto the key path, and ProxyJump onto the bastion. An operator who can already reach the fleet by name has, in effect, written most of an inventory.
Do not parse the file. Include, wildcards and Match make ssh_config a resolver, which is the trap this document keeps naming. Shell out to ssh -G <host> per host and read the resolved answer, which is the owner-resolves-it rule applied. Roles come from a flag or from a list of host patterns per role, never from the name.
IdentityFile stays a path in the generated document. It is recorded, never read: key material in a generated inventory is always named by a path and never given by value.
4. from-terraform (or OpenTofu)
Useful, with one decision that determines whether it is cheap or endless.
Reading resources[] out of state means reading provider-specific attributes – aws_instance.public_ip, proxmox_vm_qemu.default_ipv4_address, libvirt_domain.network_interface[0].addresses[0], vsphere_virtual_machine.default_ip_address – and grouping by tags, name prefixes, for_each keys or module paths, all per-shop convention. That is a provider mapping table in Go which is never finished.
Reading outputs is not. The operator writes one output block whose value is the contract from item 1, in HCL, where they already know their own provider’s attribute names:
output "cargoship_inventory" {
value = {
groups = {
controller = [for k, v in aws_instance.kc : v.private_ip]
worker = [for k, v in aws_instance.kw : v.private_ip]
}
hostvars = { for k, v in aws_instance.kc : v.private_ip => {
ansible_user = "ubuntu"
cargoship_hostname = v.tags["Name"]
}}
}
}
tofu output -json then runs outside the airlock, the JSON crosses it, and the management node reads a file. No backend access, no network, no tofu binary on the inside, and no provider knowledge in cargoship. At that point from-terraform is barely distinct from from-json: unwrap .<name>.value and decode.
A middle tier exists if writing an output block is too much to ask – read terraform show -json, which is documented and carries a format_version, with the mapping supplied by flags such as --address-attribute private_ip and --controller-filter 'tags.role=controller'. That keeps the provider table out of Go but gives cargoship a small query language of its own, and small query languages grow. Do it on real demand, not in advance.
HCL itself is out. The hosts are not in it: count, for_each, variables, modules and data sources mean the configuration describes a shape rather than a fleet, and hashicorp/hcl/v2 would be a dependency bought in order to get a wrong answer.
Security, because it is easy to get wrong here specifically: a Terraform state file is secret-bearing. It holds provider credentials and every sensitive value in plaintext. An adapter that reads raw state must not log the file, must not echo attributes into error messages, and must refuse any output marked "sensitive": true rather than copy it into a generated inventory that lands on disk unencrypted. Reading outputs rather than state keeps that blast radius small, which is a second reason to prefer it.
5. from-kubernetes
The reverse direction, for a cluster cargoship did not build. kubectl get nodes -o json gives an InternalIP per node and node-role.kubernetes.io/* labels, which is exactly the controller and worker split, and that is enough to run engine-config-sync or reset against a brownfield cluster.
What decides its priority is that it yields half an inventory. Addresses and roles, yes; SSH user, port and key, no – those are not in the Kubernetes API and have to arrive by flag. It is a discovery step rather than a source.
6. from-netbox, from-maas
NetBox is the real source of truth in a good number of the environments this tool targets, and its model maps cleanly: devices, device roles, primary IPs, tags. The obstacles are that it needs network from the management node, which the airlock does not give, and an API client in the binary, which is dependency weight for one integration. Better served by a script that emits item 1. Worth reconsidering only if several users ask for the same field mapping.
Explicitly not
from-inventory, reading an Ansible INI or YAML inventory file directly without Ansible installed. Tempting because it would drop the Python dependency the action plugin adds; it reintroduces group_vars, host_vars, children and precedence, which is the thing the module architecture deliberately does not reimplement.
from-vsphere, from-proxmox, and the cloud providers. Provider attribute tables that are never finished, and network from a node that has none.
Scanning: DHCP leases, ARP tables, a subnet sweep. These guess which machine is a controller, and a wrong guess installs a control plane on someone else’s server.
Suggested order
from-json, then from-csv, then from-ssh-config, then stop until someone asks. from-terraform slots in wherever demand puts it, built on the outputs path rather than the state path.
The engine’s disable key does not gate a chart’s images
spec.config.engine.manifest.<chart>.enabled decides whether a chart’s images are exported and uploaded. .config.disable – the engine’s own key, which every example exposes as addons.disabled – decides whether the engine installs the chart at all. They are not wired together, and the asymmetry is visible to an operator:
| How the chart is turned off | HelmChartConfig written | Images uploaded |
|---|---|---|
enabled: false | no | no |
addons.disabled: [...] | no | yes |
So a cluster that disables cilium the documented way, addons.disabled: [rke2-cilium], still has all eleven cilium images exported to tarballs and pushed to every node. helmChartConfigs (types/distrocfg/rancher_common.go) consults both keys; Config.EnabledImages (api/colonel-byte.dev/v1alpha1/distro/chart.go) consults only enabled.
Why it was left
Wiring the disable list into EnabledImages is not a one-liner, and the reasons are worth writing down before someone decides it is:
- The normalization is not where it would be needed.
disabledChartshandles a YAML list, a bare string, and surrounding whitespace, and it is unexported intypes/distrocfg, which imports the distro API package. Reusing it fromEnabledImagesmeans moving it rather than calling it, and duplicating that parsing is how the two switches would drift in a second, quieter way. - It would make the API package opinionated about an engine key.
disableis rke2’s and k3s’s;manifestis already only read by those two distros, butConfig.EnabledImagesis on the shared type and upstream distros call it too. - The right answer may be fewer switches rather than more plumbing.
enabledarrived to gate images. If it becomes a real install switch – which a user-supplied Helm chart would need, since a disabled supplied chart should not be installed at all rather than merely have its images withheld – then one of the two keys should probably defer to the other rather than both being read everywhere.
What to decide
Whether enabled and disable stay two mechanisms or become one. Revisit alongside the user-supplied chart work, which is what forces the question: until then the cost is wasted upload for a chart disabled through addons.disabled, which is real but is no worse than the behaviour before per-chart images existed.
See package-values for what each key does today.
TODO: persist archives fetched for an extractPath file
This is a punch list, not a decision record like the choice-*.md files beside it. Every walk of the cluster suite builds its own package, each distro.yaml names an etcd release to pull etcdctl out of, and none of those archives is ever cached between runs - so a cluster walk downloads them again every time and a momentary DNS failure fails the walk at the package build. That accounted for three failed runs out of five in one session, at Test_00_UpgradePackage and Test_00_JoinMachine, neither of which has anything to do with the phases under test.
Why they are not cached
assembleFile in pkg/packager/assemble/assemble.go splits on whether a file declares extractPath:
| Entry | Download call | Persisted? |
|---|---|---|
plain source | DownloadToFileWithChecksum(ctx, source, dst, file.Shasum, ...) | yes - keyed by shasum under files/ |
with extractPath | DownloadToFileWithChecksum(ctx, source, compressedFile, "", ...) | no - archives is per-build only |
The empty shasum is deliberate and correct: with extractPath set, file.Shasum is the digest of the extracted file, and the archive it came out of has a different one. The archives map it uses instead is reserved and populated within a single assemble call, and the defer removes an archive whose download failed. Nothing writes it to ~/.cache/cargoship/files, so the next build starts from nothing.
The evidence is the cache itself: 1.7G and 109 entries under files/, every one an RPM or a .deb, and find ~/.cache/cargoship -iname '*etcd*' returns nothing after five full runs that each downloaded one.
What to do about it
The archive needs a cache key that does not depend on a digest the definition does not carry. Two shapes, in preference order:
- Let a
filesentry declare the archive’s own digest alongside the extracted file’s, and key the persistent cache on it. This keeps the integrity guarantee that the plain-source path has and is the only option that lets a cached archive be trusted rather than merely reused. - Key on the resolved URL and verify the extracted file as today. Cheaper and needs no schema change, but a cached archive is then trusted on its name - acceptable for the suite, weaker than what this project does everywhere else, and worth saying out loud in the code rather than leaving a reader to notice.
Do not reach for a suite-local pre-warm step that downloads the five etcd versions before the walks. It hides the problem from the one place that keeps hitting it, leaves a real cargoship package build re-downloading on every invocation, and the examples drift - the versions in use today are v3.6.6, v3.6.7, v3.6.12, v3.6.14 and v3.7.1, and nothing would keep that list in step.
Scope
This is about the archive, not about the extraction or the checksum of what comes out of it, both of which work. Nor is it about node egress - see todo-microvm-egress-blackout.md; a package build legitimately reaches the network, and the point here is that it should not have to do so twice for the same artifact.
TODO: cut the microvm fleet’s internet access once its nodes are up
This is a punch list, not a decision record like the choice-*.md files beside it. Cargoship’s whole premise is that a target host receives everything over SSH from a staged package and reaches nothing itself, and no backend currently proves that: a phase that quietly fetched something from the internet would pass on every one of them. The microvm fleet has working egress today, measured from kc0 mid-run - github.com resolves and curl https://github.com returns 200. This document is what it would take to take that away, and what not to try.
Why it is not one command
Each node gets two NICs from test/microvm/vm.go:
| NIC | QEMU netdev | Address | Carries |
|---|---|---|---|
enp0s3 | -netdev user,id=mgmt,hostfwd=... (SLIRP) | 10.0.2.15/24 | egress and the inbound SSH every phase runs over |
enp0s4 | -netdev socket,id=lan,mcast=... | 10.73.193.x/24 | the fleet’s own layer 2 segment, no egress |
So the mgmt NIC cannot simply be dropped - that is the SSH. The detail that makes a blackout possible is that SSH arrives from 10.0.2.2, the SLIRP gateway, which is reachable through the kernel’s own 10.0.2.0/24 dev enp0s3 scope link route rather than through the default route.
The approach to take
Apply an nftables blackout inside each guest after cloud-init has finished, gated behind an environment variable so the default behaviour does not change:
nft add table inet airgap
nft add chain inet airgap out '{ type filter hook output priority 0; policy accept; }'
nft add rule inet airgap out oifname enp0s3 ct state established,related accept
nft add rule inet airgap out oifname enp0s3 drop
Inbound SSH survives on established/related, anything the guest initiates is dropped, and enp0s4 and the CNI’s interfaces are untouched. It belongs in the microvm provisioning path after the cloud-init wait, not in the suite, so that mage dev:vmUp can produce an air-gapped fleet to poke at by hand as well.
Two things not to try
-netdev user,restrict=on has the right semantics and the wrong timing. It isolates the guest while leaving hostfwd working, which is exactly what is wanted, but it is a launch-time flag and cloud-init installs firewalld, fapolicyd and python3-dnf-plugin-versionlock with dnf (test/microvm/image.go). A node that boots restricted never provisions. Hot-swapping the netdev over QMP after bringup avoids that at the cost of a QMP socket and dropping the NIC’s connections, which is more machinery than this is worth.
ip route del default is half a blackout. /etc/resolv.conf points at 10.0.2.3, which is on-link on 10.0.2.0/24, so name resolution keeps working after the default route is gone and only routed egress dies. A test that still resolves names is not testing an air gap.
What to expect, and what this does not cover
- The first run will not be green. Any step quietly relying on node egress starts failing, which is the point, but budget for fallout rather than a clean pass.
- firewalld is cargoship’s to manage on these nodes. A separate
inet airgaptable should be independent of firewalld’s own and survive its reloads - that is the claim most worth testing first, because the phases reload firewalld during a walk. - Package assembly is not in scope and must stay out of it.
distro.Createruns on the machine running the suite, not on a node, and downloading the sources adistro.yamlnames is what a build is for. A transient DNS failure there already failed an upgrade walk atTest_00_UpgradePackage; blacking out the nodes neither causes nor fixes that. Pre-warming the download cache is a separate idea.
TODO: Direct loading of Skopeo oci-archive in apply, prepare, and load
This document records the plan and considerations for supporting direct ingestion of Skopeo oci-archive (and OCI layout directories) in commands like cargoship apply, cargoship prepare, and cargoship pull/load.
Problem Statement
When operating in an air-gapped network, artifacts are often moved across security boundaries via removable media using tools like Skopeo:
skopeo copy docker://registry.example.com/org/package:v1.0.0 oci-archive:package.tar:v1.0.0
While cargoship import converts this oci-archive file into a native Cargoship package (cargoship-*.tar.zst), users may want to pass package.tar directly to cargoship apply package.tar --cluster cluster.yaml without an explicit intermediate step.
However, an oci-archive can carry multiple manifests or image indices within a single archive (e.g. multi-arch builds or archives created by appending several images). Cargoship needs a deterministic way to select the desired package.
Resolution Plan
1. Identifying the Source
In pkg/utils/common.go:IdentifySource:
- When given a
.taror a directory that contains anoci-layoutandindex.json, distinguish between a native Cargoship uncompressed package (distro.yamlat root) and an OCI Image Layout (oci-layoutat root). - Support explicit prefixes if necessary, e.g.
oci-archive:PATH[:TAG_OR_DIGEST].
2. Disambiguation / Package Selection
When an oci-archive contains multiple manifests in index.json:
- Single manifest/index: If
index.jsoncontains exactly one root entry (or one index that points to package manifests), auto-select it. - Reference/Tag matching: Allow specifying the tag or digest, e.g.
oci-archive:package.tar:v1.0.0or via--tag/--reference. - Target Architecture: Match the platform/architecture requested by
--archor the host architecture against the descriptors inindex.json. - Ambiguity Error: If multiple distinct packages match without a clear tag or architecture selector, abort and list the available tags/digests in the archive (similar to how
pullOCI/constructIndexErrorreports candidates), directing the user to disambiguate or usecargoship import.
3. Execution Pipeline
When detected in pkg/distro/load.go:
- Unpack/mount the OCI layout into a temporary directory or read directly using ORAS
oci.NewFromTar. - Resolve the target manifest descriptor.
- Extract layer blobs annotated with
org.opencontainers.image.titleinto a temporary directory layout. - Delegate to
layout.LoadFromDirso lifecycle phases consume the resultingDistroLayouttransparently.
TODO: unit tests for the option-gated branches in pkg/action
This is a punch list, not a decision record like the choice-*.md files beside it. It exists because a coverage review of the repo flagged pkg/action (apply.go, prepare.go, reset.go, kube-config.go, engine_config_sync.go) at 0% under go test -short, and the obvious response – write unit tests for the whole package – turned out to be the wrong scope once test/e2e/cluster/cluster_lifecycle_test.go was checked. Read that file’s comments before touching this list; they already say more about the intended coverage boundary than this document repeats.
What is already covered, and why more of the same is not the ask
cluster_lifecycle_test.go calls action.NewPrepare, action.NewApply, action.NewReset, and action.NewKubeConfig directly – not through the CLI – against a real bootloose cluster, and asserts the cluster actually comes up healthy afterward. That is strictly stronger evidence for “does this action wire the right phases together” than a unit test asserting a phase list by name, because it also proves the phases it wires do the right thing together on real hosts. The comment on Test_ZZ2_ApplyIsIdempotent says this outright: “the phase tests build each phase themselves… this is the only step that exercises action.NewApply’s own wiring.” Duplicating that with a mocked-out phase.Manager would not close a correctness gap; it would just re-assert something already proven, at a lower bar of evidence.
Partly done since this was written. pkg/action/apply_test.go now exists. newTestApply builds an apply action the way a command does and TestApplyChecksRunBeforeTheLock, TestApplyRefusesADowngradeBeforeTheLock and TestApplyPassesAllowDowngrade assert over the assembled phase list without running a phase or touching a host. That settles steps 3 and 5 below – the seam the list calls the real blocker turned out to exist, so no choice-*.md is owed and no interface had to be introduced. apply.go is no longer at 0% under -short; prepare.go, reset.go, kube-config.go and engine_config_sync.go still are, and the option combinations enumerated below are still unvaried. Follow apply_test.go’s pattern for the rest.
The 0% figure from go test -short is consequently misleading on its own. It does not mean pkg/action is untested – it means pkg/action has no coverage under the fast suite, because its only exercise lives in mage test:endToEndCluster, which needs Docker and is skipped under -short. go test -short measures the wrong thing here; a package can legitimately have all its correctness coverage in the cluster suite and still show 0% in the fast run. Don’t let a future coverage pass flag this package again without checking test/e2e/cluster first, the way this document had to.
The actual gap: one option combination per action, not the whole package
Every action in the lifecycle test runs with a single, fixed set of options:
Test_01_Prepare—ModifyHosts,ModifyFirewall,ModifyModulesalltrue. Thefalsepath for each is never run.Test_ZZ2_ApplyIsIdempotent—ModifyHosts,ModifyFirewall,UpdateKubeConfig,LabelNodesalltrue.DisableDowngradeCheckis never set totrue, so the downgrade-check phase is never omitted by a test, only ever included.Test_1_Reset—NoWait: true, NoDrain: truealways. The wait-for-drain and actual-drain branches never run in this suite.engine_config_sync.go’s action has no call site anywhere intest/e2e/cluster. That grep has now been run, and the answer is a split one: the phases are covered –71_engine_config_sync_controller_test.goand72_engine_config_sync_worker_test.goassertEngineConfigSyncControllerandEngineConfigSyncWorkeracross the apply, join and upgrade walks – butaction.NewEngineConfigSync, which wires them together, is called only fromcmd/install_engine_config_sync.go. So its wiring has no coverage of any kind, fast or slow, and it is the first thing on this list.
So the gap is narrow and specific: the conditional wiring inside each action – “if this flag is set, add this phase; if not, don’t” – is proven for exactly one combination per action, and the untested combinations fail silently. A regression in, say, the DisableDowngradeCheck branch of apply.go would not be caught by the cluster suite as it is currently written, because that suite never sets the flag the other way.
What to actually do about it
- Close the
engine_config_syncgap. The grep above has been run and the gap is real: the phases are covered,action.NewEngineConfigSync’s own wiring is not. - Read
pkg/action/apply.go,prepare.go,reset.goend to end and enumerate every option field that changes which phases get added, not just the ones named above from a first pass. - Write narrow, table-driven unit tests per action, following
newTestApplyinpkg/action/apply_test.go: one table entry per option flag, asserting the phase list differs only in the way that flag controls, for every flag identified in step 2 – not the whole option struct at once, and not by asserting the actions produce a working cluster, which is what the e2e suite is for.
Do not expand this into a general push for pkg/action coverage. The scope is the option-gated branches the cluster suite doesn’t already vary, nothing more.
TODO: registry-backed unit tests for pkg/coci
This is a punch list, not a decision record like the choice-*.md files beside it. A coverage pass on pkg/coci, pkg/oci/archive (since removed with the containerd exporter) and pkg/oci/platform split the remaining gaps into three tiers. Tier 1 (pure helpers: layers.GetAllLayerTypes, GetOCICacheModifier) and Tier 2 (the layer-assembly logic in pull.go and fetch.go’s free-function FetchDistroYAML, all fakeable with an in-memory content.Fetcher and a hand-built *oci.Manifest – no registry involved) are both done; see pkg/coci/common_test.go, pkg/coci/pull_test.go, pkg/coci/layers/common_test.go. This document is the remainder: Tier 3, the functions that need a real (or in-memory) OCI registry to exercise meaningfully.
What’s left, and why it needs a registry
These all call through *Remote (an *oci.OrasRemote wrapper) to a content.Fetcher/oras.Target backed by an actual repository – resolving tags, fetching roots, walking indexes against real registry HTTP semantics – which the Tier 2 fake fetcher does not attempt to reproduce faithfully:
(*Remote).PullDistro(pull.go) – drivesoras.Copy/CopyToTargetagainst a real target; needs a populated registry to pull from and afile.Storedestination on disk.(*Remote).AssembleLayers(a thin wrapper around the already-tested free function) – the wrapper logic itself (FetchRootthen delegate) is what’s untested, not the delegated logic. Its two siblings,FetchDistroYAMLandFetchImagesIndex, were deleted as unused rather than tested.PushPackage(push.go) – pushes alayout.DistroLayoutto a target and updates the index; this is the inverse of pull and needs the same setup.
How to approach it
Follow the pattern already established in pkg/coci/push_test.go / fetch_test.go: test.SetupInMemoryRegistry(t) (documented in docs/agent/choice-in-memory-oci-registry.md) plus the local helpers testRemote, pushManifest/pushManifestTagged, pushBlob, readIndex.
For PullDistro/PushPackage, you’ll need a real layout.DistroLayout – reuse writeChecksummedPackage (pkg/packager/layout/package_test.go) or a similar minimal on-disk fixture rather than inventing a new one, since it already produces a package that passes validateDistroIntegrity.
Scope
Don’t expand this into re-covering Tier 1/2 logic through a registry “for realism” – that’s duplicated effort for the same reason pkg/action’s punch list (todo-pkg-action-unit-tests.md) says not to re-prove wiring the e2e suite already proves. The value here is specifically the registry-interaction code paths (auth, tagging, index read-modify-write against a real target, oras.Copy semantics) that the Tier 2 fakes cannot exercise.
Changelog
0.37.0 (2026-10-27)
⚠ BREAKING CHANGES
- remove the OpenTofu provider now that it lives in its own repository (#953)
Features
- examples: commit release image manifests and verify them against upstream (#955) (9e7f295)
- examples: expose the vSphere charts’ vCenter, scheduling and CA settings (#956) (2d282ea)
Bug Fixes
- ci: stage publish-example packages off the runner’s /tmp (#952) (43c72ba)
- microvm: check for a live fleet before sizing it against free memory (#959) (c840897)
- phase: leave a sibling install phase’s engine files alone (#960) (59cb289)
- phase: unmount under an engine path before removing it (#961) (c23db44)
Refactoring
Miscellaneous
- deps: update fast-uri to 3.1.9 (#954) (8003ab9)
- examples: regenerate the vSphere examples for the new values surface (#957) (3a997aa)
Build
- deps: bump almalinux/10-base from
ed618catof72ef25in /containers/ansible (#949) (f93b70e) - deps: bump almalinux/10-base from
ed618catof72ef25in /containers/ubi (#948) (4971bbc)
0.36.0 (2026-10-27)
⚠ BREAKING CHANGES
- drop mdbook golang docs (#914)
Features
- api: read a chart manifest wrapper carrying enabled, images, and values (#944) (ad0f036)
- examples: declare each chart’s images under the chart that needs them (#946) (2cd97f8)
- phase: keep a disabled chart’s images out of the upload (#945) (9960e00)
Bug Fixes
Refactoring
- coci: import zarf’s src/pkg/oci instead of defenseunicorns/pkg/oci (#897) (cb6e117)
- microvm: export the fleet harness as test/microvm (#920) (742dd77)
- provider: export the two symbols the OpenTofu provider needs across a module boundary (#913) (d72b98e)
Documentation
Miscellaneous
CI/CD
Build
- deps: bump github.com/andybalholm/brotli from 1.2.6 to 1.2.7 (#936) (c09e093)
- deps: bump github.com/docker/cli from 29.9.1+incompatible to 29.9.2+incompatible in the docker group (#903) (f4fe458)
- deps: bump github.com/docker/docker-credential-helpers from 0.9.10 to 0.9.12 in the docker group (#926) (fb49ebc)
- deps: bump github.com/go-openapi/strfmt from 0.27.3 to 0.27.4 in the openapi group (#898) (6cf315a)
- deps: bump github.com/go-openapi/validate from 1.0.0 to 1.0.1 in the openapi group (#928) (31240db)
- deps: bump github.com/google/certificate-transparency-go from 1.3.3 to 1.4.0 (#905) (669ef4c)
- deps: bump github.com/huandu/xstrings from 1.6.2 to 1.6.3 (#906) (fedcf2b)
- deps: bump github.com/klauspost/pgzip from 1.2.7 to 1.3.0 (#894) (1f3f7e3)
- deps: bump github.com/minio/minlz from 1.2.2 to 1.2.3 (#900) (89738f9)
- deps: bump github.com/shopspring/decimal from 1.5.0 to 1.6.0 (#895) (99e974b)
- deps: bump github.com/sigstore/timestamp-authority/v2 from 2.1.4 to 2.1.5 in the sigstore group (#902) (888b887)
- deps: bump google.golang.org/api from 0.304.0 to 0.305.0 (#899) (e448fb5)
- deps: bump google.golang.org/api from 0.305.0 to 0.308.0 (#929) (c216bd3)
- deps: bump https://github.com/astral-sh/ruff-pre-commit from v0.17.1 to 0.17.2 (#924) (c021ebb)
- deps: bump j178/prek-action from 3.0.1 to 3.0.2 in the core group (#901) (970de60)
- deps: bump j178/prek-action from 3.0.2 to 3.1.0 in the core group (#907) (0a2dbf3)
- deps: bump library/alpine from
294b683tobe26560in /containers/base (#921) (f44d214) - deps: bump library/debian from
a29215fto62bd423in /containers/deb (#923) (5c6ce07) - deps: bump oras.land/oras-go/v2 from 2.6.4 to 2.7.0 (#930) (7c8d680)
- deps: bump the cloud-kms group across 1 directory with 2 updates (#932) (e12c6bc)
- deps: bump the core group with 4 updates (#931) (2185d3a)
- deps: bump the sigstore group with 3 updates (#925) (46f5926)
0.35.0 (2026-10-21)
⚠ BREAKING CHANGES
- phase: trim the Manager surface nothing uses (#892)
- api: drop unreferenced methods and the two SetPath cul-de-sacs (#889)
- pkg: drop dead symbols across phase, coci, lint and utils (#888)
- phase: remove the NoWait and Force globals nothing read (#886)
- cmd: remove the –fapolicyd flag and config key nothing read (#885)
- release: assemble the provider artifact with oras-go (#645)
- tofu: converge a cluster through the cargoship_cluster resource (#618)
Features
- docs: generate a provider reference from the tofu schemas (#644) (a7da3dd)
- package: import an OCI Image Layout as a native package (#876) (c5b99d3)
- phase: write fapolicyd allow rules for a package’s executables (#875) (935faf6)
- tofu: converge a cluster through the cargoship_cluster resource (#618) (60f7ae1)
- tofu: define profiles and per-host overrides in the configuration (#622) (cd058b4)
- tofu: override package values and route phase logs to tofu (#643) (e922ae2)
- tofu: remove a single host from a cluster (#646) (364af84)
Bug Fixes
- cmd: remove the –fapolicyd flag and config key nothing read (#885) (dc0ea26)
- config: drop the config sections nothing reads (#879) (54143c4)
- distro: release the etcd membership before deleting an upstream controller (#864) (82f6ef6)
- phase: drain a node by the name the cluster calls it (#868) (8bb9c8c)
- phase: prune the upload manifest when a stale upload is removed (#863) (079af70)
- phase: remove the NoWait and Force globals nothing read (#886) (d97e7e6)
Refactoring
- action: report what a constructor cannot build, and name the read-only run (#617) (7cbd141)
- api: drop unreferenced methods and the two SetPath cul-de-sacs (#889) (08afc79)
- cmd: drop unreachable logging plumbing and two inert flags (#887) (a0139a8)
- drop four unused helpers from cmd and pkg/utils (#883) (6bc7f33)
- internal: drop the unused split package (#884) (a1d7ef9)
- node: drop the status helpers nothing calls (#882) (15170f2)
- phase: share the node restart steps four phases each carried (#890) (5c6a535)
- phase: trim the Manager surface nothing uses (#892) (921daa1)
- pkg: drop dead symbols across phase, coci, lint and utils (#888) (36f48af)
- pkg: drop two constructors nothing calls (#881) (9d03ced)
- release: assemble the provider artifact with oras-go (#645) (2270546)
Documentation
- config: name every setting in the sample config (#878) (3a1ef1e)
- example: add a terragrunt layout and guide for the provider (#621) (f0ff407)
- stop advertising cloud KMS key references on sign (#877) (ab5c211)
CI/CD
Build
- deps: bump github.com/gabriel-vasile/mimetype from 1.4.15 to 1.4.16 (#874) (122b4e2)
- deps: bump github.com/zarf-dev/zarf from 0.87.0 to 0.88.0 (#872) (237c3d8)
- deps: bump https://github.com/astral-sh/ruff-pre-commit from v0.17.0 to 0.17.1 (#869) (f7d2ea5)
- deps: bump https://github.com/golangci/golangci-lint from v2.14.0 to 2.15.0 (#870) (466581a)
- deps: bump oras.land/oras-go/v2 from 2.6.3 to 2.6.4 (#871) (dc1c159)
0.34.0 (2026-10-18)
⚠ BREAKING CHANGES
- deps: cut the binary from 100 MiB to 39 MiB, and drop keyless signing (#847)
- ansible: remove the zarf collection, now released from its own repository (#823)
Features
- distro: add selector.when to make a file’s upload conditional on a value (#834) (a4d14f5)
- distro: bring the container runtime up for an upstream apply (#841) (7f8a965)
- phase: warn when hosts run fapolicyd and the package carries no rules (#843) (d249e82)
Bug Fixes
- cli: report a missing distro instead of panicking on a nil action (#838) (d410145)
- distro: let an upstream reset run over a host whose packages never installed (#845) (f367807)
- phase: keep a targeted reset to the hosts it was given (#840) (99871a5)
- phase: stop leaving an rpm-selected executable under its staging name (#833) (fdfeb06)
Refactoring
Documentation
Miscellaneous
- ansible: remove the zarf collection, now released from its own repository (#823) (adcd344)
- example: regenerate the examples for selector.when (#836) (9d1ee69)
- example: regenerate the upstream examples for the fapolicyd rules (#844) (67b4258)
CI/CD
- add gitignore for build (#830) (51235ae)
- dependabot: group the gomod dependency trees that move together (#831) (2efed53)
Build
- deps: bump github.com/docker/cli from 29.9.0+incompatible to 29.9.1+incompatible in the docker group (#854) (f8eb17e)
- deps: bump github.com/go-chi/chi/v5 from 5.3.2 to 5.4.0 (#818) (50270e5)
- deps: bump github.com/go-chi/chi/v5 from 5.4.0 to 5.5.0 (#855) (82e60d2)
- deps: bump github.com/go-jose/go-jose/v4 from 4.1.5 to 4.1.6 (#860) (46e9efa)
- deps: bump github.com/godbus/dbus/v5 from 5.2.2 to 5.2.4 (#859) (34f0c3c)
- deps: bump github.com/grpc-ecosystem/grpc-gateway/v2 from 2.31.0 to 2.32.0 (#858) (0357c3a)
- deps: bump github.com/klauspost/compress from 1.20.1 to 1.20.3 (#816) (070d81c)
- deps: bump github.com/klauspost/cpuid/v2 from 2.4.0 to 2.4.1 (#826) (e1d0600)
- deps: bump github.com/Microsoft/go-winio from 0.6.3 to 0.6.4 (#857) (0aa496e)
- deps: bump github.com/nwaples/rardecode/v2 from 2.4.3 to 2.4.4 (#819) (3d0f8a2)
- deps: bump github.com/nwaples/rardecode/v2 from 2.4.4 to 2.4.5 (#827) (c1128b1)
- deps: bump github.com/open-policy-agent/opa from 1.21.1 to 1.21.2 (#817) (9dc23d6)
- deps: bump github.com/sigstore/sigstore from 1.11.0 to 1.11.1 in the sigstore group (#853) (9f5d8f6)
- deps: bump google.golang.org/api from 0.301.0 to 0.302.0 (#829) (a3caf75)
- deps: bump google.golang.org/api from 0.302.0 to 0.304.0 (#856) (b6f7c7f)
- deps: bump https://github.com/sirosen/texthooks from 0.7.1 to 1.0.0 (#851) (845e4ae)
- deps: bump oras.land/oras-go/v2 from 2.6.2 to 2.6.3 (#828) (0e241d1)
- deps: bump the core group with 2 updates (#820) (80e8e5b)
- deps: bump the core group with 3 updates (#861) (5c0cad7)
- deps: bump the opentelemetry group with 6 updates (#852) (1f94b91)
0.33.0 (2026-10-14)
⚠ BREAKING CHANGES
- record the colonel-byte.dev API group move and its transition window (#798)
- schema: rename the generated schema files off the zarf- prefix (#796)
- api: drop the Zarf prefix from the owned API types (#794)
- api: move api/zarf.dev to api/colonel-byte.dev (#793)
Features
- api: accept colonel-byte.dev/v1alpha1 with Cluster and Distro kinds (#795) (3b1ac3c)
- cli: add –receipt-path to the lifecycle commands (#781) (63ceeaa)
- cli: add redact-receipt for sharing a run receipt (#783) (d03cc39)
- cli: print a per-host summary table and document run receipts (#784) (39cc614)
- cli: wire –receipt-path into the config file (#815) (9da1dc6)
- phase: report per-host changes and per-phase duration (#779) (6cf4e53)
- phase: report which hosts changed, with diffs where the file was read (#782) (5257c0e)
- receipt: add the run receipt document and its writer (#780) (4c81c0e)
Refactoring
- api: drop the Zarf prefix from the owned API types (#794) (ed30c85)
- api: move api/zarf.dev to api/colonel-byte.dev (#793) (738c7dc)
- schema: rename the generated schema files off the zarf- prefix (#796) (29f5f69)
Documentation
Miscellaneous
Build
- deps: bump github.com/aws/aws-sdk-go-v2/service/ecrpublic from 1.47.3 to 1.47.4 (#806) (b05ecf7)
- deps: bump github.com/aws/aws-sdk-go-v2/service/kms from 1.61.3 to 1.61.4 (#808) (9bb808b)
- deps: bump github.com/fvbommel/sortorder from 1.2.0 to 1.2.1 (#804) (8007542)
- deps: bump github.com/nwaples/rardecode/v2 from 2.4.2 to 2.4.3 (#812) (8ce9758)
- deps: bump github.com/prometheus/client_golang from 1.24.1 to 1.25.0 (#811) (632a2e4)
- deps: bump github.com/prometheus/common from 0.72.0 to 0.72.1 (#805) (782e76a)
- deps: bump github.com/prometheus/procfs from 0.21.1 to 0.22.0 (#802) (ec06714)
- deps: bump github.com/pterm/pterm from 0.12.83 to 0.12.84 (#810) (70e2beb)
- deps: bump golang.org/x/vuln from 1.8.0 to 1.9.0 (#803) (c8dc4c8)
- deps: bump ossf/scorecard-action from 2.4.4 to 2.5.0 in the core group (#807) (31f6b39)
0.32.1 (2026-10-13)
Documentation
- agent: prefer prek over pre-commit (#731) (154af7d)
- agent: require an updater or dependabot entry for every pin (#745) (3594775)
CI/CD
- actions: exempt dependabot pull request titles from the 100-char cap (#730) (9edcc87)
- bot: raise the dependabot open pull request limit for gomod (#739) (2762460)
Build
- deps: bump cloud.google.com/go/auth from 0.24.1-0.20261001053825-dbc26066f70a to 0.24.1 (#761) (911c84b)
- deps: bump github.com/aws/aws-sdk-go-v2/…/accept-encoding from 1.13.19 to 1.13.20 (#722) (9a0471c)
- deps: bump github.com/aws/aws-sdk-go-v2/config from 1.33.6 to 1.33.9 (#760) (276cd89)
- deps: bump github.com/aws/aws-sdk-go-v2/feature/ec2/imds from 1.20.1 to 1.20.2 (#725) (277295e)
- deps: bump github.com/aws/aws-sdk-go-v2/internal/v4a from 1.5.4 to 1.5.5 (#752) (3288f6c)
- deps: bump github.com/aws/aws-sdk-go-v2/service/ecr from 1.66.1 to 1.66.4 (#753) (b959d02)
- deps: bump github.com/aws/aws-sdk-go-v2/service/ecrpublic from 1.47.1 to 1.47.3 (#724) (78d7da3)
- deps: bump github.com/aws/aws-sdk-go-v2/service/internal/presigned-url from 1.14.4 to 1.14.5 (#741) (f3e4180)
- deps: bump github.com/aws/aws-sdk-go-v2/service/kms from 1.61.1 to 1.61.3 (#727) (cd857c8)
- deps: bump github.com/aws/aws-sdk-go-v2/service/signin from 1.10.1 to 1.10.4 (#764) (f2dc08e)
- deps: bump github.com/aws/aws-sdk-go-v2/service/sso from 1.38.1 to 1.38.3 (#772) (bcfb114)
- deps: bump github.com/aws/aws-sdk-go-v2/service/ssooidc from 1.43.1 to 1.43.4 (#746) (6a4953f)
- deps: bump github.com/aws/aws-sdk-go-v2/service/sts from 1.51.1 to 1.51.3 (#775) (fbbe552)
- deps: bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.23.2 to 1.23.3 (#749) (5600d98)
- deps: bump github.com/docker/cli from 29.8.2+incompatible to 29.9.0+incompatible (#743) (c2684ab)
- deps: bump github.com/docker/docker-credential-helpers from 0.9.9 to 0.9.10 (#766) (41070d9)
- deps: bump github.com/emicklei/go-restful/v3 from 3.13.0 to 3.14.0 (#755) (f8d0c6a)
- deps: bump github.com/fxamacker/cbor/v2 from 2.9.4 to 2.9.6 (#771) (f3eb304)
- deps: bump github.com/go-git/go-git/v5 from 5.19.2 to 5.19.3 (#770) (cac2936)
- deps: bump github.com/go-openapi/runtime from 0.33.2 to 0.33.3 (#728) (1478337)
- deps: bump github.com/googleapis/enterprise-certificate-proxy from 0.3.22 to 0.3.23 (#776) (0344bae)
- deps: bump github.com/hashicorp/go-plugin from 1.7.0 to 1.8.0 (#765) (98217cb)
- deps: bump github.com/hashicorp/terraform-registry-address from 0.4.0 to 0.5.0 (#756) (028e75f)
- deps: bump github.com/hashicorp/terraform-svchost from 0.1.1 to 0.2.1 (#773) (63fa042)
- deps: bump github.com/klauspost/pgzip from 1.2.6 to 1.2.7 (#758) (55c7846)
- deps: bump github.com/mattn/go-colorable from 0.1.15 to 0.1.16 (#768) (82b9154)
- deps: bump github.com/mattn/go-runewidth from 0.0.30 to 0.0.31 (#769) (4d7b347)
- deps: bump github.com/moby/moby/client from 0.6.0 to 0.6.2 (#723) (32bbcc6)
- deps: bump github.com/nwaples/rardecode/v2 from 2.4.1 to 2.4.2 (#744) (791bb33)
- deps: bump github.com/olekukonko/ll from 0.1.8 to 0.1.9 (#762) (8a53e55)
- deps: bump github.com/open-policy-agent/opa from 1.21.0 to 1.21.1 (#777) (dcdb2df)
- deps: bump github.com/pjbgf/sha1cd from 0.6.0 to 0.7.0 (#747) (fc26850)
- deps: bump github.com/shopspring/decimal from 1.4.0 to 1.5.0 (#754) (1ae2caa)
- deps: bump github.com/sigstore/timestamp-authority/v2 from 2.1.3 to 2.1.4 (#751) (cebb46a)
- deps: bump github.com/vektah/gqlparser/v2 from 2.5.58 to 2.5.62 (#774) (72ef884)
- deps: bump go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc from 0.71.0 to 0.72.0 (#759) (4881a09)
- deps: bump golang.org/x/mod from 0.41.0 to 0.42.0 (#778) (1955983)
- deps: bump google.golang.org/api from 0.299.0 to 0.301.0 (#721) (e723ad4)
- deps: bump https://github.com/astral-sh/ruff-pre-commit from v0.16.10 to 0.17.0 (#740) (13ffece)
- deps: bump sigs.k8s.io/controller-runtime from 0.25.1 to 0.25.2 (#767) (37efa93)
- deps: bump the core group with 6 updates (#750) (de03019)
0.32.0 (2026-10-12)
⚠ BREAKING CHANGES
- example: release floor for kubernetes engines (#681)
- tofu: read a fleet’s facts through the provider (#616)
Features
- cmd: add a verify command for package signatures (#691) (648d920)
- distro: add spec.config.os.selinux to the package schema (#661) (3525140)
- distro: reset a named subset of hosts (#707) (360677d)
- examples: add the recommended SELinux settings to the distro templates (#665) (ffb5cd8)
- examples: bake a CNI into the upstream examples (#682) (b5e22cc)
- examples: render upstream-cilium, upstream-canal, upstream-flannel (#683) (790de49)
- magefiles: add dev:vmUp and dev:vmDown targets for the VM fleet (#700) (e9d9d53)
- microvm: add a rootless qemu VM fleet for local Enterprise Linux testing (#699) (d05b4ab)
- microvm: add an opt-in Ubuntu guest for a mixed fleet (#720) (c343aed)
- phase: apply the package’s SELinux policy on enforcing hosts (#662) (b2bd693)
- phase: remove the SELinux policy cargoship applied on reset (#663) (6b84de9)
- tofu: read a fleet’s facts through the provider (#616) (4209a45)
Bug Fixes
- deps: bump x/net to v0.61.0 and the go directive to 1.27.2 (#687) (9001e9b)
- distro: match a host to its node by any identifier they share (#711) (ff4166a)
- distro: remove the engine’s directories on uninstall (#712) (3c6ca37)
- distro: render the engine’s node-ip from the host’s private address (#717) (0ff3770)
- distro: repoint the controllers a removal keeps, on the right port (#718) (1258ed5)
Refactoring
Documentation
- agent: record the SELinux policy design decisions (#664) (4f6c5a4)
- agents: record issue tracking, stacked PR, and README upkeep rules (#675) (8ffec03)
- correct drift in README, docs/, and the agent rules files (#692) (33b821f)
- examples: document the upstream CNI flavors (#684) (1da789c)
Miscellaneous
- example: release floor for kubernetes engines (#681) (cdcd038)
- examples: regenerate the example packages (#666) (d70614b)
CI/CD
Build
- deps: bump github.com/go-openapi/analysis from 1.0.0 to 1.0.1 (#648) (3f5b368)
- deps: bump github.com/go-openapi/strfmt from 0.27.2 to 0.27.3 (#651) (0caf1ce)
- deps: bump github.com/Microsoft/go-winio from 0.6.3-0.20251027160822-ad3df93bed29 to 0.6.3 (#656) (25c5250)
- deps: bump github.com/minio/minlz from 1.2.0 to 1.2.2 (#657) (1c4604c)
- deps: bump github.com/oklog/run from 1.1.0 to 1.2.0 (#658) (a055faf)
- deps: bump github.com/phuslu/lru from 1.0.18 to 1.0.24 (#652) (7e43c50)
- deps: bump github.com/spiffe/go-spiffe/v2 from 2.8.2 to 2.9.0 (#659) (d52ce3a)
- deps: bump go.opentelemetry.io/…/otlpmetricgrpc from 1.46.0 to 1.47.0 (#649) (f70a166)
- deps: bump go.opentelemetry.io/…/otlptracegrpc from 1.46.0 to 1.47.0 (#660) (662ab8b)
- deps: bump j178/prek-action from 3.0.0 to 3.0.1 in the core group (#653) (b60781b)
- deps: bump library/debian from
a99cfc5toa29215fin /containers/deb (#655) (b43f33d) - deps: bump sigs.k8s.io/kustomize/api from 0.21.1 to 0.21.2 (#650) (81e67a4)
0.31.0 (2026-10-06)
⚠ BREAKING CHANGES
- ansible: add read-only zarf modules and the roles that wrap them (#630)
Features
- ansible: add read-only zarf modules and the roles that wrap them (#630) (6021940)
- ansible: let zarf_package_inspect require a package signature (#633) (5e81e2c)
- ansible: withhold the credentials in the zarf-state Secret (#632) (6ce4a4d)
Refactoring
Documentation
Build
- deps: bump anchore/sbom-action/download-syft from 0.24.2 to 0.24.3 in the core group (#628) (fc5d082)
- deps: bump github.com/google/s2a-go from 0.1.10 to 0.1.11 (#642) (85cc30c)
- deps: bump github.com/huandu/xstrings from 1.6.1 to 1.6.2 (#626) (33d493a)
- deps: bump github.com/moby/moby/api from 1.56.0 to 1.56.1 (#640) (6efe78f)
- deps: bump github.com/mozillazg/docker-credential-acr-helper from 0.4.0 to 0.4.1 (#623) (a93fd7b)
- deps: bump github.com/pierrec/lz4/v4 from 4.1.31 to 4.1.32 (#625) (00ab3cc)
- deps: bump github.com/pierrec/lz4/v4 from 4.1.32 to 4.1.33 (#641) (b1da6c0)
- deps: bump go.opentelemetry.io/…/otelhttp from 0.71.0 to 0.72.0 (#639) (5b9aaa5)
- deps: bump go.opentelemetry.io/…/stdoutmetric from 1.46.0 to 1.47.0 (#638) (c34887c)
- deps: bump go.opentelemetry.io/otel from 1.46.0 to 1.47.0 (#624) (bdae158)
- deps: bump sigs.k8s.io/release-utils from 0.12.4 to 0.12.5 (#627) (aeeed12)
- pre-commit: lint the collections’ Python for errors with ruff (#634) (1ae77dc)
0.30.0 (2026-10-05)
⚠ BREAKING CHANGES
- distro: upstream validate engine config against kubeadm API types (#517)
- distro: render kubeadm v1beta4 config documents (#511)
- ansible: add the colonel_byte.zarf collection (#599)
- zarfmod: answer Ansible as zarf_init and zarf_package_deploy (#598)
Features
- ansible: add the colonel_byte.zarf collection (#599) (592ffb3)
- bootstrap single-controller upstream cluster (#512) (9c4abc6)
- distro/upstream: join additional controllers via kubeadm HA path (#514) (45fc6b6)
- distro: render kubeadm v1beta4 config documents (#511) (f5a1018)
- distro: upstream validate engine config against kubeadm API types (#517) (a656ab2)
- distro: upstream: reset with kubeadm reset (#516) (7996bfe)
- distro: upstream: upgrade with kubeadm upgrade (#515) (070a758)
- docs: generate a reference per Ansible collection (#602) (7b84a08)
- release: build, package and smoke-test the zarf module files (#601) (4121315)
- schema: mark the properties that can carry a secret (#606) (1efa64a)
- tofu: publish the provider to ghcr.io as an OpenTofu mirror (#609) (7a687f9)
- upstream: apply a CNI manifest after bootstrap (#513) (bea8299)
- zarfmod: answer Ansible as zarf_init and zarf_package_deploy (#598) (5b96891)
Bug Fixes
Refactoring
- cmd: build the released binary from cmd/cargoship (#596) (8127dbe)
- containers: base the ubi image on almalinux 10-base (#593) (e3b8574)
- containers: remove labels from docker files (#594) (abd7476)
Documentation
Miscellaneous
Build
- deps: bump github.com/andybalholm/brotli from 1.2.5 to 1.2.6 (#612) (27d3f41)
- deps: bump github.com/Azure/azure-sdk-for-go/sdk/internal from 1.12.0 to 1.13.0 (#613) (08dd9c6)
- deps: bump github.com/go-git/go-billy/v5 from 5.9.1 to 5.9.2 (#591) (89ae5ac)
- deps: bump github.com/goccy/go-json from 0.11.1 to 0.11.2 (#611) (671a196)
- deps: bump github.com/zarf-dev/zarf from 0.86.0 to 0.87.0 (#615) (85538ef)
- deps: bump go.opentelemetry.io/proto/otlp from 1.11.0 to 1.11.1 (#614) (84c19d0)
- tofu: add the provider binary and record where it lives (#605) (d7f6126)
0.29.0 (2026-10-02)
⚠ BREAKING CHANGES
- remove 32bit support (#590)
- reset: remove the hosts, firewall, and fapolicyd flags it never read (#573)
- enable merge queue on main branch (#552)
Features
- create: add –file-override to mirror file downloads (#548) (814694e)
- create: record per-file provenance for overridden downloads (#549) (ac27c42)
- examples: extract rke2 files straight from the release tarball (#567) (0423ca0)
- render containerd config and hosts.toml, import bundled images (#510) (0bf420e)
Bug Fixes
- cmd: check inputs before the –confirm gate and report each failure once (#571) (0b0f17f)
- config: expand a leading ~ in age identity and recipients file paths (#586) (931ed0a)
- create: validate extracted-file shasum, not the downloaded archive (#566) (7ce6811)
- lang: correct typos and miswording in command help and phase titles (#575) (7e3029b)
- reset: remove the hosts, firewall, and fapolicyd flags it never read (#573) (2a3cf37)
Refactoring
- cmd: collapse the duplicate flag-name constants onto one each (#574) (7585b71)
- magefiles: split large package files by topic (#547) (82ea3b4)
- phase: drop the unused InstallFapolicy phase (#577) (d90456b)
Documentation
- lang: add long help for the core lifecycle and package commands (#572) (6e47965)
- readme: add installation instructions and –confirm to the examples (#570) (69a763a)
Miscellaneous
- engine: update rke2 and k3s (#563) (26b1bd0)
- examples: generate new distro (#564) (cee3ea2)
- examples: regenerate the rke2 packages from the new template (#568) (18969dd)
CI/CD
- enable merge queue on main branch (#552) (3beec00)
- ignore crypto11 in Dependabot, its module path moved upstream (#587) (d5e37e7)
- lint: run golangci-lint at the version the hooks pin (#551) (ee97211)
- re-work dependabot config (#557) (934972b)
- remove 32bit support (#590) (d7a59b8)
- update Dependabot schedule to daily for gomod and pre-commit (#578) (8b2df63)
Build
- deps: bump almalinux/10-base from
f110feetoed618cain /containers/ansible (#580) (15c57d0) - deps: bump almalinux/10-minimal from
ee053beto1c0191din /containers/ubi (#579) (829a09e) - deps: bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.23.1 to 1.23.2 (#585) (43b6dbd)
- deps: bump github.com/AzureAD/microsoft-authentication-library-for-go from 1.10.0 to 1.10.1 (#584) (ff7f71b)
- deps: bump github.com/buildkite/go-pipeline from 0.18.0 to 0.18.1 (#581) (4b13f5a)
- deps: bump github.com/docker/cli from 29.8.1+incompatible to 29.8.2+incompatible (#555) (2d580cf)
- deps: bump github.com/go-openapi/jsonreference from 1.0.2 to 1.0.3 (#562) (559b1b1)
- deps: bump github.com/goccy/go-json from 0.10.6 to 0.11.1 (#559) (6e5d1de)
- deps: bump github.com/googleapis/gax-go/v2 from 2.26.0 to 2.26.2 (#558) (42275b3)
- deps: bump github.com/grpc-ecosystem/grpc-gateway/v2 from 2.30.0 to 2.31.0 (#560) (6140c26)
- deps: bump github.com/nao1215/markdown from 1.1.0 to 1.1.1 (#561) (d9683bd)
- deps: bump github.com/pb33f/go-yaml from 0.1.0 to 0.1.1 (#589) (54a52da)
- deps: bump github.com/pb33f/ordered-map/v2 from 2.3.1 to 2.3.2 (#582) (5b0b1e2)
- deps: bump github.com/pierrec/lz4/v4 from 4.1.30 to 4.1.31 (#583) (ad1e0d0)
- deps: bump sigs.k8s.io/kustomize/kyaml from 0.21.1 to 0.21.2 (#588) (70668f0)
0.28.2 (2026-09-30)
Features
Bug Fixes
Documentation
Build
- deps: bump https://github.com/golangci/golangci-lint from v2.13.2 to 2.14.0 in the core group (#541) (4328aab)
0.28.1 (2026-09-30)
Features
- drift: move files around to make sense (#540) (098ac1b)
- install and pin upstream kubeadm packages (#509) (dbe72cb)
Documentation
CI/CD
- examples: include version for upstream (#536) (6468cbf)
- examples: update versions (#537) (dd0959b)
- releaser: update config to address build issues (#539) (10ef7cc)
Build
- deps: bump github.com/ProtonMail/go-crypto from 1.5.1 to 1.5.2 (#532) (0696e8a)
- deps: bump the core group with 3 updates (#533) (11c1e7f)
- deps: bump the cosign group with 15 updates (#529) (634975a)
- deps: bump the k8s group with 7 updates (#528) (f203186)
- deps: bump the misc group with 19 updates (#530) (fd2cfca)
0.28.0 (2026-09-27)
⚠ BREAKING CHANGES
Features
- add k3s single-controller topology for e2e-cluster CI (#504) (f95b31b)
- finish the apply walk, from the engine config to disconnect (#273) (8f4830c)
- generate example/upstream/distro.yaml instead of hand-writing it (#519) (853f22d)
- register upstream (kubeadm) distro type (#508) (8934c6d)
- tear the distro back off the cluster at the end of the run (#276) (2d67c45)
- update to v2 rig (#214) (b7c5440)
- walk the apply phases a second time against the installed cluster (#274) (1f04e4e)
- walk the phases a third time to upgrade the installed cluster (#275) (c13eaa3)
Bug Fixes
Refactoring
- move src/ contents to repo root (#503) (cdf9b5f)
- move test/ out of src/ and consolidate shared internal/ packages (#495) (cd771c4)
Documentation
- add src/fuzz AGENTS.md and rename FuzzParseRecipient to avoid -fuzz ambiguity (#500) (970d154)
- agent: require pre-commit before commit, drop em dash (#523) (458cc4d)
- ai disclosure (#496) (770b931)
- generate godoc and schema reference pages (#524) (157ab95)
- security: annotate Binary-Artifacts scorecard finding (#491) (705db3f)
- workflows: add per-workflow explanations under docs/workflows (#525) (d4467af)
Miscellaneous
- deps: update Kubernetes upstream pin in thirdparty-src/pins.json (#526) (e85c04d)
- regenerate example/upstream from the new generator (#520) (b4ac7c4)
CI/CD
- fuzz: address found error in parser (#506) (a41ca5f)
- publish artifacts on regular basis (#445) (89085d7)
- test-build container images on dependabot docker bumps (#499) (a76827f)
Build
- deps: bump chainguard/cgr.dev/chainguard/static from
90b7f22toa6d1bbf(#490) (b547eae) - deps: bump chainguard/cgr.dev/chainguard/static from
a6d1bbftof2787f1(#497) (c25c5e8) - deps: bump github.com/containerd/containerd/v2 from 2.4.0 to 2.4.1 (#507) (d0e9b6c)
- deps: bump k8s.io/streaming from 0.37.0 to 0.37.1 in the k8s group (#492) (b78c39e)
- deps: bump the golang group with 2 updates (#493) (24a532c)
- deps: bump the misc group with 5 updates (#494) (e9f76d2)
0.27.0 (2026-09-24)
⚠ BREAKING CHANGES
- release: add windows support (#487)
Documentation
CI/CD
0.26.2 (2026-09-23)
Documentation
- ansible: declare the module and role interfaces in the collection (#481) (c4437fe)
- ansible: generate the collection reference pages (#482) (d53be49)
- fix drift for new features (#479) (060493c)
- generate the book’s front matter from README.md and SECURITY.md (#484) (6e77cef)
CI/CD
- actions: address some issues with score-card (#485) (c57d82d)
- deps: add workflow for adding back osv-scanned toml (#480) (8286777)
- modify Cosign verification steps in .goreleaser.yaml (#477) (efbf83d)
0.26.1 (2026-09-22)
Features
- cli: add flag for overriding version (#469) (472b4b2)
- example: allow overriding pod sec (#470) (9055c02)
- examples: update pod sec overrides (#471) (42412ec)
Miscellaneous
CI/CD
- actions: fix look up (#474) (c02d2db)
- container: pin dnf packages (#468) (259f717)
- pin: update issue search (#473) (01e35fd)
Build
- deps: bump github.com/ulikunitz/xz from 0.5.16 to 0.5.17 in the misc group (#465) (b00556d)
- deps: bump the cosign group with 2 updates (#464) (7963065)
0.26.0 (2026-09-22)
⚠ BREAKING CHANGES
- address various unsafe repo settings (#446)
Features
- ansible: add container to goreleaser (#456) (25667e2)
- ansible: add heartbeat to ansible logs (#458) (7998f65)
- ansible: add the apply, prepare, reset and kube-config modules (#433) (f611cfb)
- ansible: build and publish the collection with the release (#434) (ec48f76)
- ansible: expand flags passed in from ansible (#459) (a6fa3f3)
- ansible: generate a cluster inventory from an Ansible inventory (#429) (46f6de8)
- ansible: report changed from what the phases did (#432) (732df4e)
- ansible: run the cargoship binary as an Ansible module (#430) (95763f9)
- cli: loosen check on registry mismatch when encrypted (#460) (4e59787)
- fuzz: add ansible (#436) (cbcf7c5)
- release: add rpm, deb, and apk signing (#461) (7c58b2c)
Bug Fixes
Documentation
- add ideas for inventory from-x (#437) (0707742)
- agent: refine pr description (#440) (9d5c4d6)
- agent: update docs folder agent (#441) (7017588)
- ansible: readme’s and examples (#462) (ec651cb)
- security md (#444) (aca4a58)
CI/CD
- add openssf score card run (#443) (83f657c)
- address various unsafe repo settings (#446) (d487baa)
- ansible: add version checking (#455) (d198d4b)
- codeql: add advanced setup for codeql (#428) (b9134df)
- e2e: add ansible tests (#435) (f2acff9)
- flow: add check for image drift in docker files (#442) (22d8106)
- mage: expose osv overrides (#463) (7838fd9)
- vendor: stop scoring vendored JavaScript and Python as cargoship vulns (#447) (d757666)
Build
- deps: bump github.com/docker/cli from 29.8.0 to 29.8.1 (#451) (54c34a2)
- deps: bump github.com/zarf-dev/zarf from 0.85.0 to 0.86.0 in the zarf group (#449) (ee26849)
- deps: bump library/alpine from
28bd5feto294b683in /containers/base (#466) (17025c5) - deps: bump the core group with 2 updates (#425) (d7cfa2f)
- deps: bump the core group with 3 updates (#452) (85bcdf8)
- deps: bump the cosign group with 10 updates (#453) (03eaa90)
- deps: bump the golang group with 2 updates (#450) (0c68c6a)
- deps: bump the misc group with 13 updates (#454) (8f4b952)
- deps: bump ubi9/ubi-minimal from
7b8e25ato984df0a(#448) (fdcd56c)
0.25.0 (2026-09-18)
⚠ BREAKING CHANGES
- dagger: remove dagger config (#419)
Features
Bug Fixes
Documentation
CI/CD
0.24.0 (2026-09-17)
⚠ BREAKING CHANGES
Features
- assemble: resolve values before the build runs (#403) (8c62cb5)
- cli: add validate command (#417) (d0ca618)
- engine-config-sync: report the files written in place (#405) (c52caef)
- engineconfig: validate disable values against generated addon lists (#409) (9947469)
- examples: give the example packages values files and schemas (#401) (a6ec737)
- examples: show values templating, and document it (#406) (129b4ef)
- helmvalues: add Helm-style value templating and values plumbing (#400) (a74a2af)
- helmvalues: add the Helm serialization functions sprig lacks (#402) (adafd63)
- refine schema more (#415) (459ba17)
- schema: generate schema file (#416) (b20df28)
- values: disable bundled engine addons from the inventory (#408) (92c9ba5)
- values: render the engine configuration and templated files at … (#404) (0545780)
CI/CD
- example: add schema support in-editor (#414) (47443df)
- example: expand packages (#413) (4bcb523)
- example: generate all examples (#412) (4326533)
Build
0.23.0 (2026-09-16)
⚠ BREAKING CHANGES
- change base image for non-ubi image (#399)
Features
- cluster: accept SSH keys as age keys (#391) (1323ef9)
- clustercfg: report registry credentials encrypt-file unchanged (#387) (b850c09)
- detect a host removed from the config (#340) (89278c0)
- distrocfg: record installed distro release metadata on hosts (#378) (6a15aac)
- engineconfig: add k3s and rke2 v1.37 generated engine config (#377) (2a2ffdc)
- utils: add DownloadToCache helper (#379) (fa4b120)
- vault: support age alongside ansible vault (#386) (d838275)
- vault: vendor age-keygen (#389) (4efb76c)
Miscellaneous
CI/CD
- action: disable require main pr check (#390) (a6502d2)
- add language_version tied to go.mod (#375) (6c191c5)
- change base image for non-ubi image (#399) (4d9f93a)
- dependabot: add docker registry auth (#395) (30c4623)
- dependabot: add registry1 auth (#398) (863afa8)
Build
- deps: bump sigs.k8s.io/controller-runtime from 0.25.0 to 0.25.1 (#396) (2a24295)
- deps: bump the misc group across 1 directory with 2 updates (#397) (8b57a7e)
- deps: bump ubi9/ubi-minimal from
186a94btoe5161a7(#381) (4811f95) - deps: bump ubi9/ubi-minimal from
d235f60to186a94b(#373) (3ff2627) - deps: bump ubi9/ubi-minimal from
e5161a7to7b8e25a(#393) (eae8bbc)
0.22.1 (2026-09-13)
CI/CD
Build
- deps: Bump the cosign group with 20 updates (#369) (8f00046)
- deps: Bump the misc group with 6 updates (#370) (7bc188f)
0.22.0 (2026-09-13)
⚠ BREAKING CHANGES
Features
Refactoring
CI/CD
- add code owners file (#366) (5053f9e)
- add docker images (#364) (a3a5070)
- add pre-commit checking (#365) (e19472a)
Build
- deps: Bump alpine from 3.22 to 3.24 (#368) (6f85407)
- deps: Bump the golang group 11 updates (#367) (d727bd3)
0.21.0 (2026-09-11)
⚠ BREAKING CHANGES
- kubeconfig as value (#337)
Features
- allow passing yaml blocks for helm manifests (#341) (d493a7c)
- kubeconfig as value (#337) (d718299)
- vault: add “vault decrypt” and “vault decrypt-path” (#346) (4d687ad)
- vault: add “vault encrypt-file” and “vault decrypt-file” (#347) (a3304fb)
- vault: add “vault encrypt-path” to encrypt a value in place (#345) (72eef41)
- vault: add “vault rekey” to move a config to a new vault password (#351) (84c743f)
- vault: group the vault commands under a “vault” subcommand (#344) (67c435c)
- vault: make the new rekey password optional and re-salt in place (#354) (b06854a)
- vault: resolve anchors and aliases before walking a config (#356) (dc16a42)
- vault: take several YAML paths per encrypt-path and decrypt-path run (#355) (3cc9b2e)
Bug Fixes
Refactoring
Documentation
CI/CD
- build required action always (#352) (43137fa)
- delete a pull request’s example packages on close, and sweep stale ones after 30 days (#359) (f6a922a)
- run publish-example once goreleaser has uploaded the release (#350) (90b9c13)
- update logic on clean (#360) (d42991e)
0.20.1 (2026-09-10)
Features
- add mage function for generating packages (#335) (5dd0c19)
- engine-config: remove files cargoship no longer writes (#331) (2f9de30)
- engine-config: write each engine config file with its own mode (#330) (d9fe4bf)
- make the registry proxy optional (#328) (d5da3d7)
- registry: check vault-encrypted registry values before connecting (#332) (e9b7f60)
- registry: support per-registry TLS configuration (#329) (46f6486)
- respect image compress on uploaded images (#265) (c517e71)
Bug Fixes
Refactoring
0.20.0 (2026-09-09)
⚠ BREAKING CHANGES
- rework backend selection (#264)
Features
- add dry-run support (#312) (c1a2cf4)
- assemble multi-arch packages with an OCI image index (#263) (db8d106)
- assert the firewall and the three upload phases (#272) (e5bc4c2)
- multi-arch: export an image tarball per host architecture (#269) (765dfe5)
- multi-arch: generate multi-architecture examples for rke2, k3s, and upstream (#323) (cb4d556)
- multi-arch: publish a package index per architecture and pull across architectures (#322) (93ac0d3)
- multi-arch: upload the files a host’s own architecture selects (#268) (ca40e5d)
- reject a host the package carries no architecture for (#267) (fe73d68)
- rework backend selection (#264) (7181dda)
- run the staging phases against a smaller cluster in CI (#293) (2b62d54)
- start work on multi-arch support (#261) (d5661d1)
- walk the apply phases against a live cluster, through ModifyHosts (#271) (5886029)
Bug Fixes
- export image tarballs for the package architecture (#266) (dbe65d7)
- honour –architecture on package create and package pull (#262) (4665c85)
Refactoring
- distro: drop the concrete-type switch from the uninstall phase (#318) (7e8f8ad)
- distro: move the engine version regex off Common (#317) (d2149a4)
- distro: read admin kubeconfig credentials through the distro interface (#319) (74c792a)
- example: retune the cilium defaults and unquote the sysctl values (#324) (bb06592)
Documentation
CI/CD
- add extra label for release (#295) (fb865a0)
- keep the cluster on failure and upload node diagnostics (#296) (a542a53)
- publish an example package to GHCR with setup-cargoship (#326) (a9be49d)
- unhide refactor section (#321) (f45dcd6)
- update dependabot config (#303) (920e515)
Build
- deps: Bump github.com/docker/cli from 29.7.2 to 29.8.0 (#300) (f399338)
- deps: Bump github.com/zarf-dev/zarf from 0.84.0 to 0.85.0 (#297) (8ffbd14)
- deps: Bump sigs.k8s.io/controller-runtime from 0.24.1 to 0.25.0 (#298) (8d6e542)
- deps: Bump the cosign group with 15 updates (#314) (7fca7b6)
- deps: bump the cosign group with 9 updates (#301) (b3cd234)
- deps: Bump the docker group with 2 updates (#316) (22ffa2b)
- deps: Bump the golang group with 2 updates (#299) (f18c2ba)
- deps: Bump the misc group across 1 directory with 36 updates (#313) (1574582)
0.19.0 (2026-09-01)
⚠ BREAKING CHANGES
Features
- add an nftables firewall backend (#251) (7541115)
- drop the update- prefix from the host modification flags (#250) (39d0d47)
- rework to add ufw support (#249) (8576563)
0.18.0 (2026-09-01)
⚠ BREAKING CHANGES
- add package verification (#248)
Features
Build
- deps: Bump github.com/docker/docker-credential-helpers from 0.9.8 to 0.9.9 (#241) (0c0d8e8)
- deps: Bump golangci-lint from v2.13.1 to 2.13.2 in the core group across 1 directory (#239) (e9bdab8)
- deps: Bump google.golang.org/api from 0.293.0 to 0.295.0 (#247) (a42d408)
- deps: Bump the core group across 1 directory with 2 updates (#242) (2d16203)
- deps: Bump the cosign group across 1 directory with 20 updates (#244) (b31baad)
- deps: Bump the k8s group across 1 directory with 7 updates (#243) (afb47df)
- deps: Bump the opentelemetry group across 1 directory with 11 updates (#245) (5d15c69)
0.17.0 (2026-08-31)
⚠ BREAKING CHANGES
- make package signature verification actually run on publish and local sources (#236)
Bug Fixes
- add logging into debug on config (#238) (28a9aba)
- make package signature verification actually run on publish and local sources (#236) (bc34895)
- remove no-op –skip-sbom flag, unblock vault-encrypt env fallbacks (#235) (8a5c5bc)
CI/CD
0.16.0 (2026-08-30)
⚠ BREAKING CHANGES
- generate rke2 and k3s example packages per CNI flavor (#228)
Features
- generate rke2 and k3s example packages per CNI flavor (#228) (d9e5c0d)
- generate typed engine config structs per distro and version (#226) (c2649b3)
- pull pinned k3s and rke2 source into thirdparty-src (#225) (b8ee106)
- validate engine config keys against the generated schemas (#227) (b959ef5)
Documentation
- add usage examples to every cargoship subcommand (#230) (d4d7528)
- expand the mage target reference (#224) (84c60f7)
CI/CD
0.15.1 (2026-08-29)
Features
0.15.0 (2026-08-28)
⚠ BREAKING CHANGES
Features
Bug Fixes
0.14.0 (2026-08-27)
⚠ BREAKING CHANGES
- add logging to file (#213)
- add logic for keeping track of files uploaded (#212)
- add registry overrides (#211)
Features
- add ability to sign packages (#208) (3646edf)
- add logging to file (#213) (b9ee105)
- add logic for keeping track of files uploaded (#212) (de757a9)
- add registry overrides (#211) (cb65777)
- add the ability to create reproducible packages (#209) (2737a87)
0.13.0 (2026-08-24)
⚠ BREAKING CHANGES
- add caching of artifacts (#204)
Features
- add caching of artifacts (#204) (854111d)
- add shell completion for arch (#207) (e81b2fb)
- fix permissions when pulling package (#206) (8990299)
CI/CD
Build
- deps: Bump google.golang.org/grpc from 1.83.0 to 1.83.1 (#195) (8878453)
- deps: Bump the core group across 1 directory with 2 updates (#198) (1b6324e)
- deps: Bump the cosign group across 1 directory with 16 updates (#199) (3bbb43d)
- deps: Bump the k8s group across 1 directory with 8 updates (#194) (9b49268)
- deps: Bump the misc group across 1 directory with 28 updates (#201) (dfb9e29)
0.12.0 (2026-08-23)
⚠ BREAKING CHANGES
- rework viper config (#191)
Features
CI/CD
0.11.0 (2026-08-21)
⚠ BREAKING CHANGES
- update zarf version (#187)
Features
Build
- deps: Bump the cosign group across 1 directory with 20 updates (#183) (eb4a6ff)
- deps: Bump the golang group across 1 directory with 10 updates (#182) (a9deafc)
- deps: Bump the misc group across 1 directory with 17 updates (#186) (b20ea24)
- deps: Bump the opentelemetry group across 1 directory with 5 updates (#184) (61ce493)
0.10.2 (2026-08-14)
Features
Documentation
0.10.1 (2026-08-12)
Documentation
0.10.0 (2026-08-11)
⚠ BREAKING CHANGES
Features
- add profiles for the config (#176) (cda1112)
- add the built dagger artifacts (#158) (9f61630)
- schema generator includes golang comments (#165) (68364ff)
- standardize the location distro interaction (#159) (3ba592e)
Documentation
CI/CD
- update dependabot to update indirect goland modules (#138) (5cd3b69)
- update the dependabot (#174) (8061c61)
- update the dependabot config (#172) (aa6652a)
- update the dependabot logic (#150) (d7b2fef)
Build
- deps: Bump github.com/alibabacloud-go/debug from 1.0.0 to 1.0.1 (#149) (6891f31)
- deps: Bump github.com/aws/aws-sdk-go-v2/internal/configsources from 1.4.29 to 1.4.34 (#144) (45d3e40)
- deps: Bump github.com/Azure/go-autorest/autorest/date from 0.3.0 to 0.3.1 (#143) (e3e647f)
- deps: Bump github.com/docker/cli in the docker group across 1 directory (#167) (8ef1011)
- deps: Bump github.com/felixge/httpsnoop from 1.0.4 to 1.1.0 (#154) (be7dd60)
- deps: Bump github.com/gabriel-vasile/mimetype from 1.4.13 to 1.4.15 (#142) (e620467)
- deps: Bump github.com/go-git/go-git/v5 from 5.19.1 to 5.19.2 (#163) (8300cdd)
- deps: Bump github.com/go-openapi/swag/pools from 0.27.3 to 0.28.0 (#145) (f712628)
- deps: Bump github.com/lestrrat-go/dsig from 1.2.1 to 1.3.0 (#141) (b280e61)
- deps: Bump github.com/lestrrat-go/jwx/v3 from 3.1.1 to 3.2.0 (#157) (b077c77)
- deps: Bump github.com/moby/sys/sequential from 0.6.0 to 0.7.0 (#156) (081a016)
- deps: Bump go.opentelemetry.io/…/google.golang.org/grpc/otelgrpc from 0.68.0 to 0.69.0 (#155) (457dfa7)
- deps: Bump go.opentelemetry.io/otel/metric from 1.44.0 to 1.45.0 (#162) (91435aa)
- deps: Bump go.opentelemetry.io/otel/metric from 1.44.0 to 1.45.0 in /.dagger (#164) (268f75a)
- deps: Bump the aws group across 1 directory with 14 updates (#151) (4af8fdf)
- deps: Bump the aws group across 1 directory with 15 updates (#168) (56ea0cf)
- deps: Bump the azure group across 1 directory with 6 updates (#152) (26a3688)
- deps: Bump the docker group across 1 directory with 2 updates (#140) (9fce9c0)
- deps: Bump the misc group across 1 directory with 28 updates (#175) (031dc81)
- deps: Bump the openapi group across 1 directory with 11 updates (#153) (8273afb)
- deps: Bump the sigstore group across 1 directory with 6 updates (#161) (26f65d3)
0.9.0 (2026-08-05)
⚠ BREAKING CHANGES
- add framework for publishing and pulling packages (#134)
Features
0.8.0 (2026-08-04)
⚠ BREAKING CHANGES
- add sha256sum command (#136)
Features
Documentation
CI/CD
0.7.0 (2026-07-28)
⚠ BREAKING CHANGES
- vendor the golang dep (#128)
CI/CD
- add envrc config (#123) (8e3154d)
- address dependabot issues (#129) (07c3a04)
- vendor the golang dep (#128) (832f376)
Build
- deps: Bump actions/checkout from 7.0.0 to 7.0.1 (#122) (0d77578)
- deps: Bump actions/setup-go from 6.5.0 to 7.0.0 (#120) (4276e9d)
- deps: Bump actions/setup-node from 6.4.0 to 7.0.0 (#119) (bde75d1)
- deps: Bump docker/build-push-action from 7.2.0 to 7.3.0 (#109) (d201529)
- deps: Bump docker/login-action from 4.2.0 to 4.3.0 (#112) (902d370)
- deps: Bump docker/login-action from 4.3.0 to 4.4.0 (#114) (a6ae875)
- deps: Bump docker/login-action from 4.4.0 to 4.5.1 (#126) (3d087d1)
- deps: Bump docker/login-action from 4.5.1 to 4.5.2 (#130) (e93e7c1)
- deps: Bump docker/setup-buildx-action from 4.1.0 to 4.2.0 (#113) (92a726b)
- deps: Bump github.com/containerd/containerd/v2 from 2.3.2 to 2.3.3 (#118) (8992739)
- deps: Bump golang.org/x/sync from 0.21.0 to 0.22.0 (#115) (1ec8814)
- deps: Bump https://github.com/google/keep-sorted from v0.9.0 to 0.9.1 (#110) (773b5f3)
- deps: Bump k8s.io/client-go from 0.36.2 to 0.36.3 in the k8s group across 1 directory (#125) (7126818)
- deps: Bump oras.land/oras-go/v2 from 2.6.1 to 2.6.2 (#116) (7140d42)
0.6.0 (2026-06-30)
⚠ BREAKING CHANGES
Features
Build
- deps: Bump golangci/golangci-lint-action from 9.2.1 to 9.3.0 (#105) (a314b7c)
- deps: Bump goreleaser/goreleaser-action from 7.2.2 to 7.2.3 (#106) (0b32174)
0.5.0 (2026-06-25)
⚠ BREAKING CHANGES
- add prepare command (#99)
Features
- add prepare command (#99) (9f191aa)
- logging: add better logic for distro output (#97) (8e96e83)
- update the rig version to use upstream package (#92) (6e34be6)
CI/CD
Build
- deps: Bump actions/checkout from 6.0.2 to 6.0.3 (#85) (e059838)
- deps: Bump actions/checkout from 6.0.3 to 7.0.0 (#94) (2c55700)
- deps: Bump actions/setup-go from 6.4.0 to 6.5.0 (#98) (6686e60)
- deps: Bump github.com/k0sproject/rig from 0.21.10 to 0.21.11 (#100) (8152958)
- deps: Bump github.com/txn2/txeh from 1.8.0 to 1.8.1 (#86) (cc8160e)
- deps: Bump github.com/zarf-dev/zarf from 0.76.0 to 0.77.0 (#83) (c9edcb8)
- deps: Bump https://github.com/google/keep-sorted from v0.8.0 to 0.9.0 (#87) (682a63b)
- deps: Bump k8s.io/client-go from 0.36.1 to 0.36.2 in the k8s group across 1 directory (#91) (1138748)
- deps: Bump oras.land/oras-go/v2 from 2.6.0 to 2.6.1 (#89) (b4a51f8)
0.4.0 (2026-05-23)
⚠ BREAKING CHANGES
- add dagger for ci building (#59)
Features
- add dagger for ci building (#59) (2545844)
- add mage for make like tool (#63) (e35aed7)
- change sysctl to create a file (#62) (aba7d57)
- cmd: expand version debug info (#68) (29fd345)
- distro: update the upgrade logic for rancher (#66) (7c4da85)
- update version to include deps (#60) (9304ae9)
Miscellaneous
CI/CD
- allow building on host system (20cac43)
- remove the mage sub-modules (#74) (9c073a4)
- update dagger deps (#64) (96ad0c8)
Build
- deps: Bump actions/create-github-app-token from 3.1.1 to 3.2.0 (#61) (ac2c0b7)
- deps: Bump docker/build-push-action from 7.1.0 to 7.2.0 (#79) (a0b4da1)
- deps: Bump docker/login-action from 4.1.0 to 4.2.0 (#80) (5d69ede)
- deps: Bump docker/setup-buildx-action from 4.0.0 to 4.1.0 (#82) (f147cf5)
- deps: Bump github.com/containerd/containerd/v2 from 2.2.3 to 2.3.0 (#73) (9639439)
- deps: Bump github.com/containerd/containerd/v2 from 2.3.0 to 2.3.1 (#78) (e6f1bd1)
- deps: Bump github.com/invopop/jsonschema from 0.13.0 to 0.14.0 (#77) (d899d71)
- deps: Bump github.com/magefile/mage from 1.15.0 to 1.17.2 (#75) (9ac36d1)
- deps: Bump github.com/stoewer/go-strcase from 1.3.0 to 1.3.1 (#76) (df7314c)
- deps: Bump github.com/zarf-dev/zarf from 0.75.1 to 0.76.0 (#67) (589b14f)
- deps: Bump golangci/golangci-lint-action from 9.2.0 to 9.2.1 (#81) (220e939)
- deps: Bump goreleaser/goreleaser-action from 7.2.1 to 7.2.2 (#71) (e27ed7b)
- deps: Bump k8s.io/client-go from 0.36.0 to 0.36.1 in the k8s group across 1 directory (#72) (920efb9)
- deps: Bump sigstore/cosign-installer from 4.1.1 to 4.1.2 (#57) (3aa3636)
0.3.0 (2026-05-07)
⚠ BREAKING CHANGES
- add proper code attribution (#55)
Features
- add command to get kubeconfig (#41) (ae53fdf)
- add proper code attribution (#55) (e971e89)
- address panic when no config is found (#51) (ff86d1a)
- update rig fork (#49) (71b9124)
CI/CD
Build
- deps: Bump github.com/zarf-dev/zarf from 0.75.0 to 0.75.1 (#46) (57327f8)
- deps: Bump https://github.com/golangci/golangci-lint from v2.11.4 to 2.12.1 (#45) (d791dd0)
- deps: Bump https://github.com/golangci/golangci-lint from v2.12.1 to 2.12.2 (#56) (8d00775)
0.2.5 (2026-05-01)
CI/CD
- release: expand release logic (10eb37f)
Build
0.2.4 (2026-04-29)
CI/CD
0.2.3 (2026-04-28)
Features
Build
0.2.2 (2026-04-28)
CI/CD
0.2.1 (2026-04-28)
CI/CD
0.2.0 (2026-04-27)
CI/CD
0.1.0 (2026-04-27)
Features
- add logic for allowing distro to manage stopping service (#11) (73010f5)
- add reset action (#19) (5b3bc07)
- core: basic logic for bootstrapping (#1) (03e1704)
- distro: add kill-all script for k3s (#12) (d0bc761)
- distro: add rke2 1.35.4 (#13) (fcf4616)
- init (7918478)
- run golangci lint (#16) (f1be9fa)
- schema: rework config schema (#23) (444214f)
CI/CD
- action: add check for go mod delta’s (#15) (9b1a40d)
- commit: add pr title check (#14) (3e071da)
- release: configure to bump minor (7bbed97)
- release: fix config settings (88a8fd1)
- release: remove versioning (30efbff)
Build
- deps: Bump actions/setup-node from 6.3.0 to 6.4.0 (#21) (846530b)
- deps: Bump github.com/invopop/jsonschema from 0.13.0 to 0.14.0 (#3) (51e8ace)
- deps: Bump github.com/stoewer/go-strcase from 1.3.0 to 1.3.1 (#4) (9ea3635)
- deps: Bump googleapis/release-please-action from 4.4.0 to 5.0.0 (#2) (0e30fa2)
- deps: Bump goreleaser/goreleaser-action from 7.1.0 to 7.2.1 (#22) (5d094f0)