Why the Ansible module is the cargoship binary, run on the management node

Cargoship’s cluster lifecycle commands – apply, prepare, reset, engine-config-sync, kube-config – should be drivable from an Ansible playbook, not only from cargoship’s own CLI. Three architectures were considered: the compiled Go binary as the module file itself, run on one node; the same binary copied to every managed node with Ansible sequencing the phases; and a conventional Python AnsibleModule wrapping cargoship as a subprocess. The first was chosen. It preserves how cargoship is operated and shipped today, and the node it runs on is the management node that air-gapped environments already stage onto.

Cargoship runs on one node, and that node is the management node

prepare, apply, and reset are already multi-host SSH orchestrators. They open their own connections to every host in the inventory and run the phase pipeline – connect, detect, validate, prepare, upload, bootstrap, kubeconfig, cleanup – across the fleet from a single process. Nothing puts a cargoship binary on a managed host: what lands there is engine artifacts and configuration. The engine binaries and RPMs (59_bin_install.go, 57_rpm_install.go), image tarballs (50_uploadfiles.go), the upload manifest under /var/lib/cargoship (03_manifest.go), /etc/sysctl.d/99-cargoship.conf, the fapolicyd rules file, and a lock file. All of those are files cargoship writes over SSH, not programs it installs.

The lock phase is the one piece that reads like an on-host agent and is not. Its explanation says it “runs a background task that will touch a file every 30 seconds on each remote node”, but startTicker is a goroutine inside the cargoship process calling Configurer.Touch over the connection it already holds (91_lock.go). The host holds a file; the process holding the lock is on the management node.

This matches how air-gapped environments already work, and how Zarf works: artifacts are staged onto a management VM that has been through the airlock, and distribution to the environment happens from there. Cargoship needs the distro package, the images, and the engine artifacts local to wherever it runs, so that node is the management node whether or not Ansible is involved – which is also how cargoship is operated today, by running the binary there directly. The module executes on exactly one node: connection: local when the Ansible controller is the management node, delegate_to when they are separate.

The inventory comes from Ansible; the connections do not

The module is handed an Ansible inventory and derives the cluster from it, rather than being pointed at a hand-written ZarfCluster document. A team that runs Ansible already has one description of its fleet, and asking for a second one in cargoship’s vocabulary means two files that drift. Group membership is where that fleet already records what a host is for, so it is where the role comes from: by default the group named controller supplies the control-plane nodes and the group named worker supplies the rest, with a role_groups mapping for inventories that use other names. The defaults are cargoship’s own role constants rather than a guess at an Ansible convention, so a site that says k8s_masters says so once, in the mapping.

Nothing in Go parses an inventory file. Resolving an Ansible inventory means group membership, group_vars, host_vars, dynamic inventory plugins, and the precedence rules over all of them; reproducing those in Go would reproduce them wrongly, and the failure would be a silent difference between what the playbook believes and what cargoship installs. Ansible resolves the inventory and hands over the result already merged, as groups and hostvars. internal/ansibleinv does only the typed translation: roles from group membership, connection details from ansible_host, ansible_user, ansible_port, and ansible_ssh_private_key_file, and everything cargoship needs beyond those from variables under a cargoship_ prefix.

Three rules in that translation are worth recording because they are decisions rather than mechanism.

• A host in none of the mapped groups is left out, and a host in two is an error. A play’s inventory legitimately carries hosts that have nothing to do with the cluster, so absence is not a mistake. Two mapped roles for one host is a mistake with no defensible resolution, and picking one silently would install a cluster whose shape nobody asked for. Neither case has a precedence rule, which is consistent with how validate and the keygen refusals behave elsewhere.
• A cargoship_ variable cargoship does not read is an error. Ansible has no notion of a variable belonging to anyone, so a misspelled cargoship_profil would otherwise sit in hostvars looking set and do nothing. An unrecognised ansible_ variable is ignored: there are hundreds and they are not ours.
• Host order is load-bearing. 60_configure_engine.go makes the first controller in the document the leader, so the generated document emits controllers first, each role’s hosts in inventory order. This is the kind of fact that looks cosmetic in a diff and is not.

The generated document is checked against the inventory schema before anything is done with it, so a translation bug surfaces as a schema error naming the field rather than as a phase failure ten minutes into an apply. It is also written to disk and inspectable: an operator debugging a wrong translation needs to see what cargoship was actually given, and cargoship inventory from-ansible reproduces the same translation outside a playbook run so the result can be diffed and fed to cargoship validate.

What does not change is who connects. Ansible supplies the inventory and nothing else – no ansible_connection to a managed node, no per-host task, no fact gathering on the fleet. Cargoship opens every SSH connection itself, from the management node, exactly as it does when run by hand. That is the whole point of the arrangement described above, and it is the constraint every part of this design is built around.

Why the binary, and not a Python wrapper around it

A conventional AnsibleModule in Python, subprocessing cargoship, is the idiomatic Ansible answer, and it was seriously considered. It was rejected on three grounds.

• It reverses a principle this project has already recorded. choice-vault-library.md rejected go-ansible in part because “Cargoship needs neither Ansible nor Python installed on the machine running apply or vault encrypt”, explicitly “consistent with Cargoship shipping as a single static binary”. A Python module reintroduces exactly that dependency on exactly that machine. Whatever the merits elsewhere, reversing a stated principle needs a stronger reason than the one the wrapper offers.
• The interpreter spread is a liability in the environments this targets, not a convenience. That a management VM has an interpreter is easy – dnf is written in Python, RHEL 8 and newer ship platform-python, Ubuntu Server ships python3. But a Python module has to work across all of them at once, from platform-python 3.6 on RHEL 8 to 3.12 and later on RHEL 10 and current Ubuntu, in estates that are mixed by nature because they are air-gapped and upgrade slowly. A static Go binary has no interpreter compatibility surface at all. Interpreter availability is what makes the wrapper possible; it is not what makes it better.
• A second language costs more than it returns here. The collection has to be packaged, linted, and covered by CI, or it silently holds less rigour than the Go side – which is worse than not having it. What the wrapper actually buys, once examined, is ansible-doc and argument errors that name the offending parameter.

That last point needs one correction to a common argument in its favour. no_log is often cited as something only the Python path provides, for the vault password and age key parameters that apply accepts. It is not: task-level no_log: true in the playbook suppresses a task’s arguments and output regardless of module type. argument_spec’s per-parameter no_log is finer-grained, which is an ergonomics difference rather than a safety one.

A thin action plugin, which is narrower than the wrapper this rejected

That rejection was of a Python module that subprocesses cargoship, and it stands. A Python action plugin is added anyway, and the distinction is the whole of the reasoning.

The collection needs groups and hostvars to project into the module’s inventory argument, and those live in task_vars, which only controller-side code can read. Ansible offers no way to get them without controller-side code – an inventory plugin describes hosts to Ansible rather than the reverse, and a templated argument in the playbook pushes the projection onto every operator instead of doing it once.

An action plugin is materially narrower than the wrapper rejected above. It executes inside the ansible-playbook process, on the interpreter Ansible is already running on, so it adds no dependency Ansible has not already added to that machine. It is bounded by ansible-core’s own controller support matrix rather than the platform-python 3.6 through 3.12 spread that made the wrapper a liability, because a controller is one machine the operator chose, not an estate. And it changes nothing about running cargoship by hand: apply on a management node still needs no Python, which is the principle choice-vault-library.md actually recorded.

The cost that survives is real: a second language in the collection to lint and test. It is bounded by keeping the plugin thin. The plugin reads task_vars, projects groups and hostvars, and delegates. Every translation rule – role derivation, variable mapping, ordering, validation – is in Go, where it is unit-tested without Ansible. A plugin that grows a rule is a plugin that has drifted from this decision.

The revisit trigger is publication. If this collection is ever consumed outside the team, missing ansible-doc and an unhelpful failure on a mistyped parameter are the first two complaints it will receive. The work is not wasted if that happens: the wrapper would subprocess the same JSON entrypoint described below, so the Go side is the part that gets kept either way.

Why not the binary on every managed node

Copying the binary to each target and letting Ansible sequence one task per phase per host is usually pitched as buying native parallelism, retries, handlers, and per-host reporting. It was rejected, and the reason is stronger than the duplicate maintenance it would obviously create.

Cargoship’s phases are not independently runnable. Lock and Unlock are a single paired phase – NewApply builds one lockPhase and appends lockPhase.UnlockPhase() at the end – so the cluster lock is scoped to one Manager.Run. Connect and Disconnect bracket the pipeline, and the SSH sessions in between are reused by every phase that runs. Manager.Run’s deferred cleanup walks the phases that already ran and calls CleanUp on each when a later phase fails. Split those across separate Ansible tasks and the lock is never held across the work it protects, the cleanup-on-failure never fires, and every phase reconnects. Reassembling that in playbook form is not a port of the existing pipeline; it is a second implementation of it with different failure semantics.

The multi-arch problem is real too – Ansible has no arch-aware module selection, so this would need a custom action plugin to choose and copy the right binary per host – but it is the smaller objection. The pipeline is the objection.

One binary, not a second release artifact

The module needs a machine-readable way into pkg/action: arguments in as JSON, exactly one JSON object out, following the WANT_JSON convention. Two things make this cheap here that are usually expensive.

• Dry run already exists, and check mode maps onto it directly. pkg/phase/05_manager.go carries Manager.DryRun plus a three-way classifier, ClassifyDryRun, that decides per phase whether a dry run skips it, runs it as itself because it only reads (readOnly), or calls its own dry path (withDryRun). A phase that declares neither is skipped, so dry-run safety is a property a phase has to claim rather than one it is assumed to have. _ansible_check_mode becomes --dry-run, not a new subsystem. One gap had to be closed for this: engine-config-sync was the only lifecycle command that never registered the flag, though it embeds the field and passes it to the manager like the others. It registers it now, on the same terms as apply and reset – a dry run needs no --confirm, because requiring confirmation to ask what would happen is what pushes someone into running the real thing to find out.
• stdout is already clean. A module that prints anything but its one JSON object is a broken module, and in most Go CLIs separating the log stream from the result stream is the bulk of the work. Cargoship’s logging already goes to stderr: the zarf logger’s DestinationDefault is os.Stderr, and rig’s own chatter is bridged into that same logger by internal/riglogger rather than being written directly. Nothing on the install path writes to stdout.

That entrypoint lives in the existing binary rather than a new cmd/ target producing its own release artifact. Adding a second build to .goreleaser.yaml is a few lines, which is what makes it tempting, but the lines are not the cost. Every release artifact carries a cosign signature and an SBOM, and the nfpms builds produce GPG-signed rpm, deb, and apk packages. A second binary is a second thing to sign, publish, verify, and carry through an airlock, and it introduces a failure mode that is otherwise impossible: a module binary and a staged cargoship on the same node disagreeing about version. One binary means the thing answering Ansible is the thing doing the work, and a management node keeps both entry paths – through a playbook, or by running cargoship by hand as operators do today.

The single binary presents itself as several module files by dispatching on the argv[0] basename Ansible invokes it under: a basename of cargoship_ followed by the name of an action the binary answers as enters module mode, and anything else is the ordinary CLI. The name has to be a real action, not merely carry the prefix. The prefix alone is not rare enough to dispatch on, because the repository builds its own binary as cargoship_linux_amd64 and the whole end-to-end suite runs that file; dispatching on the prefix turned every one of those invocations into a JSON object nobody asked for. What that costs is a misspelled module file: cargoship_engine_confg_sync now runs the CLI and fails where it parses the arguments file as a command, rather than saying it is not a module. Ansible also copies a module to the machine that runs it and prefixes the copy AnsiballZ_, so that prefix comes off before the name is read, and the name the process actually sees is AnsiballZ_cargoship_apply. The module files are symlinks to the binary, created by the same nfpms packages that install it, so there is nothing extra to sign, publish, or carry through an airlock. An environment variable, CARGOSHIP_ANSIBLE_MODULE, provides the same selection for tests, which have no reason to create symlinks to check the contract.

A module does not reimplement the command it stands for. It renders its parameters as the argument vector an operator would have typed and runs that, so package loading, keyring resolution, timeout parsing, and phase construction stay on one path and the two entry points cannot drift apart. What that costs is one indirection: cmd imports internal/ansiblemod to reach the dispatch, so ansiblemod cannot import cmd back, and it spells the flag names out rather than referencing the constants. A test in cmd parses a fully populated argument vector against the real command, so a flag renamed in the CLI fails CI instead of failing a playbook against a live fleet.

The generated ZarfCluster is written before the command runs and is named in the result. A run that fails leaves it there, because a failure that reads like the wrong cluster is one an operator needs the document to diagnose and reproducing the translation afterwards has already lost the run that produced it. A run that succeeds removes it, unless the operator named the path themselves, in which case the file is theirs.

One correction to the reasoning above belongs here, because it is load-bearing for the module contract. “Nothing on the install path writes to stdout” is true of the install path and false of the init path: cmd/viper.go built a bootstrap logger with Destination: os.Stdout, and cmd/root.go reported a failed configuration load and two failed flag-completion registrations with fmt.Printf. All of those are plain bugs for any command that emits a document, Ansible or not, and all are fixed to stderr.

Relying on the fix alone would be relying on an audit staying true, so module mode also duplicates file descriptor 1 at entry, points os.Stdout at stderr, and writes its one JSON object to the duplicate. Anything added later that reaches for os.Stdout reaches stderr instead, which Ansible captures and shows on failure.

The guard covers everything after main begins, which is not everything. cmd builds the root command in a package-level variable, so the writes listed above happen during package initialisation, before main and therefore before any guard could be installed. Those are the audited ones, and they stay audited: the fix to stderr is what holds there, and a test drives the binary in module mode with a broken configuration file so that a regression fails CI rather than a playbook. The init path is also where the binary can exit before answering at all – a configuration file that cannot be parsed calls os.Exit(1) in init() – which Ansible sees as a module failure with the reason on stderr, the same failure the CLI gives for the same file.

One module per action, not one module with an action parameter

The actions do not share an option set. apply carries roughly a dozen flags – host file updates, firewall, fapolicyd, node labelling, unmanaged nodes, kubeconfig update and path, worker concurrency, values files, signature verification, age keys. kube-config carries three, and it does not even load a package: installKubeConfigOptions.run builds a phase.Manager directly instead of going through initManager, because there is no distro package to resolve values against. Collapsing that into one module with an action parameter means a single argument surface where most parameters are invalid for most values of action, and without argument_spec every one of those relationships is hand-checked. Separate modules keep each argument surface small enough to validate by hand.

The changed signal comes from the phases, and says how far it reaches

A module that reports changed: true on every successful run teaches operators to ignore the field, and a handler wired to it fires forever. Reporting it honestly means the phases saying what they did, which is what Manager.Run now collects.

The mechanism is the optional-interface idiom the manager already uses for readOnly and withDryRun: a phase may implement changedReporter, and Run asks it afterwards rather than before, so a phase answers from what it did. A run that failed halfway has still changed the fleet and is expected to say so. Manager.Result holds the per-phase record, and RunResult answers Changed, Outstanding, Complete, and Undeclared over it.

Three decisions are worth recording:

• Silence is not a claim of no change. A phase that does not implement the interface cannot make changed true, and is not counted as having changed nothing either. The overall signal is reported as partial and the silent phases are named in cargoship.changedUndeclared. Reporting changed: false while KubeConfig and LabelNodes have said nothing would be a claim cargoship cannot support; naming them lets an operator decide. The alternative considered was assuming no change from silence, which is how a correct-looking changed: false becomes a missed handler.
• Phases filtered out by ShouldRun are still asked. EngineConfigSyncHosts writes the files the engine re-reads without a restart while Prepare is deciding which hosts have drifted (except under a dry run, which reports the write instead – see the last paragraph below), and then reports no hosts to sync. A run whose only change landed in chart manifests would otherwise read as a run in which nothing happened. RunResult therefore records a third disposition, Skipped, beside Ran and Planned.
• Under check mode, changed reads Outstanding rather than Changed. A dry run changes nothing, so the useful question is whether a real run would, and a phase only reaches Planned after its own ShouldRun found work against the live hosts. That matches what Ansible’s check mode means by changed.

The result travels back on the context. The module mode runs cargoship by building an argument vector and calling the ordinary command through cmd.ExecuteArgs, which returns an error and nothing else. Threading a return value back through cobra, the action, and the manager would change five call sites to serve one caller, so phase.WithResultSink puts a sink on the context and Manager.Run publishes to it when one is there. A run that never reached its phases publishes nothing, which the module reports as unknown rather than as no change – the two are different, and conflating them is what makes a changed: false untrustworthy.

Writing the signal turned up one thing worth fixing on its own account: the in-place write in needsUpdate runs from Prepare, and Prepare runs for every phase including the ones a dry run is about to skip. A dry run was therefore writing files to hosts. It no longer does.

Five modules, and the parameters they deliberately do not offer

The binary answers as apply, prepare, engine_config_sync, reset, and kube_config. Those are the actions that converge a fleet, which is what a playbook is for. package create stays out for the reason at the end of this document.

Each module’s parameters follow its own command’s flags, and the surfaces genuinely differ – prepare reads no encrypted value and so takes no vault_password_file, reset installs nothing and so takes no package and no values, kube_config touches the management node rather than the fleet. A parameter offered to the wrong module is an error naming it, rather than a flag quietly dropped. What is shared is shared in code: one embedded common struct carries inventory, inventory_path, log_level, and log_format for every module, so one module cannot spell inventory_path differently from its neighbour.

Two things are deliberately left out.

Keyless signature verification. The modules expose public_key and verify and stop there, which the keyless removal has since made moot: a Cosign key pair is cargoship’s only signing method, so there is no keyless surface left for a module to decline to wrap. The reasoning is kept because it is the argument-surface-size reasoning from the previous section applied one level down, and because it is what should be weighed again if keyless verification returns through #848 – the flags were mutually exclusive in ways cobra enforced and a hand-written module would have had to restate.

A check-mode answer for kube_config. cargoship kube-config has no --dry-run. A module with nothing to answer with has two options: run anyway, or refuse. Running anyway would change the management node during a run that was asked only to describe itself, so the module reports skipped: true and runs nothing. That is what Ansible does with a module that has declared it does not support check mode; a binary module has nowhere to declare it, so it says so in its result instead.

The collection ships the links, and the role hardcodes what must not fail open

Ansible resolves a task name to a module file before any plugin of ours is consulted, so an action plugin alone reaches nothing. Each action therefore needs a file named cargoship_<action>, and each of those is a symlink to the binary. The RPM and deb packages create the links through nfpms contents: entries, installing the collection tree to /usr/share/ansible/collections, which is already on ansible-core’s default collections path; a collection installed from Galaxy or from a checkout has no packaging step to hang that on, so hack/ansible-link-modules.sh does it by hand. Both were chosen over generating a per-action shim file: a shim is a second thing to sign and a second thing that can disagree with the binary.

The action plugin forwards an allowlist of host variables rather than the whole of hostvars. A host’s resolved variables routinely hold credentials for things that are not this cluster, and a module’s parameters are written to a file on disk, so forwarding everything would write unrelated secrets to disk on every run. The allowlist is the four ansible_* connection variables plus every cargoship_* key – all of them, including misspelled ones, because rejecting an unknown parameter by name is ansibleinv’s job and it cannot do that without being shown the misspelling. TestActionPluginVariableAllowlist reads the Python and fails when it drifts from the Go constants, because the symptom of drift is an unset field rather than an error.

The plugin also merges a parameters dict into the arguments. The role has a caller-supplied dict of parameters and no way to know which of them the chosen module takes, and the obvious way to pass it – making the task’s whole argument dict one template – is warned about by Ansible on every run and is genuinely unsafe. Merging in the plugin keeps the task’s keys literal and its values templates, and costs one parameter that the module itself never sees.

The role sets run_once: true on every task. Cargoship converges the whole fleet in one run, so a play over the fleet’s own inventory would otherwise run a full convergence once per host. It defaults no_log: "{{ cargoship_no_log | bool }}" with cargoship_no_log: true by default: a template failure or misconfiguration avoids exposing decrypted credentials and host connection parameters in CI or production logs, while still allowing operators to set cargoship_no_log: false for local troubleshooting. The debug task showing cargoship’s own report carries no parameters.

The interface is documented in the plugins, and nothing validates against it

Each action plugin carries a DOCUMENTATION block naming every parameter the module takes, its type, and the flag it renders. generateModuleDocs in magefiles/pkg/gen/docs/ansible_module.go reads those blocks, generateRoleDocs reads each role’s meta/argument_specs.yml, and together they write docs/ansible/. The reference pages therefore cannot be edited into disagreement with the modules, and a parameter added to the Go struct without being documented shows up as a missing row rather than as nothing at all.

The blocks are inert. An action plugin’s DOCUMENTATION exists for ansible-doc, which is not run against this collection, so nothing in Ansible reads them and no rule moved into Python: there is still no argument_spec, and argument validation is still hand-written in Go. What keeps them honest is TestActionPluginDocsMatchModuleParams, which reads the Python and fails when a documented parameter has no Go field, when a Go field is undocumented, or when a documented cli_flag is not the flag the argument vector actually renders.

cli_flag: is ours rather than a standard Ansible documentation key. Stock validate-modules would flag it; it is not run either, and the parser that reads the key is the one in this repository.

This narrows the first bullet below rather than reversing it. The interface is now documented in a structured place and published, which is part of what publishing the collection was going to require of it anyway. Adding an argument_spec that Ansible itself validates against would be the reversal, and is still declined for the reasons in “Why the binary, and not a Python wrapper around it” above: what it buys is ansible-doc and better-named argument errors, and a plugin that grows a rule has drifted from this decision.

What is being accepted

• No ansible-doc and no ansible-test sanity. The module’s interface is documented where we choose to document it – the DOCUMENTATION blocks and the pages generated from them – and checked by a test of ours rather than by Ansible’s own tooling.
• Argument validation is hand-written, so a mistyped parameter fails less helpfully than argument_spec would manage.
• Secret redaction relies on task-level no_log in the playbook rather than per-parameter declarations, which is coarser and has to be documented so operators actually set it.
• Ansible sees one task per action, not per-host results. Per-host detail stays in cargoship’s log stream on stderr.

What this leaves open

Most phases still do not report whether they changed anything. Only the engine configuration sync phases declare changedReporter today, so an ordinary run reports its signal as partial. The two undeclared phases that can genuinely change something are KubeConfig, which writes the local kubeconfig, and LabelNodes, which sets node role labels; both are worth declaring next. The rest – Connect, DetectOS, GatherFacts, ValidateHosts, Lock, Disconnect – change nothing a playbook would want a handler for, and declaring them buys only a complete signal.

engine-config-sync came first, and the rest followed it. It is the most Ansible-shaped action cargoship has – converge configuration, report drift, restart services – and pkg/phase/70_engine_config_sync_common.go already computed the per-host idempotency signal in needsUpdate, driftedFiles, and driftReason. Proving the contract and the changed plumbing there first, before taking on apply’s argument surface, is what made the shared parameter and argument-vector machinery fall out rather than be designed up front.

Note that create is not in scope despite sometimes being listed alongside these commands. cargoship package create builds a package; it touches no hosts, runs no phases, and has no dry run. Whatever it needs from Ansible is a different question.