apply phases

With --dry-run, cargoship connects to every host and runs the preflight checks for real, then reports the phases it would have run instead of running them. Each phase below is labelled with what a dry run does with it. A phase is only run when it declares that it is safe to, so a phase added later is reported until someone says otherwise.

The report is not a static list. Every phase still prepares itself and checks whether it has anything to do, and both only read, so work that is already done is filtered out against the live hosts. A phase that could not be assessed, because it reads state an earlier reported phase would have created, is reported as unassessed.

A dry run takes no cluster lock, so it does not block a real run, and it can report state that a concurrent run is already changing. It does not need --confirm.

1. Connect to hosts
   • Connects to a remote host via github.com/k0sproject/rig
   • Dry run: runs, reads only. Connects opens the SSH session to each host and does nothing else. A dry run needs it, because a preflight that never reached a host would report on a cluster it never looked at.
2. Detect host operating systems
   • Gathers information about the remote host, including: OS and OS version
   • Dry run: runs, reads only. Reads /etc/os-release and the kernel to pick a configurer for the host. Reporting what each host runs is half of what makes a dry run worth running.
3. Gather host facts
   • Gathers network related information about the remote host, including: Hostname, Private Address, Private Interface. Will also update the hosts based off the profile if configured in the config file.
   • Dry run: runs, reads only. Gather facts about each host by asking for its hostname, private interface and private address. All three are reads, and the rest of the run decides what it would do from them.
4. Validate hosts
   • Verifying that each node in the cluster has a unique name and private address, that its CPU architecture is one the package carries, and that its firewall rules are usable,
   • Dry run: runs, reads only. Validate the hosts is the preflight itself: sudo, unique hostnames and addresses, host architecture, firewall rules and clock skew. A dry run that skipped it would check nothing.
5. Gathering facts about the distro installed
   • Gathers information relating to the specific distro being installed, including: if the distro is installed, and what version it is running
   • Dry run: runs, reads only. Gather distro related facts reads the engine version already on each host. That is what tells an upgrade apart from an install, and what catches a downgrade before a real run starts one.
6. Checking for nodes no longer in the config
   • Compares the nodes joined to the cluster against the hosts in the config and stops the apply when the cluster holds a node the config does not, since nothing later in an apply removes a node
   • Dry run: runs, reads only. Checking for removed nodes lists the nodes joined to the cluster and compares them to the config. It writes nothing, and a dry run is exactly when an operator wants to be told that a host they deleted from the config is still running.
7. Acquire exclusive host lock
   • Runs a background task that touches a file every 30 seconds on each remote node. This prevents other cargoships from making any changes until the lock file has not been touched for over a minute
   • Dry run: reported, not run
8. Prepare hosts
   • Updates the remote nodes; environment variables and sysctl
   • Dry run: reported, not run
9. Prepare hosts - Enterprise Linux support
   • Installs container-selinux on systems that have SELinux enabled on them
   • Dry run: reported, not run
10. Prepare hosts - Enterprise Linux support - Fapolicyd
    • Writes the distro-supplied FAPolicy rules to /etc/fapolicyd/rules.d/31-cargoship.rules and executable rules to /etc/fapolicyd/rules.d/32-cargoship-executable.rules
    • Dry run: reported, not run
11. Prepare hosts - Enterprise Linux support - SELinux policy
    • Installs the policy modules, booleans and file contexts from spec.config.os.selinux on hosts running SELinux in enforcing mode, using semodule, setsebool and semanage fcontext
    • Dry run: reported, not run
12. Updating hosts file for clusters nodes
    • If enabled, then this will modify the /etc/hosts file on the remote nodes with the fully qualified domain name for each node in the cluster
    • Dry run: reported, not run
13. Updating hosts firewall
    • If enabled, this configures the firewall on each node that runs one, firewalld, ufw, or nftables, preferring the front end the node’s OS ships. It trusts every other node in the cluster along with the engine’s pod and service CIDRs, opens the ports in the .host.ports section, and applies the rules in the .host.firewall.rules section
    • Dry run: reported, not run
14. Upload files to hosts
    • Uploads the distro agnostic files to each remote node
    • Dry run: reported, not run
15. Upload files to hosts – RPM
    • If the remote node is an Enterprise Linux and the Distro package includes any files for those systems
    • Dry run: reported, not run
16. Upload files to hosts – APT
    • If the remote node is a Debian based Operating System and the Distro package includes any files for those systems
    • Dry run: reported, not run
17. Upload files to hosts – Binaries
    • Catch-all phase for when the combination of Operating System and Distro has no other install method
    • Dry run: reported, not run
18. Import images
    • For a distro whose engine does not import uploaded image tarballs on its own, imports them into the engine’s image store
    • Dry run: reported, not run
19. Configure engine
    • Runs distro specific operations
    • Dry run: reported, not run
20. Initialize Controller
    • If the remote node does not have a running controller service, and is a controller, install the engine and start each service sequentially
    • Dry run: reported, not run
21. Apply manifests
    • For a distro whose package declares raw manifests – a CNI, typically – kubectl applies each one from the leader
    • Dry run: reported, not run
22. Initialize Worker
    • If the remote node does not have a running worker service, and is not a controller, install the engine and start each service by the set concurrency limit
    • Dry run: reported, not run
23. Upgrade Controller
    • If the remote node is a controller and is running an older version of the engine, drain the node, stop the service, upgrade the engine, start the service, and uncordon the node sequentially
    • Dry run: reported, not run
24. Upgrade Worker
    • If the remote node is a worker and is running an older version of the engine, drain the node, stop the service, upgrade the engine, start the service, and uncordon the node by the set concurrency limit
    • Dry run: reported, not run
25. Install staged files
    • Moves staged engine files into place on any host the initialize and upgrade phases did not install
    • Dry run: reported, not run
26. Sync Registry Config Controller
    • If the remote node is a controller and its engine config (registries/audit/pss) has drifted from the desired state, drain the node, stop the service, write the new config, start the service, and uncordon the node sequentially. Chart values are written in place instead, since the engine reconciles them without a restart
    • Dry run: reported, not run
27. Sync Registry Config Worker
    • If the remote node is a worker and its engine config (registries/audit/pss) has drifted from the desired state, drain the node, stop the service, write the new config, start the service, and uncordon the node by the set concurrency limit
    • Dry run: reported, not run
28. Updating kubeconfig file with the current cluster
    • If enabled, this will update the local kubeconfig with the admin creds for the current distro
    • Dry run: reported, not run
29. Labeling nodes with their profile group
    • If enabled, this checks each node’s node-role.kubernetes.io/<profile> label and adds it, set to “true”, when missing or set to anything else
    • Dry run: reported, not run
30. Release exclusive host lock
    • Deletes the lock file from each node, allowing other cargoships to run
    • Dry run: reported, not run
31. Disconnect from hosts
    • Deletes any lingering temp files and disconnects from the remote node
    • Dry run: runs its own dry-run path