Automating Rebuildable Secure Pipelines with Zarf

Continuous Delivery (CD) isn't some slipshod method for delivering half-baked changes in a "go fast and break things" environment. In fact, the "GFABT" shops aren't using CD. Ask them. CD is about applying principles to deliver repeatable quality and traceability. It's about ensuring that we are always deployable and have quality gates in place to maintain this. It's the perfect fit for critical systems. When implementing CD, the first feature we should implement is the pipeline, which requires us to understand the software supply chain problem we are trying to solve. Engineering a CD pipeline is not about copying someone else's tooling checklist; it's about faithfully applying first principles regardless of context. Tools evolve, constraints shift, but the principles remain.

Recently, we were presented with an interesting delivery challenge.

"We have hundreds of repos in multiple languages. To meet compliance requirements, each release package must include everything needed to rebuild the artifact inside the secure network."

For most pipelines, we would never want to rebuild an artifact because the CD quality process is focused on validating artifacts, not code. This is why you should never build something on a test branch, deploy it to a test environment, validate it, promote it, rebuild, and deploy to production. If we did that, then what we deployed is not what we tested. Instead, we want an immutable artifact. There are many opportunities in that process to inject change that would invalidate one or more quality gates. So, we need to design our pipeline to minimize that risk.

The Current Flow

Their current pipelines already utilize containers and "sneakerware" delivery, where production artifacts are burned onto DVDs, which are then delivered to a secure facility, scanned for security, and loaded into an air-gapped network. This happens every week or two, though they could do it on demand if required, so we have something to build on.

Minimizing Variation risk

Since only delivering immutable artifacts isn't an option, we need to mitigate the risk of variation between builds. Our pipeline should already be tagging the release candidate container with the Git commit SHA. So we'll use that to capture and save all the dependencies, tools, and configurations used in the original artifact. We won't produce the same artifact with a rebuild, but we can have a high level of confidence that we'll have functional parity.

The first step, of course, is to collect the source for that commit tag. We can either do this as a clone of the commit tag or, if required, gather a complete Git bundle for keeping a remote version control system in sync. Along with that, we'll need to collect and package:

• All third-party and internal dependencies and their transitive dependencies
  • One issue we need to be aware of is that humans are non-deterministic in that, while they SHOULD follow good practices, they frequently don't. I've seen many teams that use mutable snapshot versions of dependencies to deliver production artifacts. There's no benefit to using mutable versions, so we shouldn't do it. However, some people do, so our process should allow for it.
• The version of the compiler/development kit
• The language's package manager version (Maven, NPM, Cargo, pip, etc.)

Make Rebuilds Fool-Proof

Because we aren't troglodytes, and actually care about making things clear for the people who touch our work later, we need to make it easy for them to rebuild the application on demand. This means that all of our build scripts need to be packaged to ensure it's easy to repeat the same process used to build the original artifact. To make this easy, we can use Zarf.

Zarf is an open-source tool built by Defense Unicorns to make air-gapped delivery very easy and to make it easy for non-engineers to deliver software updates to environments where developers may have no access, such as submarines or Sensitive Compartmented Information Facilities (SCIFs).

Here, we can use Zarf to create our final container with an application runtime container and a dedicated build container with all dependencies, tools, configuration, and scripts.

This gives us a Zarf package, which is an OCI container that we can add to the secure network's artifact registry. This gives the people with access to that network two options. They can run zarf deploy <runtime container> for the runtime container to deploy it to their operational environment. However, they can also run zarf deploy <build container> to rebuild the artifact with no other setup required.

Scaling to Everyone

Now that we have a pattern we can apply to every language family, the last thing we want is for every team to write these workflows. Instead, we need to create reusable templates for every language family. Then, teams simply need to add the workflow to their pipelines and return to business as usual.

By fusing disciplined CD principles, rather than dogmatic practice, with end-to-end version control and Zarf's air-gapped packaging, we create a fully automated, air-gapped software supply chain for secure software delivery. We maintain the principle of "always in a deployable state" and "deliverable on demand." We enable secure operations and provide full traceability between runtime artifacts and the code used to create them. And we can do all of this without any added toil.

The next time someone claims CD "won't work here," remind them that we can deliver to a SCIF or a submarine with CD. Perhaps their problem isn't as special as they think.