Skip to main content

Reproducible Builds

Purpose: For platform engineers, explains deterministic builds, pinned versions, and state management.

Concept Summary​

A reproducible build means running opencenter-airgap build twice with the same versions.env produces functionally identical Zarf artifacts. This matters in air-gapped environments because there is no way to fetch a missing dependency at deploy time — what you build is exactly what you get.

How Reproducibility Works​

Single source of truth: versions.env​

Every version in the build traces back to config/versions.env. This file pins:

  • Kubernetes and container runtime versions (KUBERNETES_VERSION, CONTAINERD_VERSION, RUNC_VERSION)
  • CNI plugin version (CALICO_VERSION)
  • Platform service versions (FLUXCD_VERSION, CERT_MANAGER_VERSION, etc.)
  • Tool versions (HELM_VERSION, K9S_VERSION, YQ_VERSION)
  • Terraform provider versions (OPENSTACK_PROVIDER_VERSION, etc.)
  • Git repository refs (KUBESPRAY_COMMIT_SHA, OPENCENTER_GITOPS_BASE_COMMIT_SHA)

When build runs, it reads versions.env and generates components.yaml with resolved URLs and version strings. No version is inferred or defaulted at build time.

Commit SHA pinning for Git repositories​

Branch names like main or master are mutable. For reproducibility, pin to a specific commit SHA:

# versions.env
KUBESPRAY_VERSION="master"
KUBESPRAY_COMMIT_SHA="a1b2c3d4e5f6"
OPENCENTER_GITOPS_BASE_VERSION="main"
OPENCENTER_GITOPS_BASE_COMMIT_SHA="f6e5d4c3b2a1"

The build clones at the specified SHA, not the branch tip. This ensures the same source code is used regardless of when the build runs.

Image tags vs. digests​

Container image tags can be overwritten (e.g., latest, or even semver tags on some registries). For maximum reproducibility, use digest-pinned references where possible:

# components.yaml — tag-based (default)
- image: registry.k8s.io/kube-apiserver:v1.34.3

# components.yaml — digest-pinned (stronger guarantee)
- image: registry.k8s.io/kube-apiserver@sha256:abc123...

The scan command records tags as discovered. You can manually replace tags with digests in components.yaml for critical images.

What Can Break Reproducibility​

FactorRiskMitigation
Mutable Git branch refsDifferent code at different timesPin COMMIT_SHA in versions.env
Mutable image tagsTag overwritten with different contentUse digest references for critical images
Upstream URL changesDownload URL returns 404 or different binaryCache build artifacts; verify checksums
OS package repo updatesDifferent package versions in apt mirrorPin package versions in Kubespray config
Build tool version driftDifferent CLI version produces different outputPin opencenter-airgap version in pyproject.toml

Build State and Checkpoints​

The CLI tracks build progress in a state file. Each phase records:

  • Start and end timestamps
  • Input checksums (versions.env hash, components.yaml hash)
  • Output artifact counts and sizes

This state enables --resume and also serves as an audit trail. If versions.env changes between runs, the state file detects the drift and forces a rebuild of affected phases.

Verifying Reproducibility​

To confirm two builds are equivalent:

# Build once
opencenter-airgap build
sha256sum dist/zarf-package-*.tar.zst > build1.sha256

# Clean and rebuild
opencenter-airgap clean
opencenter-airgap build
sha256sum dist/zarf-package-*.tar.zst > build2.sha256

# Compare
diff build1.sha256 build2.sha256

Byte-identical output depends on all upstream artifacts being unchanged. If an upstream registry serves a different layer for the same tag between builds, the checksums will differ — this is the case for digest pinning.

Trade-offs​

  • Digest pinning adds maintenance overhead: you must update digests when upgrading versions.
  • Commit SHA pinning requires looking up the SHA for each release, rather than using a branch name.
  • Caching build artifacts locally (for offline rebuilds) consumes additional disk space.

The trade-off is worth it in regulated environments where you need to prove that the artifact deployed in Zone C matches exactly what was built and audited in Zone A.

Further Reading​