Edit

Build Steps

Purpose: For platform engineers, lists the steps the build orchestrator runs in order, with inputs, outputs, and the most common failure modes for each. Derived from src/opencenter_build/orchestrator.py:BuildOrchestrator.steps and the step handlers registered in src/opencenter_build/cli.py:build().

The pipeline has eight steps. Each step writes a checkpoint to build/state.json so the orchestrator can resume after a failure.

| # | Step | Typical duration | |---|---|---| | 1 | scan_repositories | 1–2 min (clone) + 30 s (scan) | | 2 | collect_helm_charts | 1–3 min | | 3 | generate_kubespray_lists | 30–60 s | | 4 | mirror_terraform_providers | 2–5 min (OpenStack template only; otherwise skipped) | | 5 | organize_assets | 5–10 min | | 6 | generate_zarf_yaml | < 30 s | | 7 | create_zarf_package | 5–15 min (requires Zarf CLI) | | 8 | generate_manifest | < 30 s |

The slow phase before this pipeline — downloading container images, OS packages, and Python wheels — happens inside kubespray-offline and is invoked from the organize_assets step.

1. scan_repositories

Clones the repositories listed in config/versions.env into build/<repo>/ and scans them for image references.

Inputs

  • config/versions.env (repo list parsed by extract_repos_from_versions).

Outputs

  • Cloned repositories under build/.

  • config/all-images.txt — deduplicated list of every image found.

Common failures

  • Failed to clone <repo> — network or auth issue. Confirm the repo URL and any required SSH key. Use git clone <url> from the same shell to isolate.

  • No repositories available for scanning — versions.env had no *_REPO variables. The build continues with an empty image list.

2. collect_helm_charts

Walks every cloned repo for FluxCD HelmRelease and HelmRepository resources, resolves the chart for each release, and pulls images out of the chart values.

Inputs

  • build/<repo>/ directories (from step 1).

Outputs

  • config/helm-repos.txt — one name=url line per Helm repository discovered.

  • config/helm-charts.txt — one line per Helm release with its chart, source repo, and version.

  • config/all-images.txt — appended with images extracted from rendered chart values.

  • build/helm-cache/ — pulled charts, kept for the organize_assets step.

Common failures

  • helm template failed — a chart could not be rendered. The error mentions the chart path. Pin the chart version, or skip the release by removing it from the source repo.

  • no Helm releases found — not a failure. The step exits with status=completed, releases=0.

3. generate_kubespray_lists

Runs Kubespray’s own generate_list.sh to derive the canonical lists of images and files Kubespray expects.

Inputs

  • build/kubespray/ (from step 1).

Outputs

  • config/kubespray-files.txt.

  • config/kubespray-images.txt.

Common failures

  • kubespray not found at build/kubespray — step 1 did not clone Kubespray. Re-run opencenter-airgap build (resume mode) after fixing the underlying clone error.

4. mirror_terraform_providers

Builds a filesystem mirror of the Terraform providers in terraform_providers[]. Only runs when versions.env was generated from the openstack template; otherwise the step is a no-op.

Inputs

  • terraform_providers[] in config/components.yaml.

Outputs

  • assets/terraform-mirror/ — directory layout matching Terraform’s filesystem_mirror.

  • A snippet of .terraformrc config that will be installed by zarf.yaml at deploy time.

Common failures

5. organize_assets

Calls into scripts/build/collect_kubespray_offline.sh and scripts/build/organize_assets_for_zarf.sh to download every binary, container image, OS package, and Python wheel, then arranges them under assets/ in the layout zarf.yaml expects.

Inputs

  • config/kubespray-images.txt, config/kubespray-files.txt, config/all-images.txt.

  • config/helm-charts.txt, config/helm-repos.txt.

  • config/components.yaml (for tools and binaries).

Outputs

  • assets/k8s-binaries/ — kubelet, kubeadm, kubectl, runc, containerd, CNI plugins.

  • assets/repos/ — apt and pip mirrors.

  • assets/kubespray/ — Kubespray playbooks ready to run on the bastion.

  • assets/playbook/ — the offline-repo.yml Ansible playbook.

  • assets/python-wheels/, assets/target-scripts/, assets/images/.

Common failures

  • This is by far the longest step and the one most affected by network flakiness. Restart with opencenter-airgap build to resume — the orchestrator picks up from the asset that failed.

  • disk space exhausted — peak usage during this step is 60–70 GB. Free space or move build/ to a larger volume (opencenter-airgap clean then re-run).

6. generate_zarf_yaml

Renders zarf.yaml.template into zarf.yaml, substituting variables from versions.env (architecture, ports, install path, registry image version).

Inputs

  • zarf.yaml.template.

  • config/versions.env.

Outputs

  • zarf.yaml.

Common failures

  • Template variable missing — usually versions.env is missing a required key. Run opencenter-airgap validate to see the exact key.

7. create_zarf_package

Shells out to zarf package create to compress every component into a single .tar.zst.

Inputs

  • zarf.yaml, assets/, the Zarf CLI on $PATH.

Outputs

  • dist/zarf-package-opencenter-airgap-amd64-<version>.tar.zst.

  • dist/zarf-package-opencenter-airgap-amd64-<version>.tar.zst.sha256 (Zarf writes this).

  • dist/zarf-package-opencenter-airgap-amd64-<version>-sbom.json (Zarf writes this).

Common failures

  • zarf: not found — the Zarf CLI is not installed. Install it (https://zarf.dev/install/) and re-run.

  • Compression fills the disk. The Zarf staging directory needs ~30 GB free in addition to assets/.

8. generate_manifest

Writes a JSON artifact manifest of every file produced by the build, with checksums and file sizes.

Inputs

  • dist/.

  • config/components.yaml.

Outputs

  • dist/artifact-manifest.json.

Common failures

  • None typical. If create_zarf_package was skipped (no Zarf CLI), this step still runs and lists everything that was produced — the package path will be absent.

Build state

The orchestrator writes one StepCheckpoint per step into build/state.json. Each checkpoint records:

  • name, status (PENDING / RUNNING / COMPLETED / FAILED).

  • start_time, end_time (ISO 8601 strings).

  • artifacts — a small dict of named output paths, used by later steps.

  • error — the truncated exception message on failure.

The state file also stores the SHA-256 hash of versions.env plus components.yaml. A mismatch refuses resume. See ../operations/resume-failed-build.md[Resume a Failed Build].

Logs

Each build writes to build/build-<build-id>.log in addition to stdout. The log captures every step transition and shells out to subprocesses. Pass --verbose to opencenter-airgap build to print full Python tracebacks on failure.

  • ../operations/resume-failed-build.md[Resume a Failed Build].

  • cli-commands.md#build[CLI Commands → build].

  • component-manifest-schema.md[Component Manifest Schema] — the input to most steps.

  • ../concepts/architecture-overview.md[Architecture Overview] — why these eight steps and not more.