Skip to content

Build and publish signed multi-arch loreserver images to GHCR - #180

Open
aleksanderllada wants to merge 6 commits into
mainfrom
aarruda/publish-loreserver-image
Open

Build and publish signed multi-arch loreserver images to GHCR#180
aleksanderllada wants to merge 6 commits into
mainfrom
aarruda/publish-loreserver-image

Conversation

@aleksanderllada

@aleksanderllada aleksanderllada commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Fills in publish-loreserver-image.yml, which landed earlier as a stub. Two variants publish, differing only in how arm64 is compiled:

Tag arm64 build
X.Y.Z, X.Y, latest baseline armv8-a — runs on any arm64 host
X.Y.Z-graviton, latest-graviton tuned for Graviton3+, as Lore is deployed

The portable build is the default so a community chart works everywhere, and the tuned build is opt-in for Graviton deployments. linux/amd64 is baseline in both, so it is built once and both manifest lists reference that single digest.

Each variant builds on its own native runner (QEMU is far too slow for a release Rust build) and is pushed by digest; a merge job stitches the digests into manifest lists and signs each with keyless cosign. No secrets are involved: GITHUB_TOKEN authenticates to GHCR and cosign signs against Fulcio via the job's OIDC token.

Triggers on v* tags, plus manual dispatch with an optional extra tag for proving the workflow from a branch.

Why two tags rather than one

OCI cannot express this. platform.variant covers v6/v7/v8, not microarchitecture, and nothing in the pull path consults CPU features — Graviton3, Ampere Altra and Apple Silicon are all just linux/arm64. A client cannot be steered to the right arm64 build automatically, so a separate tag is the only honest mechanism.

Architecture selection itself is unaffected: one tag still serves both amd64 and arm64, which is why -graviton is also multi-arch and a mixed-architecture cluster can pin a single tag.

Dockerfile changes worth reviewing

arm64 codegen. .cargo/config.toml pins aarch64-unknown-linux-gnu to Graviton3+ with -C target-cpu=neoverse-512tvb. The Dockerfile now assembles RUSTFLAGS itself from an ARM64_TARGET_CPU build arg, empty by default. It has to be RUSTFLAGS and not CARGO_TARGET_<triple>_RUSTFLAGS: the latter is only another source for the same config key and cargo joins config arrays, so the Graviton flag survives and a supposedly-baseline build still faults.

Debug info. [profile.release-lto] sets debug = 2, and a fat-LTO link holds the whole dependency graph's DWARF at once — rustc was SIGKILLed part way through linking on a 16 GB runner. Building with -C debuginfo=0 fixes it and costs nothing observable, because strip --strip-debug discarded that DWARF a line later anyway and tracing's file/line fields come from compile-time macros. Side effect: the arm64 image drops from 100 MB to 51 MB.

DOCKER.md documents both variants and the build arg.

Test evidence

A full dispatch published both variants, each signed and verified in-run. The amd64 digest is shared, as intended:

ci-test           linux/amd64  sha256:0d523ba6fe18a113…
                  linux/arm64  sha256:a06281c562b9d2f5…
ci-test-graviton  linux/amd64  sha256:0d523ba6fe18a113…   <- identical
                  linux/arm64  sha256:a45916979f3d0653…   <- differs

Both arm64 images were then run on an Apple M3 Max, which has no SVE:

default  arm64 -> loreserver 0.9.1-nightly     exit=0
graviton arm64 -> Illegal instruction          exit=132

That is the intended behaviour of both variants, and it is the only test that actually distinguishes them. The default image also boots and serves — 10 QUIC listeners on 0.0.0.0:41337.

A limitation to be aware of

The per-arch smoke step runs the freshly pushed image, which catches a broken entrypoint or a missing shared library. It does not catch codegen aimed at a CPU the eventual host lacks: GitHub's arm64 runner is Neoverse-N2 and reports SVE, so it executes the Graviton-tuned build without complaint. This was measured, not assumed, and it is why the Apple Silicon check above matters. A CI gate running the binary under qemu-aarch64 -cpu cortex-a72 would close it and is not yet implemented.

Notes

  • Follows the conventions in the surrounding workflows: SPDX header, top-level permissions: {}, per-job least privilege with justification comments, and every action pinned to a full commit SHA.
  • The cosign identity is anchored on this workflow's path, not just the repository, so a certificate minted by any other workflow here will not verify.
  • The ci-test and ci-test-graviton tags are leftovers from proving this and will be deleted.
  • This is opened for review and CI; the change reaches main through the usual mirror path rather than by merging here.

## Summary

Fills in the publish workflow: each architecture builds on its own native
runner and is pushed to GHCR by digest, a merge job stitches the digests
into one manifest list, and cosign signs that list keylessly through the
job's OIDC token. No secrets are involved.

Also drops the Graviton3+ target-cpu from the arm64 container build. The
repo tunes aarch64-unknown-linux-gnu for Epic's own deployment, which emits
instructions that fault on older arm64 parts and, because the build passes
no --target, reaches the proc macros cargo runs on the build host too. A
published image has to run anywhere armv8-a runs.

## Test Plan

1. actionlint passes on the workflow.
2. Dispatch from this branch and confirm the manifest list carries
   linux/amd64 and linux/arm64, and that cosign verify succeeds.
3. Pull the arm64 image on a non-Graviton arm64 host and confirm the
   server starts.

Signed-off-by: Aleksander Arruda <aleksander.arruda@xa.epicgames.com>
The first full run measured 145G on the runner root with 108G still free
after both release builds, so deleting unrelated preinstalled toolchains
buys nothing and only adds a step that can fail on its own.

Signed-off-by: Aleksander Arruda <aleksander.arruda@xa.epicgames.com>
## Summary

The previous attempt set CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_RUSTFLAGS
to drop the Graviton3+ target-cpu. That does not work: the variable is
another source for the same config key, and cargo joins config arrays
rather than replacing them, so -C target-cpu=neoverse-512tvb survived and
the published arm64 image died with SIGILL on any older arm64 part.

Use RUSTFLAGS instead, which is a mutually exclusive source that suppresses
[build] and [target.*] rustflags outright, and repeat the flags the build
actually needs.

Also stop overriding the release strip setting. The workspace asks for
line-tables-only debug info on purpose, and the override was discarding the
line numbers that make a panic backtrace useful.

Finally, run the freshly pushed image on its native runner. Nothing in the
workflow executed the binary, which is exactly why a SIGILL shipped past a
fully green run.

## Test Plan

1. actionlint passes.
2. The new smoke step fails the arm64 job when the Graviton flag is present
   and passes once it is gone.
3. Pull the arm64 image on Apple Silicon, which has no SVE, and confirm
   loreserver --version prints.

Signed-off-by: Aleksander Arruda <aleksander.arruda@xa.epicgames.com>
@github-actions github-actions Bot added area:server Server, provider integrations, telemetry area:ci CI workflows and GitHub configuration labels Aug 28, 2026

@duncangrist duncangrist left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you ensure you run a /code-review in CC as my review brought up a few things.

Also, two CI jobs are failing currently.

## Summary

Review of the publish workflow turned up six problems, three of them able
to publish or sign the wrong thing.

Tags and labels are now resolved once, in a new `meta` job that runs before
anything is built. That addresses three of them together:

- `latest` was gated only on the ref starting with refs/tags/v, so
  v1.0.0-rc.1 moved and signed `latest`, and a v-prefixed tag that is not
  semver at all left `latest` as the only tag the run published. The
  default `latest=auto` flavor applies it only to a tag that parses as a
  stable semver release.
- A workflow_dispatch from a branch with a blank tag input resolved to no
  tags at all, and the job then died on `imagetools inspect ghcr.io/...:`,
  after both arch images had been pushed. A type=ref,event=branch entry
  gives that path a tag from the ref alone, as the input description
  already promised, and a gate in the meta job fails the run before any
  layer reaches GHCR if the list is still empty.
- The build job ran its own metadata-action with no tags config, so the
  labels baked into the image carried the ref-derived version (v1.2.3, or
  a branch name) while the published tag was 1.2.3.

And three smaller ones:

- The tag input was interpolated straight into the tags list, where an
  embedded newline would have smuggled in a further directive. It is held
  to Docker's tag grammar before it gets there.
- imagetools create never received the annotations, so the published index
  carried none. Building its argument list as an array also removes the
  word splitting the old command relied on, which would have broken on the
  spaces in image.description.
- The cosign identity regexp was anchored on the repository alone, so it
  accepted a signature from any workflow here that can ask for an OIDC
  token. It is anchored on this workflow's path now, and defined once,
  since the summary hands the same expression to downstream users.

Separately, the image was built with --release, which Cargo.toml labels a
fast build suitable for local development: debug-assertions, and with them
overflow-checks, stay on and there is no LTO. Acceptable for a local
convenience image, not for the artifact the Helm chart points at. Build
release-lto, "the actual release build", and strip the full DWARF it asks
for back out, keeping the symbol table so backtraces still name frames.

## Test Plan

1. actionlint, with the repo config, reports no errors.
2. docker buildx build --check on the Dockerfile is clean.
3. The tag validator accepts edge, 1.2.3 and v1.0.0-rc.1, and rejects an
   embedded newline, a leading dash, a leading dot, a space, a slash and a
   129-character input.
4. The imagetools argument construction, dry-run against a representative
   metadata-action JSON, keeps image.description as one argument.
5. Still to confirm on the first run: fat LTO links inside the memory and
   time a 4-core runner has, a prerelease tag does not move latest, and a
   branch dispatch publishes under the branch name.

Signed-off-by: Aleksander Arruda <aleksander.arruda@xa.epicgames.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@aleksanderllada
aleksanderllada force-pushed the aarruda/publish-loreserver-image branch from 13fcea7 to aaee068 Compare September 1, 2026 13:04
## Summary

Lore is tuned for and deployed on Graviton3+, so a tuned arm64 image now
ships as well — but as an opt-in `-graviton` tag rather than as the default,
because `-C target-cpu=neoverse-512tvb` faults on any older arm64 part and
the default tag is what a community chart pulls.

    X.Y.Z, X.Y, latest              arm64 baseline armv8-a
    X.Y.Z-graviton, latest-graviton arm64 tuned for Graviton3+

The Dockerfile now assembles RUSTFLAGS itself from an ARM64_TARGET_CPU build
arg, defaulting to empty. amd64 is baseline in both variants, so it is built
once and both manifest lists reference that single digest — the extra arm64
leg runs in parallel and leaves the critical path where it was.

OCI cannot express this: platform.variant covers v6/v7/v8, not
microarchitecture, so a client cannot be steered to the right arm64 build
automatically. Separate tags are the only honest mechanism.

## Also corrects the smoke test's comment

The comment claimed the step catches codegen aimed at the wrong CPU. It does
not. GitHub's arm64 runner is Neoverse-N2 and reports SVE, so it executes the
Graviton-tuned build without complaint — measured, not assumed. The step is
still worth keeping for a broken entrypoint or a missing shared library, and
the comment now says only that.

## Test Plan

1. actionlint passes.
2. Dispatch from a branch and confirm two manifest lists publish, each
   carrying linux/amd64 and linux/arm64, each signed and verified.
3. Confirm both lists reference the same amd64 digest.
4. Run the default arm64 image on Apple Silicon, which has no SVE, and
   confirm loreserver starts; confirm the -graviton arm64 image does not.

Signed-off-by: Aleksander Arruda <aleksander.arruda@xa.epicgames.com>
release-lto sets debug = 2, and a fat-LTO link holds the whole dependency
graph's DWARF in memory at once. On a 16 GB runner rustc is SIGKILLed part
way through linking loreserver, so both the amd64 and the Graviton arm64
builds died.

Build with -C debuginfo=0. strip --strip-debug discarded that DWARF a line
later regardless, and tracing's file and line fields come from compile-time
macros rather than debug info, so nothing observable is lost.

Signed-off-by: Aleksander Arruda <aleksander.arruda@xa.epicgames.com>
@aleksanderllada aleksanderllada changed the title Build and publish a signed multi-arch loreserver image to GHCR Build and publish signed multi-arch loreserver images to GHCR Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ci CI workflows and GitHub configuration area:server Server, provider integrations, telemetry

Development

Successfully merging this pull request may close these issues.

2 participants