Build and publish signed multi-arch loreserver images to GHCR - #180
Open
aleksanderllada wants to merge 6 commits into
Open
Build and publish signed multi-arch loreserver images to GHCR#180aleksanderllada wants to merge 6 commits into
aleksanderllada wants to merge 6 commits into
Conversation
## Summary Fills in the publish workflow: each architecture builds on its own native runner and is pushed to GHCR by digest, a merge job stitches the digests into one manifest list, and cosign signs that list keylessly through the job's OIDC token. No secrets are involved. Also drops the Graviton3+ target-cpu from the arm64 container build. The repo tunes aarch64-unknown-linux-gnu for Epic's own deployment, which emits instructions that fault on older arm64 parts and, because the build passes no --target, reaches the proc macros cargo runs on the build host too. A published image has to run anywhere armv8-a runs. ## Test Plan 1. actionlint passes on the workflow. 2. Dispatch from this branch and confirm the manifest list carries linux/amd64 and linux/arm64, and that cosign verify succeeds. 3. Pull the arm64 image on a non-Graviton arm64 host and confirm the server starts. Signed-off-by: Aleksander Arruda <aleksander.arruda@xa.epicgames.com>
The first full run measured 145G on the runner root with 108G still free after both release builds, so deleting unrelated preinstalled toolchains buys nothing and only adds a step that can fail on its own. Signed-off-by: Aleksander Arruda <aleksander.arruda@xa.epicgames.com>
## Summary The previous attempt set CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_RUSTFLAGS to drop the Graviton3+ target-cpu. That does not work: the variable is another source for the same config key, and cargo joins config arrays rather than replacing them, so -C target-cpu=neoverse-512tvb survived and the published arm64 image died with SIGILL on any older arm64 part. Use RUSTFLAGS instead, which is a mutually exclusive source that suppresses [build] and [target.*] rustflags outright, and repeat the flags the build actually needs. Also stop overriding the release strip setting. The workspace asks for line-tables-only debug info on purpose, and the override was discarding the line numbers that make a panic backtrace useful. Finally, run the freshly pushed image on its native runner. Nothing in the workflow executed the binary, which is exactly why a SIGILL shipped past a fully green run. ## Test Plan 1. actionlint passes. 2. The new smoke step fails the arm64 job when the Graviton flag is present and passes once it is gone. 3. Pull the arm64 image on Apple Silicon, which has no SVE, and confirm loreserver --version prints. Signed-off-by: Aleksander Arruda <aleksander.arruda@xa.epicgames.com>
duncangrist
requested changes
Sep 1, 2026
duncangrist
left a comment
Contributor
There was a problem hiding this comment.
Can you ensure you run a /code-review in CC as my review brought up a few things.
Also, two CI jobs are failing currently.
## Summary Review of the publish workflow turned up six problems, three of them able to publish or sign the wrong thing. Tags and labels are now resolved once, in a new `meta` job that runs before anything is built. That addresses three of them together: - `latest` was gated only on the ref starting with refs/tags/v, so v1.0.0-rc.1 moved and signed `latest`, and a v-prefixed tag that is not semver at all left `latest` as the only tag the run published. The default `latest=auto` flavor applies it only to a tag that parses as a stable semver release. - A workflow_dispatch from a branch with a blank tag input resolved to no tags at all, and the job then died on `imagetools inspect ghcr.io/...:`, after both arch images had been pushed. A type=ref,event=branch entry gives that path a tag from the ref alone, as the input description already promised, and a gate in the meta job fails the run before any layer reaches GHCR if the list is still empty. - The build job ran its own metadata-action with no tags config, so the labels baked into the image carried the ref-derived version (v1.2.3, or a branch name) while the published tag was 1.2.3. And three smaller ones: - The tag input was interpolated straight into the tags list, where an embedded newline would have smuggled in a further directive. It is held to Docker's tag grammar before it gets there. - imagetools create never received the annotations, so the published index carried none. Building its argument list as an array also removes the word splitting the old command relied on, which would have broken on the spaces in image.description. - The cosign identity regexp was anchored on the repository alone, so it accepted a signature from any workflow here that can ask for an OIDC token. It is anchored on this workflow's path now, and defined once, since the summary hands the same expression to downstream users. Separately, the image was built with --release, which Cargo.toml labels a fast build suitable for local development: debug-assertions, and with them overflow-checks, stay on and there is no LTO. Acceptable for a local convenience image, not for the artifact the Helm chart points at. Build release-lto, "the actual release build", and strip the full DWARF it asks for back out, keeping the symbol table so backtraces still name frames. ## Test Plan 1. actionlint, with the repo config, reports no errors. 2. docker buildx build --check on the Dockerfile is clean. 3. The tag validator accepts edge, 1.2.3 and v1.0.0-rc.1, and rejects an embedded newline, a leading dash, a leading dot, a space, a slash and a 129-character input. 4. The imagetools argument construction, dry-run against a representative metadata-action JSON, keeps image.description as one argument. 5. Still to confirm on the first run: fat LTO links inside the memory and time a 4-core runner has, a prerelease tag does not move latest, and a branch dispatch publishes under the branch name. Signed-off-by: Aleksander Arruda <aleksander.arruda@xa.epicgames.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
aleksanderllada
force-pushed
the
aarruda/publish-loreserver-image
branch
from
September 1, 2026 13:04
13fcea7 to
aaee068
Compare
## Summary
Lore is tuned for and deployed on Graviton3+, so a tuned arm64 image now
ships as well — but as an opt-in `-graviton` tag rather than as the default,
because `-C target-cpu=neoverse-512tvb` faults on any older arm64 part and
the default tag is what a community chart pulls.
X.Y.Z, X.Y, latest arm64 baseline armv8-a
X.Y.Z-graviton, latest-graviton arm64 tuned for Graviton3+
The Dockerfile now assembles RUSTFLAGS itself from an ARM64_TARGET_CPU build
arg, defaulting to empty. amd64 is baseline in both variants, so it is built
once and both manifest lists reference that single digest — the extra arm64
leg runs in parallel and leaves the critical path where it was.
OCI cannot express this: platform.variant covers v6/v7/v8, not
microarchitecture, so a client cannot be steered to the right arm64 build
automatically. Separate tags are the only honest mechanism.
## Also corrects the smoke test's comment
The comment claimed the step catches codegen aimed at the wrong CPU. It does
not. GitHub's arm64 runner is Neoverse-N2 and reports SVE, so it executes the
Graviton-tuned build without complaint — measured, not assumed. The step is
still worth keeping for a broken entrypoint or a missing shared library, and
the comment now says only that.
## Test Plan
1. actionlint passes.
2. Dispatch from a branch and confirm two manifest lists publish, each
carrying linux/amd64 and linux/arm64, each signed and verified.
3. Confirm both lists reference the same amd64 digest.
4. Run the default arm64 image on Apple Silicon, which has no SVE, and
confirm loreserver starts; confirm the -graviton arm64 image does not.
Signed-off-by: Aleksander Arruda <aleksander.arruda@xa.epicgames.com>
release-lto sets debug = 2, and a fat-LTO link holds the whole dependency graph's DWARF in memory at once. On a 16 GB runner rustc is SIGKILLed part way through linking loreserver, so both the amd64 and the Graviton arm64 builds died. Build with -C debuginfo=0. strip --strip-debug discarded that DWARF a line later regardless, and tracing's file and line fields come from compile-time macros rather than debug info, so nothing observable is lost. Signed-off-by: Aleksander Arruda <aleksander.arruda@xa.epicgames.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fills in
publish-loreserver-image.yml, which landed earlier as a stub. Two variants publish, differing only in how arm64 is compiled:X.Y.Z,X.Y,latestarmv8-a— runs on any arm64 hostX.Y.Z-graviton,latest-gravitonThe portable build is the default so a community chart works everywhere, and the tuned build is opt-in for Graviton deployments.
linux/amd64is baseline in both, so it is built once and both manifest lists reference that single digest.Each variant builds on its own native runner (QEMU is far too slow for a release Rust build) and is pushed by digest; a merge job stitches the digests into manifest lists and signs each with keyless cosign. No secrets are involved:
GITHUB_TOKENauthenticates to GHCR and cosign signs against Fulcio via the job's OIDC token.Triggers on
v*tags, plus manual dispatch with an optional extra tag for proving the workflow from a branch.Why two tags rather than one
OCI cannot express this.
platform.variantcoversv6/v7/v8, not microarchitecture, and nothing in the pull path consults CPU features — Graviton3, Ampere Altra and Apple Silicon are all justlinux/arm64. A client cannot be steered to the right arm64 build automatically, so a separate tag is the only honest mechanism.Architecture selection itself is unaffected: one tag still serves both amd64 and arm64, which is why
-gravitonis also multi-arch and a mixed-architecture cluster can pin a single tag.Dockerfile changes worth reviewing
arm64 codegen.
.cargo/config.tomlpinsaarch64-unknown-linux-gnuto Graviton3+ with-C target-cpu=neoverse-512tvb. The Dockerfile now assemblesRUSTFLAGSitself from anARM64_TARGET_CPUbuild arg, empty by default. It has to beRUSTFLAGSand notCARGO_TARGET_<triple>_RUSTFLAGS: the latter is only another source for the same config key and cargo joins config arrays, so the Graviton flag survives and a supposedly-baseline build still faults.Debug info.
[profile.release-lto]setsdebug = 2, and a fat-LTO link holds the whole dependency graph's DWARF at once — rustc was SIGKILLed part way through linking on a 16 GB runner. Building with-C debuginfo=0fixes it and costs nothing observable, becausestrip --strip-debugdiscarded that DWARF a line later anyway andtracing's file/line fields come from compile-time macros. Side effect: the arm64 image drops from 100 MB to 51 MB.DOCKER.mddocuments both variants and the build arg.Test evidence
A full dispatch published both variants, each signed and verified in-run. The amd64 digest is shared, as intended:
Both arm64 images were then run on an Apple M3 Max, which has no SVE:
That is the intended behaviour of both variants, and it is the only test that actually distinguishes them. The default image also boots and serves — 10 QUIC listeners on
0.0.0.0:41337.A limitation to be aware of
The per-arch smoke step runs the freshly pushed image, which catches a broken entrypoint or a missing shared library. It does not catch codegen aimed at a CPU the eventual host lacks: GitHub's arm64 runner is Neoverse-N2 and reports SVE, so it executes the Graviton-tuned build without complaint. This was measured, not assumed, and it is why the Apple Silicon check above matters. A CI gate running the binary under
qemu-aarch64 -cpu cortex-a72would close it and is not yet implemented.Notes
permissions: {}, per-job least privilege with justification comments, and every action pinned to a full commit SHA.ci-testandci-test-gravitontags are leftovers from proving this and will be deleted.mainthrough the usual mirror path rather than by merging here.