Skip to content

Commit baa6cfe

Browse files
feat(e2e): add apps-e2e shared workflow + composite actions
Consumer-side kurtosis-pos e2e harness, shared so adopters (lst-api today, matic.js / proof-generation-api next) call one implementation instead of vendoring copies that drift. - apps-e2e.yml — reusable workflow. Snapshot mode (default, ~30s) restores a published GHCR image via the e2e-snapshot-restore composite; cold-start mode (~10m) runs kurtosis from source via the e2e-cold-start composite for unsnapshotted kurtosis-pos refs. The snapshot/cold-start switch uses `toJSON(inputs.use_snapshot) == 'false'` because Actions' loose equality treats `null == false` as true, which would otherwise drop null-input pull_request runs into cold-start. REGISTRY_USERNAME / REGISTRY_PASSWORD (optional) let a cross-org consumer pull a private snapshot with its own credentials, falling back to the actor + GITHUB_TOKEN when unset. - actions/e2e-snapshot-restore — pull + restore a snapshot image (extract volumes, patch the pos-anvil host port, replay anvil state). - actions/e2e-cold-start — install kurtosis-cli, scale EL resource limits down for ubuntu-latest (verified, fails loud if the patch no-ops on an unexpected ref), run the enclave; exports E2E_KURTOSIS_ENCLAVE for a symmetric consumer adapter. - actions/e2e-dump — capture devnet state on failure (compose logs or enclave dump). A composite, not an inline run: block or a .github/scripts/ helper, because a reusable workflow can't reach a repo-local script cross-repo. - apps-e2e-trigger.yml — canonical thin trigger, guarded against firing on pipelines' own PRs. - README — workflow + composite tables and usage notes. Out of scope: the snapshot publisher stays in lst-api against its GHCR namespace; moving + renaming it is a separate refactor. Generic suite-secret injection is documented as a known limitation — named secrets only, never `secrets: inherit` (Team Standards).
1 parent 00ffd9b commit baa6cfe

9 files changed

Lines changed: 863 additions & 0 deletions

File tree

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: E2E cold-start
2+
description: >
3+
Bring up a kurtosis-pos devnet from source. Installs the kurtosis CLI,
4+
starts the engine, clones kurtosis-pos at the requested ref, scales EL
5+
resource limits down to fit a 2-vCPU / 7-GB ubuntu-latest runner, and
6+
runs the enclave from the consumer's args file.
7+
8+
This is the escape hatch for iterating on a kurtosis-pos branch that
9+
hasn't been snapshotted yet. ~10 minute wall-time vs the ~30 second
10+
snapshot path; consumers default to the snapshot and reach for cold
11+
start only when they need an unpublished kurtosis-pos commit.
12+
13+
inputs:
14+
kurtosis_pos_ref:
15+
description: >
16+
Git ref of 0xPolygon/kurtosis-pos to clone (branch, tag, or SHA).
17+
required: false
18+
default: main
19+
args_file:
20+
description: >
21+
Path to the kurtosis args file in the consumer's checkout (relative
22+
to GITHUB_WORKSPACE). Each consumer ships its own — the args dictate
23+
L1 backend, contract deployments, and chain IDs that the enclave
24+
brings up.
25+
required: false
26+
default: kurtosis-params.yml
27+
work_dir:
28+
description: >
29+
Scratch dir for the kurtosis-pos clone. Default `/tmp/kurtosis-pos`.
30+
required: false
31+
default: /tmp/kurtosis-pos
32+
33+
runs:
34+
using: composite
35+
steps:
36+
- name: Cold-start kurtosis-pos enclave
37+
shell: bash
38+
env:
39+
KURTOSIS_POS_REF: ${{ inputs.kurtosis_pos_ref }}
40+
ARGS_FILE: ${{ inputs.args_file }}
41+
WORK_DIR: ${{ inputs.work_dir }}
42+
run: bash "${GITHUB_ACTION_PATH}/cold-start.sh"
Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Cold-start a kurtosis-pos devnet from source.
4+
#
5+
# Used by consumers who need to test against an unpublished kurtosis-pos
6+
# branch (the snapshot publisher only refreshes weekly + only off `main`).
7+
# Total wall-time: ~10 minutes vs ~30s for the snapshot path. The snapshot
8+
# is the default; this is the escape hatch.
9+
#
10+
# # Inputs (env vars, set by the composite action with documented defaults)
11+
#
12+
# KURTOSIS_POS_REF git ref of 0xPolygon/kurtosis-pos to clone.
13+
# ARGS_FILE kurtosis args file in the consumer's checkout
14+
# (resolved against GITHUB_WORKSPACE).
15+
# WORK_DIR scratch dir for the clone.
16+
#
17+
# # After this script exits 0
18+
#
19+
# `kurtosis enclave inspect pos` reports RUNNING. Downstream steps reach
20+
# the chain via the kurtosis-emitted env vars (each consumer's e2e suite
21+
# already has its own kurtosis adapter — this script doesn't export
22+
# anything to GITHUB_ENV).
23+
24+
set -euo pipefail
25+
26+
KURTOSIS_POS_REF="${KURTOSIS_POS_REF:-main}"
27+
ARGS_FILE="${ARGS_FILE:-kurtosis-params.yml}"
28+
WORK_DIR="${WORK_DIR:-/tmp/kurtosis-pos}"
29+
30+
REPO_ROOT="${GITHUB_WORKSPACE:-$PWD}"
31+
ARGS_FILE_ABS="${REPO_ROOT}/${ARGS_FILE}"
32+
if [ ! -f "$ARGS_FILE_ABS" ]; then
33+
echo "[e2e-cold-start] ERROR: args file not found at ${ARGS_FILE_ABS}" >&2
34+
echo "[e2e-cold-start] Pass the path relative to your repo root via the action's args_file input." >&2
35+
exit 1
36+
fi
37+
38+
##############################################################################
39+
# 1. Install kurtosis CLI
40+
##############################################################################
41+
if command -v kurtosis >/dev/null 2>&1; then
42+
echo "[e2e-cold-start] kurtosis already installed: $(kurtosis version | head -n1)"
43+
else
44+
echo "[e2e-cold-start] installing kurtosis-cli via apt"
45+
echo "deb [trusted=yes] https://apt.fury.io/kurtosis-tech/ /" \
46+
| sudo tee /etc/apt/sources.list.d/kurtosis.list >/dev/null
47+
sudo apt-get update -qq
48+
sudo apt-get install -y --no-install-recommends kurtosis-cli
49+
# Kurtosis prompts for analytics consent on first interactive use; disable
50+
# explicitly so the CLI never blocks on stdin in a non-TTY shell.
51+
kurtosis analytics disable
52+
fi
53+
54+
##############################################################################
55+
# 2. Start the kurtosis engine
56+
##############################################################################
57+
echo "[e2e-cold-start] starting kurtosis engine"
58+
kurtosis engine start
59+
60+
##############################################################################
61+
# 3. Clone kurtosis-pos and run the enclave
62+
##############################################################################
63+
if kurtosis enclave inspect pos 2>/dev/null | grep -q RUNNING; then
64+
echo "[e2e-cold-start] enclave 'pos' already running"
65+
exit 0
66+
fi
67+
68+
if [ ! -d "$WORK_DIR" ]; then
69+
echo "[e2e-cold-start] cloning kurtosis-pos@${KURTOSIS_POS_REF} into ${WORK_DIR}"
70+
git clone --depth 1 --branch "$KURTOSIS_POS_REF" \
71+
https://github.com/0xPolygon/kurtosis-pos.git "$WORK_DIR"
72+
fi
73+
74+
# kurtosis-pos hardcodes `MAX_CPU = 4000, MAX_MEM = 16384` for EL services,
75+
# which exceeds what private-repo GitHub-hosted runners provide (2 vCPU,
76+
# 7 GB). Scale down in place so the enclave fits on ubuntu-latest. On a
77+
# developer machine with more headroom this rewrite is a no-op from the
78+
# suite's perspective — the services just use less of the available
79+
# capacity. Remove once kurtosis-pos exposes these as inputs.
80+
#
81+
# This patch is brittle by nature: it depends on the file path and the
82+
# variable names. `kurtosis_pos_ref` is consumer-overridable, so an
83+
# unexpected ref could rename either and make the sed a silent no-op —
84+
# the enclave would then request 16 GB and the runner OOM-kills it with a
85+
# confusing failure. Verify the override actually landed and fail loud if
86+
# not, so the cause is unambiguous.
87+
shared_star="$WORK_DIR/src/el/shared.star"
88+
if [ ! -f "$shared_star" ]; then
89+
echo "[e2e-cold-start] ERROR: expected ${shared_star} not found — kurtosis-pos@${KURTOSIS_POS_REF} may have moved the EL resource config. Update cold-start.sh for this ref." >&2
90+
exit 1
91+
fi
92+
sed -i.bak -E \
93+
-e 's/^MAX_CPU = [0-9]+.*/MAX_CPU = 1800 # CI override (ubuntu-latest 2-vCPU private-repo runner)/' \
94+
-e 's/^MAX_MEM = [0-9]+.*/MAX_MEM = 4096 # CI override (ubuntu-latest 7-GB private-repo runner)/' \
95+
"$shared_star"
96+
rm -f "${shared_star}.bak"
97+
# Confirm both overrides applied — grep for the marker comment the sed adds.
98+
# Two matches expected (MAX_CPU + MAX_MEM); anything less means a var was
99+
# renamed and we'd silently run at full resource request.
100+
override_count="$(grep -c 'CI override (ubuntu-latest' "$shared_star" || true)"
101+
if [ "$override_count" -ne 2 ]; then
102+
echo "[e2e-cold-start] ERROR: EL resource scale-down did not apply (${override_count}/2 markers found in ${shared_star}). kurtosis-pos@${KURTOSIS_POS_REF} likely renamed MAX_CPU/MAX_MEM — the runner would OOM. Update the sed patterns for this ref." >&2
103+
exit 1
104+
fi
105+
106+
echo "[e2e-cold-start] running enclave 'pos' (this takes ~5–10m)"
107+
( cd "$WORK_DIR" && kurtosis run --enclave pos --args-file "$ARGS_FILE_ABS" . )
108+
109+
# Export the enclave name so the consumer's e2e suite can locate it via the
110+
# kurtosis CLI in cold-start mode (symmetric with snapshot mode exporting
111+
# E2E_SNAPSHOT_ADDRESSES_JSON). Lets a consumer write a single env-driven
112+
# adapter instead of hardcoding the enclave name per mode.
113+
if [ -n "${GITHUB_ENV:-}" ]; then
114+
echo "E2E_KURTOSIS_ENCLAVE=pos" >> "$GITHUB_ENV"
115+
fi
116+
117+
echo "[e2e-cold-start] ready"
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
name: E2E devnet dump
2+
description: >
3+
Capture devnet state into /tmp/devnet-dump for post-mortem after an e2e
4+
failure. Snapshot mode dumps the docker-compose stack's ps + logs and the
5+
compose file; cold-start mode dumps the kurtosis enclave. Best-effort —
6+
individual failures are swallowed so a half-restored run still yields
7+
whatever artefacts it can.
8+
9+
Bundled as a composite (not a reusable-workflow inline step or a
10+
.github/scripts/ helper) so the shell logic is locally runnable AND
11+
reachable when apps-e2e.yml runs in a consumer's checkout — a raw script
12+
under .github/scripts/ is not on disk cross-repo; ${{ github.action_path }}
13+
is.
14+
15+
inputs:
16+
use_snapshot:
17+
description: '"1" for snapshot mode (compose dump), "0" for cold-start (enclave dump).'
18+
required: true
19+
compose_file:
20+
description: >
21+
Snapshot mode: path to the restored docker-compose.yaml (the restore
22+
composite's compose_file_path output). May be empty if the restore
23+
failed before writing it — the dump guards for that.
24+
required: false
25+
default: ''
26+
enclave:
27+
description: 'Cold-start mode: kurtosis enclave name to dump.'
28+
required: false
29+
default: pos
30+
31+
runs:
32+
using: composite
33+
steps:
34+
- name: Dump devnet state
35+
shell: bash
36+
env:
37+
E2E_USE_SNAPSHOT: ${{ inputs.use_snapshot }}
38+
COMPOSE_FILE: ${{ inputs.compose_file }}
39+
E2E_KURTOSIS_ENCLAVE: ${{ inputs.enclave }}
40+
run: bash "${GITHUB_ACTION_PATH}/dump-devnet.sh"
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Capture devnet state into /tmp/devnet-dump for post-mortem after an e2e
4+
# failure. Invoked by the e2e-dump composite action. Two modes, picked by
5+
# E2E_USE_SNAPSHOT:
6+
#
7+
# - Snapshot mode (=1): dump the docker-compose stack's `ps` + `logs` and
8+
# copy the compose file itself. COMPOSE_FILE is the path the restore
9+
# composite emitted; it may be unset/absent if the restore failed before
10+
# writing it, so every step is guarded.
11+
# - Cold-start mode (=0): dump the kurtosis enclave named by
12+
# E2E_KURTOSIS_ENCLAVE (default `pos`).
13+
#
14+
# Best-effort throughout: a half-restored run should still yield whatever
15+
# artefacts it can, so individual failures are swallowed with `|| true`
16+
# rather than aborting the dump.
17+
#
18+
# Environment (set by action.yml):
19+
# E2E_USE_SNAPSHOT "1" for snapshot mode, "0" for cold-start.
20+
# COMPOSE_FILE snapshot mode: path to the restored docker-compose.
21+
# E2E_KURTOSIS_ENCLAVE cold-start mode: enclave name (default `pos`).
22+
23+
set -euo pipefail
24+
25+
DUMP_DIR="/tmp/devnet-dump"
26+
mkdir -p "$DUMP_DIR"
27+
28+
if [ "${E2E_USE_SNAPSHOT:-1}" = "1" ]; then
29+
compose_file="${COMPOSE_FILE:-}"
30+
if [ -n "$compose_file" ] && [ -f "$compose_file" ]; then
31+
docker compose --file "$compose_file" ps > "${DUMP_DIR}/compose-ps.txt" 2>&1 || true
32+
docker compose --file "$compose_file" logs --no-color > "${DUMP_DIR}/compose-logs.txt" 2>&1 || true
33+
cp "$compose_file" "${DUMP_DIR}/docker-compose.yaml" || true
34+
else
35+
echo "[dump-devnet] no compose file at '${compose_file:-<unset>}' — restore likely failed before writing it" >&2
36+
fi
37+
else
38+
enclave="${E2E_KURTOSIS_ENCLAVE:-pos}"
39+
kurtosis enclave dump "$enclave" "${DUMP_DIR}/${enclave}" || true
40+
fi
Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
name: E2E snapshot restore
2+
description: >
3+
Restore a kurtosis-pos devnet from a published snapshot image. Pulls the image
4+
from GHCR, extracts volumes + docker-compose + addresses sidecar, patches the
5+
pos-anvil host-port mapping, brings the devnet up, and replays the captured
6+
anvil state. Exports E2E_SNAPSHOT_ADDRESSES_JSON to GITHUB_ENV so consumer
7+
steps inherit the path to the addresses sidecar.
8+
9+
inputs:
10+
image:
11+
description: >
12+
Snapshot image reference (full registry/repo:tag). Default points at the
13+
lst-api-published snapshot. Pin to a SHA tag (e.g. ":22c449550213") to
14+
lock to a specific kurtosis-pos commit.
15+
required: false
16+
default: ghcr.io/0xpolygon/lst-api-e2e-snapshot:latest
17+
kurtosis_pos_ref:
18+
description: >
19+
Git ref of 0xPolygon/kurtosis-pos to clone for the upstream extract.sh /
20+
restore.sh helpers. Should match the ref used to build the snapshot —
21+
the extract/restore scripts are tightly paired with their snapshot
22+
contents. Default 'main' tracks the snapshot publisher's default.
23+
required: false
24+
default: main
25+
out_dir:
26+
description: >
27+
Directory the snapshot is extracted into, relative to the consumer's
28+
checkout root. Default './tmp/e2e-snapshot' avoids clobbering anything
29+
else under ./tmp.
30+
required: false
31+
default: ./tmp/e2e-snapshot
32+
registry:
33+
description: >
34+
Container registry to authenticate against. Default 'ghcr.io'. Override
35+
only when the snapshot image lives elsewhere.
36+
required: false
37+
default: ghcr.io
38+
registry_username:
39+
description: >
40+
Username for the registry login. Defaults to the workflow actor; override
41+
when calling with a service account or PAT.
42+
required: false
43+
default: ${{ github.actor }}
44+
registry_password:
45+
description: >
46+
Token for the registry login. Defaults to GITHUB_TOKEN, which works for
47+
same-org GHCR pulls when the calling workflow grants `packages: read`.
48+
required: false
49+
default: ${{ github.token }}
50+
51+
outputs:
52+
addresses_json_path:
53+
description: Absolute path to the extracted addresses sidecar JSON.
54+
value: ${{ steps.restore.outputs.addresses_json_path }}
55+
compose_file_path:
56+
description: Absolute path to the bundled docker-compose.yaml.
57+
value: ${{ steps.restore.outputs.compose_file_path }}
58+
59+
runs:
60+
using: composite
61+
steps:
62+
- name: Log in to container registry
63+
shell: bash
64+
env:
65+
REGISTRY: ${{ inputs.registry }}
66+
REGISTRY_USERNAME: ${{ inputs.registry_username }}
67+
REGISTRY_PASSWORD: ${{ inputs.registry_password }}
68+
run: echo "$REGISTRY_PASSWORD" | docker login "$REGISTRY" -u "$REGISTRY_USERNAME" --password-stdin
69+
70+
- name: Restore devnet from snapshot
71+
id: restore
72+
shell: bash
73+
env:
74+
IMAGE: ${{ inputs.image }}
75+
KURTOSIS_POS_REF: ${{ inputs.kurtosis_pos_ref }}
76+
OUT_DIR: ${{ inputs.out_dir }}
77+
run: bash "${GITHUB_ACTION_PATH}/restore.sh"
78+
79+
- name: Export snapshot addresses path to GITHUB_ENV
80+
shell: bash
81+
env:
82+
ADDRS_PATH: ${{ steps.restore.outputs.addresses_json_path }}
83+
run: |
84+
# The consuming repo's e2e suite reads this env var to switch from the
85+
# kurtosis CLI path to the sidecar JSON path. Writing to GITHUB_ENV
86+
# propagates it to every subsequent step in the calling job without
87+
# the consumer having to know the magic name.
88+
echo "E2E_SNAPSHOT_ADDRESSES_JSON=${ADDRS_PATH}" >> "$GITHUB_ENV"

0 commit comments

Comments
 (0)