Skip to content

Commit 49c9210

Browse files
willemnealclaude
willemneal
and
claude
committed
fix: start mainnet archive pipeline at Soroban launch (#29)
The mainnet archive pipeline started at ARCHIVE_START_AT=63359333 (the registry's own deploy on 2026-07-06), so contracts deployed on-chain before that — i.e. everything added via `register-contract` — had no archive.deploys row and never enriched in v1.contracts_enriched, breaking the contract→wasm link (e.g. tansu shows blank wasm_name/version/channel and v1.versions returns 0 rows). Lower ARCHIVE_START_AT to 50457423 (Soroban/Protocol 20 went live on mainnet at ledger 50457424; start one earlier). No contract can predate Soroban, so this captures every mainnet contract deploy and wasm upload. No SQL/view change is needed — versions/contracts_enriched populate automatically once archive.* is backfilled. Applying an earlier start_at requires a clear-state replay; `turbo apply` alone never rewinds the checkpoint. Add .github/workflows/redeploy-pipeline.yml: a guarded, manual workflow_dispatch that renders a network pipeline and runs redeploy.sh (DROP schema + turbo restart --clear-state). Deliberately not wired to merge — each run wipes the schema and triggers a multi-hour replay. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019acVJ66J2HeRa47kQvT8oo
1 parent e721a35 commit 49c9210

2 files changed

Lines changed: 153 additions & 5 deletions

File tree

‎.github/workflows/redeploy-pipeline.yml‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
name: Redeploy Goldsky Pipeline (clear-state backfill)
2+
3+
# Manual, destructive redeploy of a single Goldsky pipeline: render the
4+
# network template, DROP its schema, re-apply, and `turbo restart
5+
# --clear-state` so the source checkpoint rewinds to the pipeline's
6+
# start_at and the full history replays into the freshly-dropped sink
7+
# tables.
8+
#
9+
# This is the only way to pull in history EARLIER than the pipeline's
10+
# current checkpoint (e.g. lowering ARCHIVE_START_AT to backfill contract
11+
# deploys that predate the registry — see stellar-registry/indexer#29).
12+
# A plain `turbo apply` cannot do it: applying a changed start_at updates
13+
# the definition but never rewinds the checkpoint.
14+
#
15+
# Deliberately workflow_dispatch only — NOT wired to push/merge. Each run
16+
# wipes the pipeline's schema and triggers a multi-hour replay, so it must
17+
# be an explicit, one-off operator action guarded by `confirm`.
18+
19+
on:
20+
workflow_dispatch:
21+
inputs:
22+
network:
23+
description: "Network (matches goldsky/networks/<network>.env)"
24+
required: true
25+
type: choice
26+
options:
27+
- mainnet
28+
- testnet
29+
default: mainnet
30+
pipeline:
31+
description: "Pipeline to redeploy"
32+
required: true
33+
type: choice
34+
options:
35+
- archive
36+
- v1
37+
default: archive
38+
confirm:
39+
description: "Type the exact pipeline name to confirm the DROP (e.g. contract-lifecycle-mainnet)"
40+
required: true
41+
type: string
42+
subregistries:
43+
description: "v1 only: expected initial subregistry count for the race audit (leave blank for archive)"
44+
required: false
45+
type: string
46+
default: ""
47+
48+
concurrency:
49+
group: redeploy-pipeline
50+
cancel-in-progress: false
51+
52+
permissions:
53+
contents: read
54+
55+
jobs:
56+
redeploy:
57+
runs-on: ubuntu-24.04
58+
timeout-minutes: 20
59+
steps:
60+
- uses: actions/checkout@v5
61+
62+
- name: Guard — confirm matches the pipeline being dropped
63+
env:
64+
NETWORK: ${{ inputs.network }}
65+
PIPELINE: ${{ inputs.pipeline }}
66+
CONFIRM: ${{ inputs.confirm }}
67+
run: |
68+
set -euo pipefail
69+
case "$PIPELINE" in
70+
archive) expected="contract-lifecycle-$NETWORK" ;;
71+
v1) expected="registry-$NETWORK-v1" ;;
72+
*) echo "error: unknown pipeline '$PIPELINE'" >&2; exit 1 ;;
73+
esac
74+
if [[ "$CONFIRM" != "$expected" ]]; then
75+
echo "error: confirm '$CONFIRM' does not match '$expected' — refusing to drop." >&2
76+
exit 1
77+
fi
78+
echo "confirmed: will redeploy '$expected' (drops its schema and replays from start_at)"
79+
80+
- name: Install psql and envsubst
81+
run: |
82+
set -euo pipefail
83+
sudo apt-get update
84+
sudo apt-get install -y --no-install-recommends postgresql-client gettext-base
85+
86+
- name: Install Goldsky turbo CLI
87+
run: |
88+
set -euo pipefail
89+
curl -fsSL https://install-turbo.goldsky.com | bash
90+
echo "$HOME/.goldsky/bin" >> "$GITHUB_PATH"
91+
92+
- name: Authenticate Goldsky CLI
93+
env:
94+
GOLDSKY_API_KEY: ${{ secrets.GOLDSKY_API_KEY }}
95+
run: |
96+
set -euo pipefail
97+
if [[ -z "${GOLDSKY_API_KEY:-}" ]]; then
98+
echo "error: GOLDSKY_API_KEY secret is not set" >&2
99+
exit 1
100+
fi
101+
mkdir -p "$HOME/.goldsky"
102+
printf '%s' "$GOLDSKY_API_KEY" > "$HOME/.goldsky/auth_token"
103+
chmod 600 "$HOME/.goldsky/auth_token"
104+
105+
- name: Render pipeline definition
106+
env:
107+
NETWORK: ${{ inputs.network }}
108+
PIPELINE: ${{ inputs.pipeline }}
109+
run: |
110+
set -euo pipefail
111+
./goldsky/scripts/render.sh "$NETWORK" "$PIPELINE"
112+
113+
- name: Redeploy (drop + clear-state replay)
114+
env:
115+
NETWORK: ${{ inputs.network }}
116+
PIPELINE: ${{ inputs.pipeline }}
117+
SUBREGISTRIES: ${{ inputs.subregistries }}
118+
MAINNET_DATABASE_URL: ${{ secrets.MAINNET_DATABASE_URL }}
119+
TESTNET_DATABASE_URL: ${{ secrets.TESTNET_DATABASE_URL }}
120+
run: |
121+
set -euo pipefail
122+
case "$NETWORK" in
123+
mainnet) DATABASE_URL="$MAINNET_DATABASE_URL" ;;
124+
testnet) DATABASE_URL="$TESTNET_DATABASE_URL" ;;
125+
*) echo "error: unknown network '$NETWORK'" >&2; exit 1 ;;
126+
esac
127+
if [[ -z "${DATABASE_URL:-}" ]]; then
128+
echo "error: DATABASE_URL secret for '$NETWORK' is not set (expected ${NETWORK^^}_DATABASE_URL)" >&2
129+
exit 1
130+
fi
131+
echo "::add-mask::$DATABASE_URL"
132+
export DATABASE_URL
133+
134+
dir="goldsky/rendered/$NETWORK/$PIPELINE"
135+
args=()
136+
if [[ "$PIPELINE" == "v1" && -n "${SUBREGISTRIES:-}" ]]; then
137+
args+=(--number-of-initial-subregistries "$SUBREGISTRIES")
138+
fi
139+
140+
# redeploy.sh returns once the clear-state restart is issued; the
141+
# replay itself runs asynchronously on Goldsky (monitor with
142+
# `turbo status <pipeline>` / query the sink tables).
143+
./goldsky/scripts/redeploy.sh "${args[@]}" "$dir"

‎goldsky/networks/mainnet.env‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,16 @@ DATASET_PREFIX=stellar_mainnet
99
# 2026-07-06T20:42:04Z).
1010
V1_START_AT=63359485
1111

12-
# The root registry's UploadContractWasm landed in ledger 63359334
13-
# (2026-07-06T20:27:29Z) — stellar-cli deploys upload and create in two
14-
# separate transactions. Start one ledger before the upload so the
15-
# archive pipeline captures it regardless of start_at edge semantics.
16-
ARCHIVE_START_AT=63359333
12+
# Soroban (Protocol 20) went live on mainnet at ledger 50457424
13+
# (2024-02-20T17:00Z). Start one ledger earlier so the archive pipeline
14+
# captures every contract deploy/upload since Soroban launch. Without this
15+
# the pipeline only saw ledgers from the registry's own deploy (63359334,
16+
# 2026-07-06) onward, so contracts added via `register-contract` — which
17+
# were deployed on-chain before the registry existed — had no
18+
# archive.deploys row and never enriched in v1.contracts_enriched (their
19+
# wasm_name/version/channel stayed blank, breaking the contract→wasm link).
20+
# See stellar-registry/indexer#29.
21+
ARCHIVE_START_AT=50457423
1722

1823
ROOT_REGISTRY=CDU4M3LDIOUJJ5F3YXKJ4EJEP5VPRPG6N2LJ5HOQIMN7MNGL3NS3EGUY
1924

0 commit comments

Comments
 (0)