Predict open-items hardening + localnet harness & capacity experiments#1094
Conversation
- run.sh: SIM_PORT_OFFSET -> isolated --fullnode-rpc-port/--with-faucet so N localnets run concurrently (offset 0 = byte-identical to before). Only client.yaml's RPC url is rewritten; the genesis blob / swarm ports are never touched (they are genesis-disjoint already). - sim.ts: env-gated stress knobs (SIM_STRESS_MINT_BATCH_SIZE / _LEVERAGE / _SINGLE_STRIKE) extending the existing duplicate-mint stress mode; unset = current parity behaviour. - stress/: worktree-pool orchestration (setup_pool.sh, sweep.sh, gen_configs.py audit example) + README documenting the workflow, the stress knobs, and the one-localnet-per-worktree / never-rewrite-genesis-blob gotchas. - rules/predict-simulations.md + move.md: record the measured findings -- the ~5M computation wall (surfaces as InsufficientGas), data-dependent normal_cdf cost (variable flush-OOG threshold), multi-command-PTB dynamic-field scan amplification (~45x/candidate), and the single-PTB full-pool flush capacity. Report-only audit; no contract changes. Findings tracked in the gitignored .claude/predict-review/OPEN-ITEMS.md + .claude/predict-design/audit/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014RN3DbRuMUU6u1fo3E2o1n
packages/predict/harness/ — a worktree-free Sui localnet harness for running Predict against real market data. Self-contained (own harness/ts TS package); does not depend on packages/predict/simulations. - Python (harness/): file-locked slot/port registry, scratch-workspace staging (publish from copies -> zero checkout mutation, N parallel from one clone, no git worktrees), localnet lifecycle, closure publish, wormhole/pyth/account init, dedicated funded updater address. CLI: run / run-many / up / status / cleanup. - 'harness up' = the propbook oracle substrate, held alive: streams real Pyth Pro (Lazer) + Block Scholes onto the on-chain feeds ~1/s over a pre-warmed expiry grid, re-signing Pyth with a local key, stamped with real (clamped) provider timestamps. - harness/ts/: self-contained TS executor (own package.json/tsconfig) — oracle service + Move builders + local re-sign, ported from simulations/src (localPyth + a trimmed artifacts util in place of shared.ts). Provider API keys load from harness/.env (gitignored). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve a semantic instruction ("2x UP @ ~30c, spend $100") into concrete mint
args off-chain and execute it against live Pyth + Block Scholes data on a localnet.
- pricer.ts: float port of pricing::compute_nd2 (SVI total-variance BS tail).
- resolver.ts: probability-targeted strike search + admission curve
(moneyness-capped leverage) + DUSDC-scaled sizing; rejects infeasible combos.
- mintSpike.ts + 'harness spike-mint': oracle-ready localnet -> market + trader
-> resolve + mint. Validated: minted 2x UP @ ~30c, net_premium ~$100.
- runtime.ts: ORACLE_TICK_SIZE -> $0.01 (testnet cadence tick_size, verified on-chain).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
config-gatherer audited testnet (deployment.testnet.json @ predict-testnet-6-24 + live ProtocolConfig/Registry RPC): everything the harness mints matches the contract default except oracle freshness, which testnet loosened from 2s/3s to 10s. - predictConfig.ts: centralized testnet config (per-cadence tick/admission/alloc/cash/ window, oracle freshness, bootstrap supply, resolver market params). - runtime.ts: setOracleFreshnessTx (pin pyth/bs/svi read freshness to 10s/10s/60s). - mintSpike.ts: consume predictConfig; correct 1h cadence sizing ($250k alloc / $50k cash). - Re-validated spike-mint: minted 2x UP @ ~30c, net_premium $100.00. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`harness keeper` runs an off-chain-decided tick loop on a held substrate: roll the cadence (create + fund), flush + settle + compact expired markets (one flush values every active market), and liquidate — each priced PTB self-refreshes the oracle in the same tx. Validated on a 1m cadence: markets create, expire, settle, and compact on live data. - predictSetup.ts: shared bring-up (feeds/config/cadence/freshness/cap) + bare-flush bootstrap (genesis PLP, no market — avoids a fast cadence racing the first expiry). mintSpike refactored onto it. - runtime.ts: keeperFlushTx (multi-market flush + settlement insert_at + in-PTB refresh), keeperLiquidateTx, bareFlushTx, addPythFeedInsert, addOracleRefreshGrid. - keeperService.ts: in-memory market tracking; create-then-fund across txs. - Review fixes: drop dead resolver fields (maxProbability1e9/maxCost1e9 — mint uses U64_MAX) + positionLotSize; fix stale quantity comment; remove unused pricer rangePrice; refresh DESIGN.md. Known follow-up: the keeper self-refreshes (per-tick fetchSnapshot) which can time out on fast cadences; fold onto the updater's shared feed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
harness up now runs keeper + updater together. The keeper is the single setup owner (publishes feeds.json) + runs the lifecycle; the updater is the sole WS consumer (warms the keeper's cadence via GRID_SPEC, writes snapshot.json). The keeper does NO provider/WS work: live valuation reads the updater-fresh on-chain feed, settlement price comes from snapshot.json, and markets are created without a seed (create_expiry_market reads no oracle). Drops the self-refresh + per-tick fetchSnapshot, removing the WS churn and the snapshot-timeout fragility. Validated: harness up (1m cadence, 200s) -- 168 updater pushes / 1 skip, keeper settled + compacted 4 markets, window held, no snapshot timeouts, no liquidate-stale aborts. - predictSetup: setupFeedsAndConfig writes feeds.json; split createMarket (no seed) from createAndSeedMarket (kept for the standalone mint spike). - oracleService: read feeds.json (no own setup); GRID_SPEC-driven grid; write snapshot.json. - keeperService: read snapshot.json for settlement; no self-refresh/fetch. - runtime: keeperFlushTx/keeperLiquidateTx drop the in-PTB refresh (feed is updater-fresh). - live.py/cli.py: harness up = keeper + updater (+ --cadence); removed standalone keeper cmd. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…down) - Rolling grid: the updater re-evaluates its boundary grid each loop (gridNow) and subscribes newly-wanted expiries (ensureExpiries), so the warmed grid rolls forward as boundaries pass and the keeper's new markets stay warm over long runs. - Process-group teardown: keeper + updater launched with start_new_session; teardown killpg's the whole group (npx -> tsx -> node), so no orphans. - Gas auto-refill: the hold loop faucet-funds either actor below 2 SUI (localnet.balance helper). In practice the updater starts ~2000 SUI (~4.6 days at 1/s) so it's a safety net; balance() verified returning correct values. Validated: harness up (1m cadence, 8 min) -- keeper rolled + settled 8 markets continuously past the static-grid window (rolling grid holds), 407 updater pushes / 1 skip over 477s, clean teardown (no orphans). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
harness up --traders N launches N dedicated trader actors. Each is keeper-funded with DUSDC (publisher owns the cap), creates an account, then fuzzes FEASIBLE semantic mints (random direction/leverage/moneyness/spend within the admission curve) + periodic live redeems against the keeper's live markets, pricing off the updater's shared snapshot. One stream (the updater's): traders do no provider/WS work; they read feeds.json / markets.json / snapshot.json. Now the keeper's settlement + liquidation exercise REAL positions (no longer no-ops). - runtime: mintTx/redeemTx (no-refresh, read the live feed), fundAddressDusdcTx + depositOwnedCoinTx (keeper funds traders, they deposit the received coin), export DUSDC_TYPE. - keeperService: fund TRADER_ADDRESSES at setup; publish markets.json each tick. - traderService: the fuzz loop (feasible-by-construction leverage; tracks open orders). - live.py/cli.py: harness up --traders N (per-trader process group + gas refill). Validated: harness up --traders 2 (1m cadence, 215s) -- ~63 mints + ~17 redeems across 2 traders, keeper settled+compacted markets holding real orders, no aborts, clean teardown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each actor appends a per-actor JSONL trace (INSTANCE_DIR/trace/{actor}.jsonl) of every op
with gas + outcome; `harness analyze [instance]` reads them and reports op counts,
gas-vs-moneyness for mints, the pool-NAV trend (drain heuristic from FlushExecuted
pool_value), and the BUG ORACLE -- any tx failure whose abort is NOT an expected guard
from our packages (arithmetic/framework errors), the original bug signal.
- trace.ts: appendTrace + gasOf + abortInfo (parses raw AND JSON-escaped MoveAbort).
- trader/keeper: trace mints (moneyness/gas), redeems, flushes (pool NAV), liquidations,
and failures.
- analyze.py + 'harness analyze': the report.
Validated: harness up --traders 2 (120s) -> 31 mints / 11 redeems / 2 flush / 5 liquidate,
gas-vs-moneyness bucketed, NAV stable (fuzz traders have no edge), bug oracle clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ord/replay harness up-many N runs ONE hub (a single Pyth+BS WS pair) writing a global snapshot, and N localnets whose updaters read it via HubSource instead of each opening its own WS. The hub records its stream (HUB_RECORD); harness up --replay <file> re-plays a recording (ReplaySource remaps recorded expiries onto the current grid + stamps fresh times) -- no live WS, for deterministic re-runs. - marketSource.ts: extracted DirectWsSource + the MarketSource interface from oracleService; added HubSource (reads the hub snapshot) + ReplaySource (recorded replay). - oracleService.ts: pick the source by env (HUB_SNAPSHOT / REPLAY_FILE / direct-ws). - hub.ts: the shared hub (one WS pair -> global snapshot + optional JSONL record). - live.py/cli.py: harness up-many N (ExitStack teardown) + up --replay <file>. Validated: up-many 2 -- one hub (190 snapshots), both updaters source=hub, both localnets' keepers settled + both traders traded; record (84 snapshots) + replay (source=replay, 44 pushes, keeper settled) clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The resolver emits max_probability (quoted p + 10%) and max_cost (spend + 20%); addMint threads them into mint_exact_quantity (defaulting to U64_MAX when omitted, so the bootstrap + spike stay uncapped). Fuzz traders mint with these caps, so a mint rejects if the on-chain quoted probability / all-in cost drifts past the slack -- the off-chain<->on-chain slippage guard. Also: analyze.py classifies validator-equivocation / consensus rejections as transient (submission races), not contract bugs, so the bug oracle isn't false-flagged. Validated: harness up --traders 2 (120s) -- 36 guarded mints succeeded, NAV stable, bug oracle clean (2 expected pricing guards, 1 transient). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The keeper settled every market that expired in a tick with the single latest snapshot spot inserted at the expiry key -- up to a tick late, one price reused for all -- which mis-resolves binary payoffs. (The contract correctly pins the settlement TIMESTAMP via read_at(expiry); only the harness's inserted VALUE was a stand-in, which the chain trusts.) Now the updater captures recent whole-second Pyth prints (the fixed_rate@200ms channel lands one at every minute boundary), flows them through snapshot.recent, and the keeper settles each market with ITS OWN at-expiry print. Latest-spot fallback fires only on an updater gap, loudly logged. - marketSource.ts: MarketSnapshot.recent; DirectWsSource records whole-second prints; snapshotFrom parses them (so HubSource carries them in parallel runs). - oracleService.ts / hub.ts: write recent into the snapshot. - keeperService.ts: settlementPrice(expiry) = exact boundary print (fallback flagged). Validated: harness up --traders 2 (150s) -- 3 markets settled, each with its own minute-boundary price (60113 / 60101 / 60151), zero FALLBACK warnings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per direction: (1) stream the full real_time Pyth channel (not fixed_rate@200ms) so each on-chain push carries the freshest live spot, replicating production; (2) settle each market independently via the Pyth Lazer history endpoint (POST /v1/price at the exact expiry timestamp), the same way deepbook-services' predict-market-settlement keeper does -- not by capturing stream prints. Supersedes the snapshot.recent boundary-capture (f8a8c31): with a real_time stream the prints aren't boundary-aligned, so settlement must query the history endpoint for the exact-expiry price. The keeper fetches each expiry's spot (fetchExactSpot1e9, mirrors deepbook-services' fetchExactLazerPayload), re-signs it with the harness's local oracle key, inserts it at the expiry key; a fetch failure defers the flush a tick rather than settling with a non-exact price. - marketSource.ts: channel real_time; fetchExactSpot1e9 (history POST, exact-ts assertion); drop the recent-buffer machinery. - keeperService.ts: settle via fetchExactSpot1e9 (drop snapshot.recent read). - oracleService.ts / hub.ts: drop recent from the snapshot (now serves only the traders). Validated: smoke test (exact spot at a past boundary) + harness up --traders 1 (150s) -- real_time stream (125 pushes, 0 skips), 2 markets settled via the history endpoint (0 deferrals), bug oracle clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- io.ts: harnessKey() resolves ../.env via import.meta.url (no hardcoded /Users path) + atomicWriteFile() (temp+rename). Kept out of env.ts so the hub (which has no per-instance localnet env) can import it without triggering the deployment-id load. - marketSource/predictSetup use the portable harnessKey (was a hardcoded absolute path that broke on any other machine/CI). - snapshot.json / hub-snapshot.json / markets.json / feeds.json now written atomically, and the updater's waitForFeeds parse is guarded -- a torn read can no longer kill the run. - Remove dead fetchPythSpot (settlement uses the Pyth history endpoint). Audit fixes #4, #6, #7. tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ion + restart-safe - #1 Reconcile the flush/settlement set from chain each tick (devInspect plp::active_expiry_markets + an id->expiry cache) instead of an in-memory list. A lost create response or a keeper restart can no longer orphan an on-chain market and brick finish_flush's all-active-valued assertion (was a permanent pool freeze). - #2 Each tick step (flush / liquidate / roll / markets.json) is isolated in its own try/catch, and liquidate re-filters live against a FRESH clock so a market expiring mid-tick can't pricing:9-abort the step. A transient sub-step failure defers that step a tick instead of skipping the rest. - #3 Restart-safety: setupFeedsAndConfig re-attaches to an existing feeds.json (no new feed objects overwriting it under the streaming updater); bootstrapPool skips when plp_total_supply>0; live.py supervises the core (restarts a dead keeper/updater up to 5x, which re-attaches + reconciles) instead of tearing down on first death. - runtime.ts: devInspect readers readActiveMarketIds / readPlpTotalSupply / readMarketExpiry. Audit fixes #1, #2, #3. Validated: harness up --traders 1 (130s) -- reconcile-driven roll (4 markets) + settle (no brick), NAV stable, bug oracle clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ion paths) A fraction (20%) of trades now send a deliberately-rejectable mint -- over-cap leverage, max_cost below net_premium, or max_probability below the quote -- so the admission + P6 slippage guards are actually exercised (they never fired under feasible-only fuzzing). A guard abort is the expected outcome (traced as a fail with the abort code = an expected guard); a wrongly-accepted probe is traced as adversarial-accepted (a guard gap). analyze reports rejection-path coverage and flags any gap. Audit fix #5. tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The stress test flagged 1 'wrongly accepted' over-cap probe -- a false positive: the probe set leverage to a feasible base (<= cap) x 1.5-2.5, which a low base x low multiplier can keep under the cap (so it was correctly accepted). Exceed maxAdmissionLeverage (the cap's ceiling) so the probe is always genuinely over-cap. Re-validated: 0 wrongly accepted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Python block-buffers stdout on a pipe, so [supervise]/[gas]/lifecycle prints were reordered vs subprocess output in captured (background/autonomous) run logs (noticed while reading the keeper-restart test log). Flush every print; harmless on a terminal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DESIGN.md described a pre-hardening harness (in-memory keeper, per-tick self-refresh, implicit fixed_rate channel, an unbuilt 'Remaining' list) and is superseded. Replace it with two purpose-built context files: - README.md (human entry point): current command set + how-it-works (one stream, chain- reconciled keeper, Pyth-history-endpoint settlement, hub/replay scaling, the bug oracle). - .claude/rules/predict-harness.md (Claude read-before-editing rule): build/verify; the architecture invariants (one-stream; reconcile-from-chain-or-brick; history settlement; real_time + Clock-1 clamp); units/clock gotchas (DUSDC 1e6 vs 1e9, no clock warp); the resilience + secrets invariants; the abort-only bug-oracle caveat. - CLAUDE.md: route packages/predict/harness/** to the new rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deep audit found the restart-safety fix's idempotency key (plp_total_supply>0) trips after lock_capital -- genesis step 2 of 4, which mints the min-liquidity lock -- so a crash between lock and the $10M supply+flush made a restart silently skip the rest and run an under-capitalized pool. Now skip the whole genesis only when supply >= BOOTSTRAP_SUPPLY (the $10M has landed), and make each step idempotent so a mid-genesis crash RESUMES without double-creating the account or double-queueing the supply (objectExists / supply==0 / supply_requests_pending guards). runtime.ts: readSupplyRequestsPending + objectExists. Validated: up --traders 2 with a mid-run keeper crash -> re-attach takes the supply>=bootstrap skip (not the supply>0 bug); bootstrap + settlement clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ce aggregation - M2: classify aborts by module CLASS, not module-prefix-only. INVARIANT modules (math, i64, plp, pool_accounting, strike_payout_tree, lp_book, expiry_cash, liquidation_book, range_codec, account*) ALWAYS flag -- previously their arithmetic/accounting/index invariants (math:3, i64:0, lp_book EInvalidDrainMark, ...) were swallowed as expected guards. GUARD modules stay expected business preconditions. - M1: actor-aware (_actor per file) + keeper liveness (failing-with-no-flush = stuck); add HTTP/rate-limit/timestamp shapes to the transient set so a Pyth-history 429 isn't false-flagged as a contract bug. - M4: aggregate across ALL instance dirs (up-many is multi-instance), not just latest-by-mtime. - M3 (exit code): analyze() returns non-zero on any flagged abort / wrongly-accepted probe / missing keeper trace, so background runs have a programmatic signal. Validated: synthetic (math:3/lp_book flag, guards expected, http->transient, 2-instance aggregate EXIT=1, clean EXIT=0) + real run (18 guards expected, oracle clean, EXIT=0). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- keeper/trader top-level catch writes a crash trace before exit(1), so a setup crash leaves a trace -- a never-bootstrapped run no longer looks identical to a healthy one. - live.py hold()/up_many() return non-zero on a supervised give-up / core-proc death. - trace.ts warns once if an appendTrace write fails, so dropped fail/crash records aren't silent. Completes M3 (analyze() exit code landed with the oracle rewrite). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…alth A sustained Pyth-history outage could brick the pool permanently: the all-or-nothing flush defers on any settlement-fetch failure, while the roll kept minting (~1/min) and the active-set cap counts only LIVE markets -- so expired-unsettled markets grew unbounded past the single-PTB flush gas wall, after which finish_flush can never complete. - marketSource.ts: fetchExactSpot1e9 gets bounded retry/backoff (0.5/1/2s), so a transient 429/5xx/not-yet-available doesn't defer the flush on the first blip. - keeperService.ts: a settledOk flag + consecutive-defer counter; GATE the roll on settledOk so the active set can't grow while settlement is behind (bounding it under the gas wall); a hard error after 8 consecutive defers surfaces a real outage/beyond-retention miss instead of a silent stall. Validated: up --traders 2 -- rolls + settlements normal (gate doesn't block the happy path), bug oracle clean, exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d localnets The slot registry stored os.getpid() (the owner) but _publish_localnet overwrote slot.pid with the localnet CHILD pid, so liveness tracked the child: a dead harness (OOM/SIGKILL) left the localnet orphaned and the slot held FOREVER (cleanup --stale a no-op), exhausting the 32-slot pool; the inverse (child dies, harness lives) risked premature reclaim + port cross-talk. - run.py: record localnet_pid/localnet_pgid separately; keep slot.pid = the owner. - state.py: reserve + reap_stale reclaim dead-OWNER slots AND SIGKILL the orphaned localnet by its process-group id (start_new_session leader) so its ports free up. Validated: unit test (slot.pid == owner; reap reclaims a dead-owner slot, would killpg a live orphan); live run clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Strategy abstraction: a strategy is a code module (ts/strategies/) selected by the STRATEGY env; traderService is a thin run-to-completion runner. Built-ins: fuzz (default, preserves up/up-many), mint-only, mixed-churn, liq-churn. `harness campaign S1 S2 ...` runs each on its own localnet off one shared hub, then a per-strategy analyze report + aggregate verdict. New trader primitives: partial redeem (tracks the replacement order id) and LP supply-from-custody + withdraw (reads the on-chain PLP balance first). Code-review hardening: - bug-oracle exit code gates on keeper-stuck / fatal-crash / missing-trace, not just no-keeper-trace; module:code aborts matched before the HTTP-transient substrings. - keeper advertises only funded markets (rebalance-then-advertise + per-tick retry). - Ctrl-C during bring-up no longer orphans the localnet (BaseException). - pid/pgid reuse guards in the slot registry. - held order dropped on a redeem abort; consecutive (not lifetime) restart cap. - replay PnL warning; removed dead gridExpiry; move.md/predict-simulations.md doc-drift. Validated: tsc 0 errors, py_compile clean, backward-compat up (fuzz) + a 3-strategy campaign + a mixed-churn re-run all clean (bug oracle exit 0). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
Add .claude/rules/harness-strategy.md — a trigger-engaged guide for building a new harness strategy: intake the user's test goal (or ask), map it to the StrategyCtx, then scaffold + register + validate via campaign if supported, or state the gap + extend the harness (new runtime.ts builder + strategy.ts ctx method, never the contracts) + tee up a PR if not. Route it in CLAUDE.md's Manual-Trigger Rules so "I want to add a harness strategy" engages it (same mechanism as wrap-up/code-review). Points at all 4 built-ins (fuzz / mint-only / mixed-churn / liq-churn) as templates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
- Scope a campaign's analyze to ITS run's instance dirs (analyze(instances=...)) so a
stale retained trace can't fail — or falsely satisfy expect for — the current verdict;
bare `analyze` stays the explicit aggregate-all mode.
- Bug oracle: whitelist expected business-precondition codes in the mixed modules at
module:code granularity (liquidation_book:4 the 5000 leveraged-order cap; lp_book:0-3
request preconditions); lp_book:4 (EInvalidDrainMark) + liquidation_book:0-3 (index
invariants) stay flagged. Drop bare HTTP codes ("500"/"502"/"503"/"429") from the
transient set — a gas/exec failure containing those digits stays flagged; real HTTP
transients carry "http"/"pyth history".
- oracleService writes snapshot.json only after the on-chain refresh lands, so traders
never price off oracle data that didn't make it on-chain.
- mixedChurn: document the min_withdraw_request (1e6) magic number.
Addresses code-review findings (scoping / transient / snapshot from the prior round +
the lp_book/liquidation_book granularity audit). Validated: tsc 0, py_compile, +
classification/scoping proofs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
- up/campaign/up-many now trim the run-time scratch (validator DB `localnet/` + staged
closure `workspace/`) on teardown, keeping only the trace + last-state JSONs — an instance
drops from ~150M to ~40K, so long multi-campaign sessions don't accumulate GBs.
run/run-many retention is unchanged; the trim keeps the gitignored .env.localnet /
local_pyth.json in place.
- Docs: README retention line; predict-harness.md cleanup note + a guardrail ("never blind-rm
.localnets/instances while a run may be live — `harness status` first, then the slot-aware
`cleanup --instances`") + a precise Secrets section (.env via .env/*.env; .env.localnet +
local_pyth.json covered by .localnets/, since *.env does not match .env.localnet).
Validated: py_compile; a real `up` run trims to a 40K instance with the trace intact.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
…on rejects) Root cause: every actor's submit re-resolved its gas coin's version via a fresh RPC read at build time. The gas coin's version bumps on EVERY tx, so under RPC read-after-write lag (which worsens as a long run saturates the box) the build used a stale version -> "Transaction needs to be rebuilt because object version unavailable" rejects on create-market / liquidate / flush, stalling the keeper. Fix: signExecThreaded (runtime.ts) pins the gas payment to the exact ref the prior tx's effects returned — the chain is the source of truth for the version, not the RPC view. A MoveAbort still executes + advances the coin, so the pin is updated from its effects; a submission reject (no effects, e.g. gas depletion) drops the pin so the next tx re-resolves a fresh coin. Sequential per sender, so no equivocation. Routed all three rapid submit paths (keeper executeAndWait, updater, trader) through it. No retry, no object cache: the gas coin is the sole churning owned object (shared objects are consensus-versioned; the stale object 0x9006 was confirmed in every keeper tx type = the shared gas coin, not the lifecycle cap). Steady state does zero gas-coin RPC reads, so the version race cannot occur regardless of load. Validated: tsc 0; a 3-localnet campaign produced 0 version-conflict artifacts (vs 99 create-market rejects before the fix) — create-market now succeeds, markets roll, all strategies bug-oracle clean. Remaining failures are expected freshness/boundary guards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
…replace-wholesale
The harness ran a single cadence and warmed a hardcoded `period:6` oracle grid — for the default
1m that's 6 consecutive 1-minute expiries, beyond Block Scholes' 1m surface horizon. Under the
prod-faithful replace-wholesale BS subscription, a single unmodeled/expired batch entry poisons the
whole forwards/svi batch, so the grid silently drained (the hub stall we diagnosed). Prod is immune
only because its grid contains exactly the BS-valid expiries (confirmed: 48h of live testnet logs
show 0 batch rejects / 0 staleness, BS WS reconnecting in ~1s).
Replicate the testnet deployment (deployment.testnet.json @ predict-testnet-6-24): the keeper now
enables + rolls all three prod cadences {1m,5m,1h} (cadences 0/1/2, windowSize 3) and the oracle
GRID_SPEC warms each cadence's windowSize boundaries — a BS-valid grid, so replace-wholesale no
longer poisons. The on-chain config already matched (predictConfig.ts); only the keeper, the grid,
and the obsolete single-cadence plumbing changed.
- keeperService: per-cadence roll over {0,1,2}; classify live markets via cadenceOf(expiry) (exact
from the contract rank partition — 1h owns :00:00, 5m owns 5-min marks off-the-hour, 1m the rest);
catch ECadenceWindowExceeded (the on-chain window cap) as expected.
- marketSource: BS subscription = stable client_ids (forwards/svi) + resend the full future-only
batch each roll (replace-wholesale evicts rolled-off expiries) — prod's exact pattern.
- live.py: GRID_SPEC "60000:3,300000:3,3600000:3" built from CADENCES via meta.ts (single source);
dropped the cadence param + KEEPER_CADENCE from hold/up_many/campaign.
- meta.ts emits the enabled cadence set; dropped the per-strategy `cadence` field (strategy.ts + the
4 strategies). cli.py: removed --cadence. oracleService/hub: gridNow dedups overlapping boundaries
(periods share the top of the hour).
- Docs: fixed stale --cadence/cadence refs (README, predict-harness.md) + a predict-harness
invariant guarding against re-widening the grid past BS surface availability.
Validated: tsc 0; py_compile OK. `up --traders 2 --seconds 300` — grid holds 7-9 across 1m/5m/1h (no
poison decline), keeper rolls all three cadences, 1m settles, 0 flush deferrals, bug oracle clean
(exit 0), NAV stable. `campaign mint-only --timeout 120` — clean (exit 0) with the new
{strategies,cadences} meta shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
…NAV can value A `nav-stress` strategy + analyze instrument to find the maximum leverage-book size the on-chain NAV calc (the keeper flush, which values every active market in ONE PTB) can tolerate before it can't be computed in one tx. - navStress.ts: piles low-leverage (1.1x) orders into ONE persisting 1h market (low leverage -> never liquidated; far 1h -> never settles mid-run; sole minter) so ctx.held.length == that market's leveraged-order count, traced each tick. Re-locks (pruneSettled) if a slow pile outlasts the lock. - analyze.py: a NAV-stress section joins the keeper flush gas with the book size by ts -> the gas-vs- size curve, gas/order slope, extrapolated PTB-cap crossing, and a gas-gated empirical breakpoint (a deferral while gas is far below the cap is an ordinary settlement race, not the break). - keeperService.ts: the flush gas budget is bumped to the Sui max (5e10) so the NAV calc is measured against the protocol one-PTB ceiling, not the 1e9 harness default (also correct generally — the pool-valuation flush should get the full budget, not brick a large pool prematurely). - Docs: nav-stress in the README built-ins; the stale `cadence` field dropped from harness-strategy.md. Validated: tsc 0; py_compile OK. campaign nav-stress (SIM_GAS_BUDGET=5e10) — book grows monotonically, the flush gas curve forms, bug oracle clean. Preliminary (extrapolated) result: ~1.44M gas/order -> the flush hits the 50-SUI one-PTB cap at ~35k total leveraged orders, far above the per-market 5000 cap, so for a single market the order cap binds, not NAV gas. Full empirical run (to the 5000 cap) pending for the definitive number. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
…cument findings The nav-stress analyzer used the 50-SUI gas budget as the OOG ceiling, but the wall is the per-tx COMPUTATION cap (max_gas_computation_bucket = 5M units x RGP = 5e9 MIST on localnet/testnet), independent of both the budget and storage. The bug made a completed run report "no breakpoint" AND false-flag the flush OOGs as 22 bugs. - analyze.py: ceiling 5e10 -> the 5e9 computation cap; compare `compGas` (computation cost), not net `gasOf` (which folds in storage the cap ignores); RGP/network-independence noted. The flush's InsufficientGas at the OOG book is now reported as the nav-stress breakpoint + excluded from the oracle (was falsely flagged). - trace.ts: `computationOf` (computationCost only); keeperService flush trace records `compGas`. - Docs: NAV_STRESS_FINDINGS_2026-06-30.md (the benchmark answering the 2026-06-28 investigation's open NAV-capacity question — single-market flush OOGs at ~4,580 leveraged orders, BELOW the 5,000 cap, linear ~1,086 units/order, pool-flush brick reproduced; 5M-unit cap verified identical on testnet+mainnet; cap-setting recommendation); OPEN-ITEMS.md #cap-flush24 live-confirmation pointer; predict-harness.md computation-cap gotcha; reports/ chart + README. Validated: tsc 0; py_compile OK; re-analyzing the retained full-run trace now reports the empirical breakpoint (~4,577 orders = 98% of the 5e9 cap, OOG'd) with the bug oracle clean (0 flagged, was 22). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
…mitives #1/#2) Extensions that unblock the mint-batch (100-mint scaling / #cap-mintbatch) and lp-adversary (#lp-nav-zero-brick, #plp-unbounded-mark-flow) experiments: - runtime.mintBatchTx(mints[]): N mint_exact_quantity calls in ONE PTB (mint-only, prices against the updater feed like mintTx) -> ONE computationCost, the #cap-mintbatch measurement vehicle (vary N + the lev1/lev2 mix to isolate the ~45x batch amplification, mechanism still unproven). - runtime.readCurrentNav(market, feeds) + readIdleBalance(): devInspect NAV-mark reads (Pricer has copy,drop, so the borrow-only inspect is valid); export KeeperFeeds. - strategy.ts: ctx.submitMintBatch / currentNav / idleBalance wired (trace {n, gas, compGas}). - predict-harness.md: ctx surface synced to the new methods. tsc 0. The four strategies + analyzer + the scripted-oracle mode (#3) follow. Tracker: .claude/predict-design/HARNESS_EXPERIMENT_TRACKER.md (local/gitignored). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
…ti, mint-batch (+ analyzer)
Three capacity strategies + the mint-batch analyzer section:
- nav-stress-atm: worst-case moneyness variant (near-ATM prob ~0.5 -> the expensive normal_cdf/
exp_series d2 branch, ~3,644 u/order); reuses the nav-stress flush-vs-book breakpoint. Confirms the
binding per-market cap (~1,372, #cap-flush24) vs nav-stress's cheap-branch ~4,580.
- nav-stress-multi: spreads the leverage book across ALL live markets, so the flush (one PTB, every
active market) is measured against the POOL-TOTAL leveraged count (#cap-flush24).
- mint-batch: the #cap-mintbatch root-cause probe. A fixed script of controlled batches settles the
~45x batch amplification (mechanism unproven) by DIFFERENTIAL on real computationCost: a sweep N in
{1..100} identical lev2 mints (cost(N) linear vs super-linear; N=1 == standalone) + a K=20 lev1-prefix
discriminator (lev1 mints don't write the liq book, so lev2-in-prefix vs standalone lev2 separates
dirtied-writes from tx-metering). An OOG at the ~5e9 cap is traced as the atomic-batch ceiling. Run
with SIM_GAS_BUDGET=50000000000 so batches can reach the cap.
- strategy.ts: submitMintBatch takes an optional `meta` -> one rich {type:"mintBatch"} trace record.
- analyze.py: mint-batch section (fit cost(N), batched/standalone ratio, OOG ceiling, discriminator
verdict). The two nav-stress variants reuse the existing flush-vs-book section.
tsc 0; py_compile 0. Strategies BUILT, not yet run (Phase 3 = the parallel campaign).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
…etering, not liq-book dirtying
The mint-batch differential (harness) settles WHY a batched leveraged mint costs more than a standalone
one. Discriminator at a saturated book (localnet): a leveraged mint appended after 20 lev1 (1x) mints —
which NEVER write the liquidation book — is amplified 20.2x, MORE than a leveraged mint inside a
full-leveraged batch (14.0x). So the amplification is per-TRANSACTION metering / command-position
accumulation, NOT the liq-book-page dirtying that #cap-mintbatch / move.md asserted. The sweep is
super-linear (per-mint 1.9M@N=1 -> 32M@N=50); N=100 OOGs the 5e9 cap (the ~110-op ceiling). Identical
computation shape batched-vs-separate; the cost is Sui's per-tx metering growing with accumulated tx state.
- mintBatch.ts: redesigned as a clean differential — sweep {1..100} + a saturated-book discriminator
(S / [K lev1] / [K lev1 + 1 lvg] / [K lvg]); 1.1x leverage (inserts into the liq book, never
liquidated -> no churn). Run with SIM_GAS_BUDGET=50000000000 so batches reach the cap.
- analyze.py: hoisted COMP_CAP to module scope (fixes an UnboundLocalError when a mint-batch run has no
{type:"book"} records); rewrote the mint-batch section (sweep w/ book column + super-linearity flag,
OOG ceiling, the (AB-A)/S vs (BB/K)/S discriminator + verdict).
- move.md: CORRECTED the "Predict Gas & Capacity" mint-batch bullet (was: dirtied liq-book pages; now:
per-tx metering / position, evidence-backed). Detail in MINT_BATCH_FINDINGS_2026-07-01 (local).
Validated: tsc 0; py_compile 0; smoke campaign clean (bug oracle 0 flagged, verdict clean, 44 trader ops).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
…gas plumbing (M1/L4/L5) + nav-atm moneyness Background code review of PR #1094 confirmed 3 harness issues that made the pre-testnet safety net report false-clean on the exact paths this branch adds. Fixed: - M1: mintBatch.ts catch labeled ANY error oog:true -> a real module:code abort in a batched PTB was masked from the bug oracle. Now OOG (InsufficientGas) stays oog:true; any other abort emits type:"fail" so analyze.py's oracle classifies it. - L4: traderService GAS_BUDGET was hardcoded 2e9 and SIM_GAS_BUDGET only reached the keeper, so mint-batch OOG'd at 2e9, not the 5e9 computation cap. Now the trader reads SIM_GAS_BUDGET; the analyzer OOG line no longer asserts the wall == the cap (wall = min(budget, cap)). - L5: analyze.py nav-stress _navbreak excluded EVERY keeper fail near the cap from the oracle -> a genuine invariant abort at the wall was masked. Now only gas-exhaustion fails are excluded. Plus nav-stress-atm targeted tight ATM (d2~0 = the CHEAP normal_cdf Horner branch, caught by the smoke); retargeted to MODERATE moneyness (prob 0.65-0.85 -> the expensive exp_series branch) for the worst case. Review verdict: safe to merge, no blockers; the Move open-items diff is a genuine hardening pass with no demonstrated defect (residual items are cap-sizing/coverage/doc calls). Report: PR1094_REVIEW_2026-07-01 (local). Validated: tsc 0; py_compile 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
…B; ceiling ~110-150, replicated) The L4 trader-budget fix (e0561ed) let the mint-batch re-run reach the real 5e9 cap: a 100-mint PTB is ~3.4B (68% of the cap), NOT an OOG — the earlier "~110 ops" OOG was the harness's own 2e9 trader budget. The ceiling is ~110-150 leveraged mints/PTB (data-dependent on book). The tx-metering verdict replicated across 2 runs (20.2x/14x and 16.4x/15.2x). Detail in MINT_BATCH_FINDINGS_2026-07-01 (local). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
Contract-side review follow-ups (test-only + one comment; no behavior change; Move 347/347, build clean): - L1: add gc_mutated_tree_matches_rebuilt_survivor_tree — removes an interior range so its boundary nodes GC out via merge_subtrees, then verifies settled_payout_liability (7 hand-computed points) + net_payout_reserve_terms against independently-derived values AND a rebuilt survivor-only tree. Closes the #c24d8d payout-tree GC fix's regression-coverage gap (prior remove tests never walked the canonical evaluators over a GC-mutated tree). - L6: live_market_cap_tests synthesizes market IDs programmatically (address::from_u256) instead of a fixed 25-entry vector coupled to max_live_expiry_markets!()+1; drops the dead FIRST_ID_INDEX so a cap bump no longer breaks the tests with a confusing native out-of-bounds abort. - L7: drop the tautological !contains_active_order(&one_x) assert (a 1x order short-circuits false regardless of book state); the unchanged candidate-count-at-cap check is the real property. - I2: correct the lp_pool_value floor comment — saturating_sub avoids the arithmetic underflow, but a 0/dust pool NAV still bricks finish_flush via the dust-floor + PLP circuit-breaker guards (#lp-nav-zero-brick); the floor is not a full liveness guarantee. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
…ist (L3), cap-test coverage (L1), docs (I1/L4) Third full PR review returned SAFE TO MERGE (no blockers); fixed every actionable item. - M1 [Medium, harness]: the bug-oracle liveness proxy was blind both ways — a settlement outage that began AFTER the first flush exited clean (stuck required keeper_flushes==0), and a transient blip before the first expiry false-FAILed. keeperService now emits a machine-readable keeper-stall trace past the defer threshold; analyze.py flags stalls, excludes transient RPC/history fails from stuck, and gates the never-flushed brick on elapsed > 2x the shortest cadence. - L3 [harness]: whitelist strike_payout_tree:1 (EMaxPayoutTreeNodes) as an expected admission cap (same class as liquidation_book:4) so hitting the 1000-node cap is not a spurious invariant-bug flag. - L1 [tests]: added node_count_tracks_real_boundary_accumulation (64 REAL inserts, hand-computed) to prove the node counter tracks genuine accumulation. KEPT debug_set_node_count with a justification: ~1000 real inserts TIME OUT the Move test framework (measured), so the exact cap-boundary tests must seed the count — the seam is genuinely irreducible (Rule 18), not avoidable as the review assumed. - I1 [info]: documented the (0, pos_inf] precondition (rejected upstream) that the cap pre-count relies on. - L4 [harness]: fixed the stale "used by lp-adversary" comment on the NAV-read primitives; kept them as documented Phase-2b (E5) scaffolding (the review's alternative is to drop — deferred to that phase). - L2 (cap-vs-wall joint budget) stays a documented follow-up (CAPACITY_AUDIT + tracker + PR); the audit tracker's #e9c8d2 was reopened for it. Validated: tsc 0; py_compile 0; Move 348/348, build --warnings-are-errors clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd
Finalization status — ready to mergeThree independent full multi-agent code reviews over this branch: all SAFE TO MERGE, no blockers — zero Critical/High, no demonstrated contract defect, and the new flush/liquidation caps are a strict improvement over
Tests: Move Tracked follow-ups (not blockers): the caps' joint gas budget vs the ~5M single-PTB flush wall ( |
There was a problem hiding this comment.
LGTM. Nit: New predict-simulations.md "Multi-command PTBs amplify liquidation-book scans" contradicts the new move.md bullet (and the PR's own #cap-mintbatch finding), which say the batched-mint amplification is per-tx metering, NOT liq-book dirtying
Summary
walk_linear/settlement no longer visit all-time-distinct ticks (#c24d8d), scopes cadence rank-overlap global → per-underlying, adds the valuation-lock guard to the 4 mark-affecting config setters, and centralizes the accumulator + cash-backing helpers (+ enforces the PLP live-expiry cap). A background multi-agent review (below) confirmed this is a clean hardening pass with no demonstrated defect — the caps are a strict improvement overmain, which had none.packages/predict/harness/): a worktree-free, real-data Sui-localnet staging sim — publishes the Predict stack into a fresh localnet, streams live Pyth Pro + Block Scholes onto the on-chain oracles, and runs the full market lifecycle (a lifecycle keeper + strategy traders) with analyzers and a code-aware bug oracle. Production-faithful oracle path (real-time spot + Pyth Lazer history-endpoint settlement, re-signed locally); parallel scaling + record/replay.deployment.testnet.json @ predict-testnet-6-24): the keeper enables + rolls all three prod cadences (1m/5m/1h,windowSize3); the BS subscription uses prod's stable-client_id replace-wholesale pattern (the earlier add-only60000:6grid tripped BS's "one bad entry poisons the whole batch" and drained).campaign+ Phase-2a capacity strategies: strategies are self-contained modules (ts/strategies/);campaign S1 S2 …runs each on its own localnet off one shared hub, then auto-analyzes. Built-ins:fuzz,mint-only,mixed-churn,liq-churn,nav-stress, andnav-stress-atm(worst-case moneyness),nav-stress-multi(pool-total),mint-batch(the#cap-mintbatchprobe) — plus harness primitives (a batched-mint PTB builder, NAV-mark devInspect reads)..claude/predict-design/*_FINDINGS_*.md):nav-stress→#cap-flush24confirmed: a single market's NAV flush OOGs at ~4,580 leveraged orders (cheap branch) — below the 5,000 cap — then the whole-pool flush bricks. Linear ~1,086 units/order; the wall is the per-tx computation cap (5M units × RGP), network-independent.mint-batch→#cap-mintbatchresolved: a batched leveraged mint costs ~16–20× a standalone one, and the discriminator (a leveraged mint appended after 20 liq-book-untouching 1x mints is amplified just as much) proves the cause is per-transaction metering / command-position accumulation, NOT liq-book-page dirtying (which the prior audit had asserted;move.mdcorrected). Ceiling ~110–150 leveraged mints/PTB; a 100-mint PTB is ~3.4B (68% of the 5e9 cap). Replicated across 2 runs.README.md+.claude/rules/predict-harness.md(routed inCLAUDE.md); design/audit findings + the experiment tracker live under.claude/predict-design/+.claude/predict-review/(local).Key decisions
Clockis real wall-clock (not warpable) → real-time sim; throughput scales by parallel localnets. Settlement is decoupled from the live stream (Pyth history endpoint at the exact expiry ts; the contract pins it viaread_at(expiry)). The keeper reconciles the active-market set from chain each tick.nav-stressmeasures the per-tx COMPUTATION cap (max_gas_computation_bucket = 5M units × RGP), not the 50k-SUI budget or storage — so the OOG order-count is network-independent (only the SUI cost differs: testnet 5, mainnet 0.5).#cap-flush24/ review L2): a single market can brick the flush below its own 5,000 cap (the flush values every active market in one PTB). Sizing a joint budget across the leveraged/payout/market caps is the follow-up cap-setting decision; the caps here still strictly improve onmain.dusdc..claude/predict-review/PR1094_REVIEW_2026-07-01.md.Test plan
sui move build --path packages/predict --warnings-are-errors+sui move test --path packages/predict --gas-limit 100000000000cd packages/predict/harness/ts && npx tsc --noEmit(0 errors) +python3 -m py_compile harness/*.pypython3 -m harness up --traders 2 --seconds 300thenpython3 -m harness analyze— grid holds ~7–9 across 1m/5m/1h, keeper rolls all cadences, lifecycle settles, bug oracle cleanSIM_GAS_BUDGET=50000000000 python3 -m harness campaign nav-stress mint-batch— nav-stress OOGs ~4,580; mint-batch prints the super-linear sweep + the tx-metering discriminator; bug oracle cleanpython3 -m harness up-many 2(parallel) + a keeper-crash restart🤖 Generated with Claude Code
https://claude.ai/code/session_0152ryjDZu25BfDv1raij4Zd