File tree Expand file tree Collapse file tree
reproduction-results/gptmodel/results Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -48,3 +48,15 @@ For each contract: `*_fillme_L<line>.sol` (with the hole) and `*_patched_L<line>
4848| 202409_Bedrock_DeFi | 2403 | ` uniBTCAmount * 1e10 < msg.value ` | ` !paused[NATIVE_BTC] && msg.value > 0 && msg.value / EXCHANGE_RATE_BASE > 0 && ISupplyFeeder(supplyFeeder).totalSupply(NATIVE_BTC) + msg.value <= caps[NATIVE_BTC] ` | ❌ |
4949| 202409_OnyxDAO | 794 | ` repayAmount == borrowedAmount ` | ` borrowedAmount <= repayAmount ` | ❌ |
5050| 202603_AlkemiEarn | 3458 | ` msg.sender != targetAccount ` | ` — ` | ⚠️ empty due to limited tokens |
51+
52+ ## Comparison with FLAMES-CodeLlama
53+
54+ The table below compares this frontier-model run against the FLAMES-CodeLlama
55+ (FLAMES-100k) results.
56+
57+ | Metric | FLAMES-CodeLlama | GPT-4o (this run) |
58+ | ---| ---| ---|
59+ | Exact match (syntactic, FLAMES rules) | 2/28 — ** 7.1%** | 10/30 — ** 33.3%** |
60+ | Garbage output (context truncation) | 24/28 — ** 85.7%** | 0 |
61+ | Empty (token limit) | 0 | 1/31 |
62+
You can’t perform that action at this time.
0 commit comments