Skip to content

Commit d615a9c

Browse files
authored
Update README with comparison metrics
Added comparison metrics between frontier-model run and FLAMES-CodeLlama results.
1 parent 8e7771f commit d615a9c

1 file changed

Lines changed: 12 additions & 0 deletions

File tree

  • reproduction-results/gptmodel/results

reproduction-results/gptmodel/results/README.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,3 +48,15 @@ For each contract: `*_fillme_L<line>.sol` (with the hole) and `*_patched_L<line>
4848
| 202409_Bedrock_DeFi | 2403 | `uniBTCAmount * 1e10 < msg.value` | `!paused[NATIVE_BTC] && msg.value > 0 && msg.value / EXCHANGE_RATE_BASE > 0 && ISupplyFeeder(supplyFeeder).totalSupply(NATIVE_BTC) + msg.value <= caps[NATIVE_BTC]` ||
4949
| 202409_OnyxDAO | 794 | `repayAmount == borrowedAmount` | `borrowedAmount <= repayAmount` ||
5050
| 202603_AlkemiEarn | 3458 | `msg.sender != targetAccount` | `` | ⚠️ empty due to limited tokens |
51+
52+
## Comparison with FLAMES-CodeLlama
53+
54+
The table below compares this frontier-model run against the FLAMES-CodeLlama
55+
(FLAMES-100k) results.
56+
57+
| Metric | FLAMES-CodeLlama | GPT-4o (this run) |
58+
|---|---|---|
59+
| Exact match (syntactic, FLAMES rules) | 2/28 — **7.1%** | 10/30 — **33.3%** |
60+
| Garbage output (context truncation) | 24/28 — **85.7%** | 0 |
61+
| Empty (token limit) | 0 | 1/31 |
62+

0 commit comments

Comments
 (0)