Skip to content

Commit 0197aa5

Browse files
github-actions[bot]github-actions[bot]
committed
Update weekly LLM benchmark
1 parent 533e5e6 commit 0197aa5

18 files changed

Lines changed: 4341 additions & 28 deletions

File tree

‎README.md‎

Lines changed: 29 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -43,52 +43,53 @@ This benchmark may be a good proxy to measure how well LLMs are able to coordina
4343

4444
The most recent prompt with the clear guidelines can be see [here](./cmd/create/prompt.txt) and [here](./cmd/list/prompt.txt).
4545

46-
## Current Portfolio (2026-06-01)
46+
## Current Portfolio (2026-06-08)
4747

4848
### Portfolio Value by Model
4949

5050
```mermaid
5151
pie showData
52-
"deepseek" : 189757
53-
"chatgpt" : 130510
54-
"gpt-oss" : 101044
52+
"deepseek" : 191732
53+
"chatgpt" : 128537
5554
"mistral" : 100000
5655
"qwen" : 100000
57-
"gemini" : 82365
58-
"llama" : 48975
56+
"gpt-oss" : 99509
57+
"gemini" : 76853
58+
"llama" : 48920
5959
```
6060

6161
| Model | Ticket | Sum | Quantity |
6262
|-------|-------|-------|--------|
6363
|`chatgpt`|`USD`|69|69|
64-
|`chatgpt`|`AAPL`|130441|418|
65-
|`deepseek`|`AMD`|1548|3|
66-
|`deepseek`|`ASML`|161276|100|
67-
|`deepseek`|`MSFT`|3152|7|
68-
|`deepseek`|`SNPS`|23781|50|
64+
|`chatgpt`|`AAPL`|128468|418|
65+
|`deepseek`|`AMD`|1399|3|
66+
|`deepseek`|`ASML`|164174|100|
67+
|`deepseek`|`MSFT`|2917|7|
68+
|`deepseek`|`SNPS`|23242|50|
6969
|`gemini`|`USD`|6335|6335|
70-
|`gemini`|`AAPL`|13107|42|
71-
|`gemini`|`AMD`|9806|19|
72-
|`gemini`|`ASML`|6451|4|
73-
|`gemini`|`CRDO`|4721|20|
74-
|`gemini`|`GFS`|4798|60|
75-
|`gemini`|`NXPI`|9640|30|
76-
|`gemini`|`ON`|5066|42|
77-
|`gemini`|`QCOM`|11547|46|
78-
|`gemini`|`TSLA`|10895|25|
70+
|`gemini`|`AAPL`|12908|42|
71+
|`gemini`|`AMD`|8861|19|
72+
|`gemini`|`ASML`|6567|4|
73+
|`gemini`|`CRDO`|4138|20|
74+
|`gemini`|`GFS`|4532|60|
75+
|`gemini`|`NXPI`|8879|30|
76+
|`gemini`|`ON`|4925|42|
77+
|`gemini`|`QCOM`|9933|46|
78+
|`gemini`|`TSLA`|9775|25|
7979
|`mistral`|`USD`|100000|100000|
80-
|`llama`|`USD`|48975|48975|
81-
|`gpt-oss`|`AAPL`|100795|323|
82-
|`gpt-oss`|`CMCSA`|249|10|
80+
|`llama`|`USD`|975|975|
81+
|`llama`|`AAPL`|47945|156|
82+
|`gpt-oss`|`AAPL`|99271|323|
83+
|`gpt-oss`|`CMCSA`|238|10|
8384
|`qwen`|`USD`|100000|100000|
8485

8586

8687
| Model | Total Sum | Change |
8788
|-------|-----------|--------|
88-
|`deepseek`|189757|—|
89-
|`chatgpt`|130510|—|
90-
|`gpt-oss`|101044|—|
89+
|`deepseek`|191732|—|
90+
|`chatgpt`|128537|—|
9191
|`mistral`|100000|—|
9292
|`qwen`|100000|—|
93-
|`gemini`|82365|—|
94-
|`llama`|48975|—|
93+
|`gpt-oss`|99509|—|
94+
|`gemini`|76853|—|
95+
|`llama`|48920|—|

‎llm100kbench.db‎

0 Bytes
Binary file not shown.

‎logs/chatgpt/2026-06-08.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Model Decision Log
2+
3+
- Alias: `chatgpt`
4+
- Provider: `github`
5+
- Model: `openai/gpt-4.1`
6+
- Status: `ok`
7+
- Generated at: `2026-06-08T12:42:29Z`
8+
9+
## Updates
10+
11+
No trades executed.
12+
13+
## Context
14+
15+
- Holding current positions as available cash is insufficient to purchase even one share of any listed asset at current prices.

‎logs/deepseek/2026-06-08.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Model Decision Log
2+
3+
- Alias: `deepseek`
4+
- Provider: `github`
5+
- Model: `deepseek/deepseek-r1-0528`
6+
- Status: `failed`
7+
- Generated at: `2026-06-08T12:44:33Z`
8+
9+
## Updates
10+
11+
No trades executed.
12+
13+
## Context
14+
15+
- No trades executed because the model did not return a valid order.
16+
17+
## Validation Notes
18+
19+
- first response rejected: invalid order: invalid character 'B' after top-level value
20+
- retry response rejected: llm request failed: 429 Too Many Requests: Too many requests. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service).
21+
- model failed validation after retry; no trades executed

‎logs/gemini/2026-06-08.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Model Decision Log
2+
3+
- Alias: `gemini`
4+
- Provider: `gemini`
5+
- Model: `gemini-2.5-flash-lite`
6+
- Status: `failed`
7+
- Generated at: `2026-06-08T12:44:36Z`
8+
9+
## Updates
10+
11+
No trades executed.
12+
13+
## Context
14+
15+
- No trades executed because the model did not return a valid order.
16+
17+
## Validation Notes
18+
19+
- first response rejected: invalid order: cannot sell 10000 USD with 6335 held
20+
- retry response rejected: invalid order: insufficient USD to buy GOOGL; need 4422.36 have 1412.60
21+
- model failed validation after retry; no trades executed

‎logs/gpt-oss/2026-06-08.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Model Decision Log
2+
3+
- Alias: `gpt-oss`
4+
- Provider: `groq`
5+
- Model: `openai/gpt-oss-120b`
6+
- Status: `ok`
7+
- Generated at: `2026-06-08T12:44:48Z`
8+
9+
## Updates
10+
11+
No trades executed.
12+
13+
## Context
14+
15+
- Holding current allocation; no trade improves portfolio value given constraints.

‎logs/llama/2026-06-08.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Model Decision Log
2+
3+
- Alias: `llama`
4+
- Provider: `groq`
5+
- Model: `meta-llama/llama-4-scout-17b-16e-instruct`
6+
- Status: `ok`
7+
- Generated at: `2026-06-08T12:44:47Z`
8+
9+
## Updates
10+
11+
| Action | Ticket | Quantity | Price | Reason |
12+
|--------|--------|----------|-------|--------|
13+
|`SELL`|`USD`|48000|1.0000|Free cash for the target allocation.|
14+
|`BUY`|`AAPL`|156|307.3400|Investing in AAPL for long-term growth.|
15+
16+
## Context
17+
18+
- Monitoring market for further opportunities
19+
20+
## Validation Notes
21+
22+
- first response rejected: invalid order: insufficient USD to buy AAPL; need 48867.06 have 1000.00

‎logs/mistral/2026-06-08.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Model Decision Log
2+
3+
- Alias: `mistral`
4+
- Provider: `mistral`
5+
- Model: `mistral-small-2603`
6+
- Status: `ok`
7+
- Generated at: `2026-06-08T12:44:36Z`
8+
9+
## Updates
10+
11+
No trades executed.
12+
13+
## Context
14+
15+
- No trades justified with current holdings and market conditions

‎logs/qwen/2026-06-08.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Model Decision Log
2+
3+
- Alias: `qwen`
4+
- Provider: `groq`
5+
- Model: `qwen/qwen3-32b`
6+
- Status: `failed`
7+
- Generated at: `2026-06-08T12:44:59Z`
8+
9+
## Updates
10+
11+
No trades executed.
12+
13+
## Context
14+
15+
- No trades executed because the model did not return a valid order.
16+
17+
## Validation Notes
18+
19+
- first response rejected: invalid order: insufficient USD to buy AAPL; need 99885.50 have 0.00
20+
- retry response rejected: invalid order: insufficient USD to buy AAPL; need 99885.50 have 0.00
21+
- model failed validation after retry; no trades executed

‎orders/chatgpt/2026-06-08.json‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"updates": [],
3+
"context": [
4+
"Holding current positions as available cash is insufficient to purchase even one share of any listed asset at current prices."
5+
],
6+
"metadata": {
7+
"alias": "chatgpt",
8+
"provider": "github",
9+
"model": "openai/gpt-4.1",
10+
"status": "ok",
11+
"generated_at": "2026-06-08T12:42:29Z"
12+
}
13+
}

0 commit comments

Comments
 (0)