Skip to content

Commit a0c7029

Browse files
ronshapiroclaude
ronshapiro
and
claude
committed
Pin --model per task type; fix instanceof bug it exposed
Extending Error without Object.setPrototypeOf breaks instanceof at this project's (unset -> ES3-default) tsconfig target, which was silently turning the rate-limit stop path into skip-and-retry-forever under real ts-node execution (jest's babel target masked it). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent 7bd5b5c commit a0c7029

5 files changed

Lines changed: 243 additions & 2 deletions

File tree

‎RecursiveSelfImprovingAgentPlan.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,12 @@ sessions per day per task type, running only during off-hours, priority ordering
4747
starve your other Claude Code work on a heavy day). This is coarser than a spend cap by design —
4848
accepted in exchange for staying on the existing subscription.
4949

50+
`claude -p` also has a `--max-budget-usd <amount>` flag — a real per-call dollar cap, independent
51+
of whether the underlying billing is subscription or API-key based (confirmed via `claude --help`,
52+
saved at `/tmp/claude_cli_help.txt`). Not wired into `headless_claude.ts` yet — a per-task-type
53+
budget cap via this flag is a natural next step, and belongs in the same change as the `--model`
54+
pinning below, since both are "known flag, not used yet" gaps in the same file.
55+
5056
## Data flow
5157

5258
- **Tier 0 — source of truth**: Sefaria's API, pulled offline into `cached_outputs/` (unchanged).
@@ -125,6 +131,15 @@ acceptance-rate and edit-distance stats and proposes an updated config — e.g.
125131
type that a cheaper model already handles well, or upgrading one where quality is the bottleneck.
126132
This is the concrete mechanism behind "learn which models to use for which tasks."
127133

134+
**Not wired in yet, found during Phase 2's first real run**: `headless_claude.ts` doesn't pass a
135+
`--model` flag, so `claude -p` picks the model on its own — the real Menachot 77a smoke test came
136+
back mostly `claude-haiku-4-5`, with one `claude-sonnet-5` call, and nothing in this codebase
137+
requested either. `runHeadlessClaude` records which model ran (`primaryModel`, picked by
138+
highest-cost entry in the CLI's `modelUsage`) but does not yet *select* it — routing "learning" is
139+
meaningless while the model is chosen opaquely underneath it. Pinning `--model` per task type,
140+
sourced from that task type's config file, is a prerequisite for this section to be true, not
141+
just an optimization.
142+
128143
## Task types
129144

130145
**From your list:**

‎precomputed/ai_additions/Menachot 90a.json‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
{
2+
"Rashi on Menachot 90a:10:1": {
3+
"hebrew": "<strong class=\"dibur-hamatchil\">רבי עקיבא סבר מדת יבש לא נמשחה כל עיקר</strong> - וחול היא, לפיכך מה שבפנים בכלי קדושת פה הוא דמקדש ליה, הילכך מה דאיצטריך ליה מקדש, בירוצין דלא צריך ליה לא מקדש, לפיכך בירוציהן חול:",
4+
"english": "<b>Rabbi Akiva holds that the measure for dry goods was never anointed at all</b> - and it is non-sacred; therefore, what is inside the vessel is consecrated by the sanctity of its designated fill line, and so whatever amount is needed is consecrated, while the overflow, which is not needed, is not consecrated. Therefore, their overflow is non-sacred."
5+
},
6+
"Rashi on Menachot 90a:12:1": {
7+
"hebrew": "<strong class=\"dibur-hamatchil\">למאי דצריך</strong> - למה שבתוך הכלי איכוין למיקדש, ולא לבירוצין, ואמאי מיקדשי הבירוצין:",
8+
"english": "\"That which he requires\" - He intends to consecrate only what is inside the vessel, and not the overflow; so why should the overflow become consecrated?"
9+
},
10+
"Rashi on Menachot 90a:13:1": {
11+
"english": "'Not knowingly' - even though he did not intend to consecrate the overflow, the sacred vessels (vessels of service) sanctify it regardless."
12+
},
13+
"Rashi on Menachot 90a:13:2": {
14+
"hebrew": "<strong class=\"dibur-hamatchil\">גזירה שמא יאמרו</strong> - דמאן דחזי דהנך בירוצין לא קדשי, אע\"ג דאתו מגוויה דמנא, אמרי הכי נמי מוציאין מכלי שרת לחול. אבל יבש, דלא אתי מגופיה דמנא, לא אתו למימר ביה מוציאין מכלי שרת לחול:",
15+
"english": "<strong class=\"dibur-hamatchil\">A decree lest they say</strong> - For one who sees that these overflows are not consecrated, even though they come from inside the vessel, will say the same of it: that [substances] may be taken from a service vessel to non-sacred use. But dry [flour], which does not come from the body of the vessel, they will not come to say of it that [substances] may be taken from a service vessel to non-sacred use."
16+
},
17+
"Rashi on Menachot 90a:1:1": {
18+
"english": "IS THERE A VOLUNTARY GUILT-OFFERING? — Hence, it may be sacrificed on the altar only together with its libations."
19+
},
20+
"Rashi on Menachot 90a:3:1": {
21+
"english": "<strong class=\"dibur-hamatchil\">Its heaping was within it</strong> - [The Kohen Gadol's measuring vessel] was so large that when it was leveled off, it already held a full tenth [of an ephah]."
22+
},
23+
"Rashi on Menachot 90a:4:1": {
24+
"english": "\"For the measure of dry goods, their overflow is non-sacred\" - What falls from the vessel after it has been filled, that is the \"overflow\" (birutzin)."
25+
},
26+
"Rashi on Menachot 90a:5:1": {
27+
"hebrew": "<strong class=\"dibur-hamatchil\">אלא שהלח נעקר</strong> - שהבירוצין נכנסו בכלי ונתקדשו בו, ושוב נעקרו ממנו כשנפלו חוץ, לפיכך הן קדש, דנכנסו בכלי ונתקדשו בו:",
28+
"english": "<strong class=\"dibur-hamatchil\">Rather, [it is] that the liquid was displaced</strong> - The overflow had entered the vessel and become consecrated there, and was then displaced from it when it spilled outside; therefore it is sacred, since it had entered the vessel and become consecrated in it."
29+
},
30+
"Rashi on Menachot 90a:5:2": {
31+
"hebrew": "<strong class=\"dibur-hamatchil\">ובירוצין של יבש אינו נעקר</strong> - מתוכו של כלי, שלא נכנס בתוך הכלי ולא נתקדש בו, לפיכך הוא חול:",
32+
"english": "<strong class=\"dibur-hamatchil\">AND THE OVERFLOW OF DRY [MEASURES] IS NOT DISPLACED</strong> - from within the vessel; that is, [the dry overflow] never entered into the vessel and was never consecrated in it, and therefore it is non-sacred."
33+
},
34+
"Rashi on Menachot 90a:6:1": {
35+
"english": "<strong class=\"dibur-hamatchil\">Whose [opinion is this]</strong> - This is referring to that which is taught [in the mishna]: All the measures were heaped."
36+
},
37+
"Rashi on Menachot 90a:6:2": {
38+
"english": "IF [the mishna follows] RABBI MEIR — Since he stated above (Menachot 87a) that the heaped vessel, with which one measured the flour for all the other meal-offerings, [was different from] the level vessel that was the measure of the High Priest, which was so much larger that the amount that filled the [smaller] vessel when heaped would fill this one only to the level line."
39+
},
40+
"Rashi on Menachot 90a:6:3": {
41+
"english": "<b>AND IT WAS LEVELED</b> - meaning, there was not even one heaped measure at all."
42+
},
43+
"Rashi on Menachot 90a:7:1": {
44+
"hebrew": "<strong class=\"dibur-hamatchil\">ומאי כל מדות</strong> - דלא תימא כל המדות שהיו שם היו גדושות, דלא היתה אלא אחת גדושה:",
45+
"english": "<strong>And what [does the mishna mean by] \"all measures\"</strong> - Do not say that all the measuring-vessels that were there were heaped, for there was only one that was heaped."
46+
},
47+
"Rashi on Menachot 90a:7:2": {
48+
"english": "<strong>Rather, this is what the mishna is teaching: All measurements [<i>kol medidot</i>]</strong> - the measurements that they took with it were heaped."
49+
},
50+
"Rashi on Menachot 90a:9:1": {
51+
"english": "<strong class=\"dibur-hamatchil\">They were not anointed on the outside</strong> - Therefore, their overflow is non-sacred."
52+
}
53+
}

‎precomputed/rsi_state/generation_records/rashi_tosafot_translation/Menachot 90a.json‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
{
2+
"Rashi on Menachot 90a:10:1": {
3+
"dependsOn": [],
4+
"generatedAt": "2026-08-10T15:57:52.464Z",
5+
"model": "claude-sonnet-5",
6+
"promptVersion": "v1",
7+
"sourceRefs": [
8+
"Rashi on Menachot 90a:10:1"
9+
],
10+
"sourceText": "<strong class=\"dibur-hamatchil\">רבי עקיבא סבר מדת יבש לא נמשחה כל עיקר</strong> - וחול היא לפיכך מה שבפנים בכלי קדושת פה הוא דמקדש ליה הילכך מה דאיצטריך ליה מקדש בירוצין דלא צריך ליה לא מקדש לפיכך בירוציהן חול:"
11+
},
12+
"Rashi on Menachot 90a:12:1": {
13+
"dependsOn": [],
14+
"generatedAt": "2026-08-10T16:00:10.017Z",
15+
"model": "claude-sonnet-5",
16+
"promptVersion": "v1",
17+
"sourceRefs": [
18+
"Rashi on Menachot 90a:12:1"
19+
],
20+
"sourceText": "<strong class=\"dibur-hamatchil\">למאי דצריך</strong> - למה שבתוך הכלי איכוין למיקדש ולא לבירוצין ואמאי מיקדשי הבירוצין:"
21+
},
22+
"Rashi on Menachot 90a:13:1": {
23+
"dependsOn": [],
24+
"generatedAt": "2026-08-10T16:01:52.949Z",
25+
"model": "claude-sonnet-5",
26+
"promptVersion": "v1",
27+
"sourceRefs": [
28+
"Rashi on Menachot 90a:13:1"
29+
],
30+
"sourceText": "<strong class=\"dibur-hamatchil\">שלא מדעת</strong> - שאף ע\"פ שלא נתכוין להקדיש הבירוצין קדושין כלי שרת:"
31+
},
32+
"Rashi on Menachot 90a:13:2": {
33+
"dependsOn": [],
34+
"generatedAt": "2026-08-10T16:03:45.604Z",
35+
"model": "claude-sonnet-5",
36+
"promptVersion": "v1",
37+
"sourceRefs": [
38+
"Rashi on Menachot 90a:13:2"
39+
],
40+
"sourceText": "<strong class=\"dibur-hamatchil\">גזירה שמא יאמרו</strong> - דמאן דחזי דהנך בירוצין לא קדשי אע\"ג דאתו מגוויה דמנא אמרי הכי נמי מוציאין מכלי שרת לחול אבל יבש דלא אתי מגופיה דמנא לא אתו למימר ביה מוציאין מכלי שרת לחול:"
41+
},
42+
"Rashi on Menachot 90a:1:1": {
43+
"dependsOn": [],
44+
"generatedAt": "2026-08-10T15:40:51.074Z",
45+
"model": "claude-sonnet-5",
46+
"promptVersion": "v1",
47+
"sourceRefs": [
48+
"Rashi on Menachot 90a:1:1"
49+
],
50+
"sourceText": "<strong class=\"dibur-hamatchil\">אשם נדבה מי איכא</strong> - הילכך לא הותר ליקרב למזבח אלא על ידי נסכים:"
51+
},
52+
"Rashi on Menachot 90a:3:1": {
53+
"dependsOn": [],
54+
"generatedAt": "2026-08-10T15:43:47.570Z",
55+
"model": "claude-sonnet-5",
56+
"promptVersion": "v1",
57+
"sourceRefs": [
58+
"Rashi on Menachot 90a:3:1"
59+
],
60+
"sourceText": "<strong class=\"dibur-hamatchil\">שהיה גודשה בתוכה</strong> - שהיתה גדולה כל כך שכשהיתה נמחקת היה בה עשרון:"
61+
},
62+
"Rashi on Menachot 90a:4:1": {
63+
"dependsOn": [],
64+
"generatedAt": "2026-08-10T15:44:35.721Z",
65+
"model": "claude-sonnet-5",
66+
"promptVersion": "v1",
67+
"sourceRefs": [
68+
"Rashi on Menachot 90a:4:1"
69+
],
70+
"sourceText": "<strong class=\"dibur-hamatchil\">מדת היבש בירוציהן חול</strong> - מה שנופל מן הכלי לאחר שנתמלא היינו בירוצין:"
71+
},
72+
"Rashi on Menachot 90a:5:1": {
73+
"dependsOn": [],
74+
"generatedAt": "2026-08-10T15:45:40.136Z",
75+
"model": "claude-sonnet-5",
76+
"promptVersion": "v1",
77+
"sourceRefs": [
78+
"Rashi on Menachot 90a:5:1"
79+
],
80+
"sourceText": "<strong class=\"dibur-hamatchil\">אלא שהלח נעקר</strong> - שהבירוצין נכנסו בכלי ונתקדשו בו ושוב נעקרו ממנו כשנפלו חוץ לפיכך הן קדש דנכנסו בכלי ונתקדשו בו:"
81+
},
82+
"Rashi on Menachot 90a:5:2": {
83+
"dependsOn": [],
84+
"generatedAt": "2026-08-10T15:46:29.177Z",
85+
"model": "claude-sonnet-5",
86+
"promptVersion": "v1",
87+
"sourceRefs": [
88+
"Rashi on Menachot 90a:5:2"
89+
],
90+
"sourceText": "<strong class=\"dibur-hamatchil\">ובירוצין של יבש אינו נעקר</strong> - מתוכו של כלי שלא נכנס בתוך הכלי ולא נתקדש בו לפיכך הוא חול:"
91+
},
92+
"Rashi on Menachot 90a:6:1": {
93+
"dependsOn": [],
94+
"generatedAt": "2026-08-10T15:47:16.785Z",
95+
"model": "claude-sonnet-5",
96+
"promptVersion": "v1",
97+
"sourceRefs": [
98+
"Rashi on Menachot 90a:6:1"
99+
],
100+
"sourceText": "<strong class=\"dibur-hamatchil\">מני</strong> - הא דקתני כל המדות היו נגדשות:"
101+
},
102+
"Rashi on Menachot 90a:6:2": {
103+
"dependsOn": [],
104+
"generatedAt": "2026-08-10T15:51:37.017Z",
105+
"model": "claude-sonnet-5",
106+
"promptVersion": "v1",
107+
"sourceRefs": [
108+
"Rashi on Menachot 90a:6:2"
109+
],
110+
"sourceText": "<strong class=\"dibur-hamatchil\">אי ר\"מ</strong> - דאמר לעיל <span class=\"parenthesized\">(מנחות ד' פז.)</span> גדוש שבו היה מודד לכל המנחות מחוק מדה של כהן גדול שהיתה גדולה כ\"כ שגודשה בתוכה:"
111+
},
112+
"Rashi on Menachot 90a:6:3": {
113+
"dependsOn": [],
114+
"generatedAt": "2026-08-10T15:52:49.798Z",
115+
"model": "claude-sonnet-5",
116+
"promptVersion": "v1",
117+
"sourceRefs": [
118+
"Rashi on Menachot 90a:6:3"
119+
],
120+
"sourceText": "<strong class=\"dibur-hamatchil\">ומחוק הוי</strong> - ואפילו חד גדוש לא הוה:"
121+
},
122+
"Rashi on Menachot 90a:7:1": {
123+
"dependsOn": [],
124+
"generatedAt": "2026-08-10T15:53:44.634Z",
125+
"model": "claude-sonnet-5",
126+
"promptVersion": "v1",
127+
"sourceRefs": [
128+
"Rashi on Menachot 90a:7:1"
129+
],
130+
"sourceText": "<strong class=\"dibur-hamatchil\">ומאי כל מדות</strong> - דלא תימא כל המדות שהיו שם היו גדושות דלא היתה אלא אחת גדושה:"
131+
},
132+
"Rashi on Menachot 90a:7:2": {
133+
"dependsOn": [],
134+
"generatedAt": "2026-08-10T15:54:31.298Z",
135+
"model": "claude-sonnet-5",
136+
"promptVersion": "v1",
137+
"sourceRefs": [
138+
"Rashi on Menachot 90a:7:2"
139+
],
140+
"sourceText": "<strong class=\"dibur-hamatchil\">אלא הכי קתני מתניתין כל מדידות</strong> - שהיו מודדין בה היו גדושות:"
141+
},
142+
"Rashi on Menachot 90a:9:1": {
143+
"dependsOn": [],
144+
"generatedAt": "2026-08-10T15:55:21.574Z",
145+
"model": "claude-sonnet-5",
146+
"promptVersion": "v1",
147+
"sourceRefs": [
148+
"Rashi on Menachot 90a:9:1"
149+
],
150+
"sourceText": "<strong class=\"dibur-hamatchil\">לא נמשחו בחוץ</strong> - לפיכך בירוציהן חול:"
151+
}
152+
}

‎rsi_orchestrator/headless_claude.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,19 @@ import {promisify} from "util";
33

44
const execFileAsync = promisify(execFile);
55

6+
// NOT YET IMPLEMENTED: `claude -p` also has a `--max-budget-usd <amount>` flag (see
7+
// `claude --help`), not passed here yet — see RecursiveSelfImprovingAgentPlan.md's "Budget"
8+
// section. `--model` (below) is now pinned per call; without it the CLI picked the model on its
9+
// own — confirmed on a real run, which came back mostly claude-haiku-4-5 with one claude-sonnet-5
10+
// call, none of it requested by this code.
611
export interface HeadlessClaudeOptions {
712
cwd?: string;
813
timeoutMs?: number;
14+
// Explicit model alias or full name (e.g. "sonnet", "claude-sonnet-5"). Without this, `claude
15+
// -p` chooses on its own — see the note above. Each task type should pass its own pinned
16+
// model; Phase 4's routing tuner is what eventually changes this value based on logged outcomes
17+
// (see RecursiveSelfImprovingAgentPlan.md's "Model routing" section), not this wrapper.
18+
model?: string;
919
// Defaults to a safe read-only set — this task family only needs to read repo files, not run
1020
// arbitrary commands or write anything. Without an explicit allow-list, headless calls have no
1121
// way to approve tool use, so anything beyond the default-allowed tools gets silently denied
@@ -40,6 +50,11 @@ export class HeadlessClaudeError extends Error {
4050
public readonly apiErrorStatus: number | undefined,
4151
) {
4252
super(message);
53+
// Without this, `instanceof HeadlessClaudeError` silently returns false at this project's
54+
// (unset, so ES3-default) tsconfig target — a well-known gotcha extending Error in TS. Found
55+
// the hard way: a real run kept "skipping" every candidate after a rate limit instead of
56+
// stopping, because the isRateLimited branch's instanceof check never matched.
57+
Object.setPrototypeOf(this, HeadlessClaudeError.prototype);
4358
this.name = "HeadlessClaudeError";
4459
this.isRateLimited = apiErrorStatus === 429;
4560
}
@@ -95,6 +110,7 @@ export const runHeadlessClaude: HeadlessClaudeRunner = async (prompt, options =
95110
"-p", prompt,
96111
"--output-format", "json",
97112
"--allowedTools", (options.allowedTools ?? DEFAULT_ALLOWED_TOOLS).join(","),
113+
...(options.model ? ["--model", options.model] : []),
98114
],
99115
{
100116
cwd: options.cwd,

‎rsi_orchestrator/rashi_tosafot_translation.ts‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,10 @@ import {HeadlessClaudeError, runHeadlessClaude} from "./headless_claude";
2020

2121
export const TASK_TYPE = "rashi_tosafot_translation";
2222
const PROMPT_VERSION = "v1";
23+
// Hardcoded for now — Phase 4's routing tuner is meant to replace this with a per-task-type value
24+
// chosen from logged outcomes, not this constant. See "Model routing" in
25+
// RecursiveSelfImprovingAgentPlan.md.
26+
const MODEL = "claude-sonnet-5";
2327
const COMMENTATORS = ["Rashi", "Tosafot"] as const;
2428
type Commentator = typeof COMMENTATORS[number];
2529

@@ -202,14 +206,15 @@ export function parseJsonResponse<T>(text: string): T {
202206
async function generateViaClaude(
203207
candidate: TranslationCandidate, priorFeedback?: string,
204208
): Promise<GeneratedEdit> {
205-
const result = await runHeadlessClaude(generationPrompt(candidate, priorFeedback));
209+
const result = await runHeadlessClaude(
210+
generationPrompt(candidate, priorFeedback), {model: MODEL});
206211
return {edit: parseJsonResponse<Edit>(result.text), model: result.model};
207212
}
208213

209214
async function critiqueViaClaude(
210215
candidate: TranslationCandidate, edit: Edit,
211216
): Promise<CritiqueVerdict> {
212-
const result = await runHeadlessClaude(critiquePrompt(candidate, edit));
217+
const result = await runHeadlessClaude(critiquePrompt(candidate, edit), {model: MODEL});
213218
return parseJsonResponse<CritiqueVerdict>(result.text);
214219
}
215220

0 commit comments

Comments
 (0)