Skip to content

Substitute {placeholder} patterns in system message from env vars - #3

Open
MEllis-github wants to merge 3 commits into
skydiscover-ai:mainfrom
MEllis-github:fix/system-message-placeholder-substitution
Open

Substitute {placeholder} patterns in system message from env vars#3
MEllis-github wants to merge 3 commits into
skydiscover-ai:mainfrom
MEllis-github:fix/system-message-placeholder-substitution

Conversation

@MEllis-github

@MEllis-github MEllis-github commented Mar 5, 2026

Copy link
Copy Markdown
Contributor

Problem

Bundled benchmark configs (e.g. benchmarks/frontier-cs-eval/config.yaml) use
{problem_statement} and {problem_constraints} placeholders in
prompt.system_message, but DefaultContextBuilder._get_system_message() returns the
raw string without substitution. The literal text {problem_statement} gets sent
to the LLM.

This is distinct from the known template variables ({current_program}, {metrics},
etc.) which are substituted in the user message via .format() — the system message
has no equivalent mechanism.

Fix

  • Add _substitute_placeholders() which resolves {name} patterns from the
    corresponding NAME environment variable (uppercase). Unresolved placeholders are
    left as-is.
  • Warn once per instance if unsubstituted placeholders remain after resolution.
  • Apply substitution in _get_system_message() after template lookup.

Includes tests for substitution, warning, and fallback behavior.

Alternative considered

Instead of env-var-based substitution, placeholders could be resolved via explicit
kwargs passed through build_prompt() (similar to how user message template vars
work). This would be more explicit and avoid potential collisions with shell
environment variables (e.g. {language} matching $LANGUAGE), but would require
changes to the Runner and config schema to thread values through. The env var
approach keeps the change contained to the context builder and matches how wrapper
scripts already inject problem-specific content.

MEllis-github and others added 3 commits March 4, 2026 21:02
The system message in configs can contain {placeholder} patterns
(e.g. {problem_statement}) but these were never substituted,
causing literal placeholder text to be sent to the LLM.

- Add _substitute_placeholders() resolving {name} from env var NAME
- Warn on first prompt build if unsubstituted placeholders remain
- Apply substitution in _get_system_message() after template lookup

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Format _KNOWN_TEMPLATE_VARS with black
- Remove unused pytest import
- Change _system_message_validated from class var to instance var
  so warnings fire per-instance rather than per-process

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@shubham3-ucb

Copy link
Copy Markdown
Collaborator

@MEllis-github One correctness issue: _substitute_placeholders (builder.py:68) only catches ValueError/IndexError, but a system message with a dotted or subscripted placeholder like {model.name} or {data[key]} raises AttributeError/TypeError from format_map, which propagates and crashes prompt construction. On main these were returned verbatim, so this is a regression. Broadening the except (or restricting substitution to bare \w+ names via the existing regex) fixes it.

Minor: _EnvFormatDict.__missing__ resolves any uppercased placeholder against os.environ, so arbitrary host env vars get interpolated silently. No default template triggers it, but an allowlist would be safer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants