[Nice to have] expandFields: to apply variable substitution to agent step fields beyond prompt

[tbrandenburg]

Problem

expandPrompt: true substitutes $VAR tokens from vars steps into the prompt text at runtime. The same substitution is not available for other agent step fields: model, agent, and command.

This means a workflow param for model selection (equivalent to --model in the shell scripts) cannot be threaded into the model: field — it must be hardcoded. Users who want to run the same workflow with different models must either edit the YAML or maintain separate workflow copies.

Proposed feature

Extend the existing expandPrompt mechanism to cover other string fields on agent steps, either via a broader flag or field-by-field:

Option A — single flag covering all expandable fields:

- type: agent
  expandFields: true   # expands model:, agent:, command:, prompt:
  model: $MODEL
  prompt: |
    ...

Option B — explicit per-field opt-in (mirrors expandPrompt pattern):

- type: agent
  expandPrompt: true
  expandModel: true
  model: $MODEL
  prompt: |
    ...

Option A is simpler and consistent with how expandPrompt already works.

Generated bash

Without the feature, model is hardcoded:

opencode run --model 'gpt-4o' "$_prompt"

With expandFields: true and MODEL set via param or vars:

opencode run --model "$MODEL" "$_prompt"

Why nice to have

The workaround — hardcode the model in the YAML — is intentional and explicit. It is only inconvenient when the same workflow needs to be run repeatedly with different models without editing the file each time.

[github-actions]

Investigation: [Nice to have] expandFields: to apply variable substitution to agent step fields beyond prompt

Issue: #34 (#34)
Type: ENHANCEMENT
Investigated: 2026-06-28T00:00:00Z

Assessment

Metric	Value	Reasoning
Priority	MEDIUM	Not blocking; workaround (hardcode model in YAML) is simple but inconvenient for repeated use
Complexity	LOW	2 files (+tests), reuses existing substitution machinery, no new integration points
Confidence	HIGH	Change is additive, well-understood code path, existing tests to mirror, clear proposal

---

Problem Statement

expandPrompt: true substitutes $VAR tokens from vars steps into the prompt text at runtime.
The same substitution is not available for other agent step string fields (model, agent, command).
Users who want to thread a workflow param (e.g., MODEL) into model: must either hardcode it or maintain separate YAML files.

---

Analysis

Change Rationale

Purely additive enhancement. The existing expandPrompt mechanism (Python regex scan + emitted Bash string-replacement lines) can be reused for model, agent, and command with minimal modification. Same safety properties apply (single-quoted heredoc capture, no shell evaluation, only uppercase [A-Z_][A-Z0-9_]* variable names).

Evidence Chain

WHY: Users cannot set model: $MODEL and have it substituted at runtime
↓ BECAUSE: model is rendered as local model='$MODEL' (literal via bash_quote), and no substitution lines are emitted for it
Evidence: src/flowsh_cli/render.py:387-389 - lines.append(f" local model={bash_quote(step.model or '')}")

↓ BECAUSE: The variable-substitution block (if step.expandPrompt:) only scans step.prompt and emits lines targeting the prompt variable
Evidence: src/flowsh_cli/render.py:377-386 - only prompt is scanned and substituted

↓ ROOT CAUSE: The substitution logic is field-specific (hardcoded to prompt) rather than configurable per-field
Evidence: src/flowsh_cli/render.py:377-386 - substitution only covers prompt

Affected Files

File	Lines	Action	Description
src/flowsh_cli/models.py	108-118	UPDATE	Add expandFields field to AgentStep
src/flowsh_cli/models.py	7,142	UPDATE	Import Self; relax agent validator conditionally
src/flowsh_cli/render.py	366-396	UPDATE	Reorder locals before substitution, expand fields
tests/test_workflow_to_harness.py	NEW	CREATE	Tests for expandFields behavior

Integration Points

• src/flowsh_cli/render.py:178-221 (run_agent function) already uses $model, $agent, $command in the shell — no changes needed there since substitution happens on local vars before calling run_agent
• src/flowsh_cli/models.py:142-147 (validate_agent) restricts agent to ^[A-Za-z0-9_-]+$ — $AGENT would fail validation, must be relaxed conditionally

Git History

• expandPrompt introduced: commit 83aaa40 — initial Python blueprint
• Implication: Limitation to prompt only has been present since inception

---

Implementation Plan

Step 1: Add expandFields field to AgentStep

File: src/flowsh_cli/models.py
Lines: 108-118 (after expandPrompt)
Action: UPDATE

Current code (lines 108-118):

    expandPrompt: bool = Field(
        default=False,
        description=(
            "When true, substitutes ${VAR} and $VAR tokens from vars steps into the prompt "
            "at runtime using safe plain-text replacement. "
            "Shell expressions such as $(cmd), `cmd`, $((expr)), and globs are NOT evaluated — "
            "they pass through to the agent literally. "
            "Only uppercase variable names matching [A-Z_][A-Z0-9_]* found in the prompt text "
            "are replaced. All other shell syntax in the prompt is safe to use."
        ),
    )

Required change: Add expandFields field after expandPrompt:

    expandPrompt: bool = Field(
        default=False,
        description=(
            "When true, substitutes ${VAR} and $VAR tokens from vars steps into the prompt "
            "at runtime using safe plain-text replacement. "
            "Shell expressions such as $(cmd), `cmd`, $((expr)), and globs are NOT evaluated — "
            "they pass through to the agent literally. "
            "Only uppercase variable names matching [A-Z_][A-Z0-9_]* found in the prompt text "
            "are replaced. All other shell syntax in the prompt is safe to use."
        ),
    )
    expandFields: bool = Field(
        default=False,
        description=(
            "When true, substitutes ${VAR} and $VAR tokens from params or vars steps "
            "into all string fields (model, agent, command, and prompt) at runtime "
            "using safe plain-text replacement. "
            "Implies expandPrompt behavior for the prompt field. "
            "Only uppercase variable names matching [A-Z_][A-Z0-9_]* are replaced."
        ),
    )

Why: Minimal API surface — Option A from the issue, single flag covering all expandable fields.

---

Step 2: Add Self import and conditionally relax agent validator

File: src/flowsh_cli/models.py
Lines: 7 (import), 142-147 (validator)
Action: UPDATE

Current import (line 7):

from typing import Annotated, Literal

Required change: Add Self:

from typing import Annotated, Literal, Self

Current validator (lines 142-147):

    @field_validator("agent")
    @classmethod
    def validate_agent(cls, value: str | None) -> str | None:
        if value is not None and not re.fullmatch(r"[A-Za-z0-9_-]+", value):
            raise ValueError("must match ^[A-Za-z0-9_-]+$")
        return value

Required change: Replace with @model_validator(mode="after") that checks expandFields:

    @field_validator("agent")
    @classmethod
    def validate_agent(cls, value: str | None) -> str | None:
        if value is not None and not re.fullmatch(r"[A-Za-z0-9_-]+", value):
            raise ValueError("must match ^[A-Za-z0-9_-]+$")
        return value

    @model_validator(mode="after")
    def validate_agent_with_expand_fields(self) -> Self:
        if self.agent is not None:
            pattern = r"[A-Za-z0-9_{}$-]+" if self.expandFields else r"[A-Za-z0-9_-]+"
            if not re.fullmatch(pattern, self.agent):
                raise ValueError(
                    f"agent must match ^{pattern}$"
                    + (" when expandFields is true" if self.expandFields else "")
                )
        return self

Why: The agent field regex ^[A-Za-z0-9_-]+$ rejects $, {, } characters needed for $AGENT/${AGENT} patterns. When expandFields is true, these are valid variable-reference characters.

---

Step 3: Reorder locals and extend substitution logic

File: src/flowsh_cli/render.py
Lines: 366-396
Action: UPDATE

Current code (lines 366-396):

    elif isinstance(step, AgentStep):
        delimiter = heredoc_delimiter("PROMPT", step.prompt)
        lines.extend(
            [
                "  local prompt",
                f"  prompt=$(cat <<'{delimiter}'",
                *step.prompt.splitlines(),
                delimiter,
                "  )",
            ]
        )
        if step.expandPrompt:
            _prompt_no_code = re.sub(r"```.*?```", "", step.prompt, flags=re.DOTALL)
            braced = re.findall(r"\$\{([A-Z_][A-Z0-9_]*)\}", _prompt_no_code)
            bare = re.findall(r"\$([A-Z_][A-Z0-9_]*)(?!\w)", _prompt_no_code)
            seen: dict[str, None] = {}
            for var in braced + bare:
                seen[var] = None
            for var in seen:
                lines.append(f'  _p=\'${{{var}}}\'; prompt="${{prompt//"$_p"/"${var}"}}"')
                lines.append(f'  _p=\'${var}\'; prompt="${{prompt//"$_p"/"${var}"}}"')
        lines.append(f"  local agent={bash_quote(step.agent or '')}")
        lines.append(f"  local model={bash_quote(step.model or '')}")
        lines.append(f"  local command={bash_quote(step.command or '')}")
        lines.append(f"  local capture={bash_quote(step.capture or '')}")
        dangerous_skip_permissions = "true" if step.dangerouslySkipPermissions else "false"
        lines.append(f"  local dangerously_skip_permissions={dangerous_skip_permissions}")
        lines.append(
            '  run_agent "$prompt" "$agent" "$model" '
            '"$command" "$capture" "$dangerously_skip_permissions"'
        )

Required change: Declare ALL local vars first, then apply substitution to applicable fields, then call run_agent:

    elif isinstance(step, AgentStep):
        delimiter = heredoc_delimiter("PROMPT", step.prompt)
        lines.extend(
            [
                "  local prompt",
                f"  prompt=$(cat <<'{delimiter}'",
                *step.prompt.splitlines(),
                delimiter,
                "  )",
            ]
        )
        lines.append(f"  local agent={bash_quote(step.agent or '')}")
        lines.append(f"  local model={bash_quote(step.model or '')}")
        lines.append(f"  local command={bash_quote(step.command or '')}")
        lines.append(f"  local capture={bash_quote(step.capture or '')}")
        dangerous_skip_permissions = "true" if step.dangerouslySkipPermissions else "false"
        lines.append(f"  local dangerously_skip_permissions={dangerous_skip_permissions}")

        fields_to_expand: dict[str, str] = {}
        if step.expandFields or step.expandPrompt:
            _prompt_no_code = re.sub(r"```.*?```", "", step.prompt, flags=re.DOTALL)
            fields_to_expand["prompt"] = _prompt_no_code
        if step.expandFields:
            if step.model is not None:
                fields_to_expand["model"] = step.model
            if step.agent is not None:
                fields_to_expand["agent"] = step.agent
            if step.command is not None:
                fields_to_expand["command"] = step.command

        if fields_to_expand:
            seen: dict[str, None] = {}
            for text in fields_to_expand.values():
                braced = re.findall(r"\$\{([A-Z_][A-Z0-9_]*)\}", text)
                bare = re.findall(r"\$([A-Z_][A-Z0-9_]*)(?!\w)", text)
                for var in braced + bare:
                    seen[var] = None
            for var in seen:
                for field_name in fields_to_expand:
                    lines.append(
                        f'  _p=\'${{{var}}}\'; {field_name}='
                        f'"${{{field_name}//"$_p"/"${{{var}}}"}}"'
                    )
                    lines.append(
                        f'  _p=\'${var}\'; {field_name}='
                        f'"${{{field_name}//"$_p"/"${{{var}}}"}}"'
                    )

        lines.append(
            '  run_agent "$prompt" "$agent" "$model" '
            '"$command" "$capture" "$dangerously_skip_permissions"'
        )

Why: All local vars must exist before substitution runs (substitution modifies them in place via ${var//pattern/replacement}). The approach avoids duplicating logic: variables are deduplicated across all expanded fields, and each field gets the same set of substitution lines.

---

Step 4: Add/Update Tests

File: tests/test_workflow_to_harness.py
Action: CREATE (add new test functions)

Test cases to add (insert after the existing expandPrompt tests around line 1082):

def test_render_harness_expand_fields_emits_substitution_for_model() -> None:
    workflow = Workflow(
        id="wf_expand_fields",
        name="Expand Fields",
        steps=[
            AgentStep(
                type="agent",
                prompt="Some prompt.",
                model="$MODEL",
                expandFields=True,
            )
        ],
    )

    script = render_harness(workflow)

    # Substitution lines emitted for model
    assert '_p=\'${MODEL}\'; model="${model//"$_p"/"$MODEL"}"' in script
    assert '_p=\'$MODEL\'; model="${model//"$_p"/"$MODEL"}"' in script
    # Prompt still captured via heredoc
    assert "prompt=$(cat <<'PROMPT_EOF'" in script


def test_render_harness_expand_fields_emits_substitution_for_agent_and_command() -> None:
    workflow = Workflow(
        id="wf_expand_fields_all",
        name="Expand Fields All",
        steps=[
            AgentStep(
                type="agent",
                prompt="Prompt with $VAR.",
                model="$MODEL",
                agent="$AGENT",
                command="$CMD",
                expandFields=True,
            )
        ],
    )

    script = render_harness(workflow)

    for field in ("prompt", "model", "agent", "command"):
        assert f'_p=\'${{VAR}}\'; {field}="${{{field}//"$_p"/"$VAR"}}"'
        assert f'_p=\'$VAR\'; {field}="${{{field}//"$_p"/"$VAR"}}"'
        assert f'_p=\'${{MODEL}}\'; {field}="${{{field}//"$_p"/"$MODEL"}}"'
        assert f'_p=\'$MODEL\'; {field}="${{{field}//"$_p"/"$MODEL"}}"'
        assert f'_p=\'${{AGENT}}\'; {field}="${{{field}//"$_p"/"$AGENT"}}"'
        assert f'_p=\'$AGENT\'; {field}="${{{field}//"$_p"/"$AGENT"}}"'
        assert f'_p=\'${{CMD}}\'; {field}="${{{field}//"$_p"/"$CMD"}}"'
        assert f'_p=\'$CMD\'; {field}="${{{field}//"$_p"/"$CMD"}}"'

Wait — the cross-product of variables × fields would generate a LOT of lines (4 variables × 4 fields × 2 forms = 32 substitution lines). Let me simplify the test to be more focused:

def test_render_harness_expand_fields_emits_substitution_for_agent_model_command() -> None:
    workflow = Workflow(
        id="wf_expand_fields_all",
        name="Expand Fields All",
        steps=[
            AgentStep(
                type="agent",
                prompt="Prompt with $VAR.",
                model="$MODEL",
                agent="$AGENT",
                command="$CMD",
                expandFields=True,
            )
        ],
    )

    script = render_harness(workflow)

    # Each field gets substitution lines for every unique variable
    for field in ("prompt", "model", "agent", "command"):
        for var in ("VAR", "MODEL", "AGENT", "CMD"):
            assert (
                f'_p=\'${{{var}}}\'; {field}="${{{field}//"$_p"/"${{{var}}}"}}"'
                in script
            )
            assert (
                f'_p=\'${var}\'; {field}="${{{field}//"$_p"/"${{{var}}}"}}"'
                in script
            )


def test_render_harness_expand_fields_implies_expand_prompt() -> None:
    """expandFields: true should expand prompt even without expandPrompt: true."""
    workflow = Workflow(
        id="wf_expand_fields_prompt",
        name="Expand Fields Prompt",
        steps=[
            AgentStep(
                type="agent",
                prompt="Work on $ISSUE.",
                model="$MODEL",
                expandFields=True,
            )
        ],
    )

    script = render_harness(workflow)

    # Prompt also gets substitution (expandFields implies expandPrompt)
    assert '_p=\'${ISSUE}\'; prompt="${prompt//"$_p"/"$ISSUE"}"' in script
    assert '_p=\'${MODEL}\'; model="${model//"$_p"/"$MODEL"}"' in script


def test_render_harness_expand_fields_code_fence_respected_for_prompt() -> None:
    """expandFields: true should still protect code-fenced vars in prompt."""
    workflow = Workflow(
        id="wf_expand_fields_fence",
        name="Expand Fields Fence",
        steps=[
            AgentStep(
                type="agent",
                prompt="Outside $OUTSIDE.\n\n```\nInside $INSIDE\n```",
                model="$MODEL",
                expandFields=True,
            )
        ],
    )

    script = render_harness(workflow)

    # OUTSIDE appears outside code block -> expanded
    assert '_p=\'${OUTSIDE}\'; prompt="${prompt//"$_p"/"$OUTSIDE"}"' in script
    # INSIDE appears only inside code block -> NOT expanded in prompt
    assert '_p=\'${INSIDE}\' not in script
    # But INSIDE might appear if also in model (it's not)
    # MODEL is in model field -> expanded in model
    assert '_p=\'${MODEL}\'; model="${model//"$_p"/"$MODEL"}"' in script


def test_parse_workflows_accepts_agent_expand_fields_boolean() -> None:
    workflow_file = Path("tests/workflows/agent_expand_fields.yml")
    workflow_file.parent.mkdir(parents=True, exist_ok=True)
    workflow_file.write_text(
        """\
workflows:
  - id: wf_expand_fields_parse
    name: Expand Fields Parse
    steps:
      - type: agent
        prompt: Test
        model: $MODEL
        expandFields: true
""",
        encoding="utf-8",
    )
    try:
        result = parse_workflows(workflow_file)
        step = result.workflows[0].steps[0]
        assert isinstance(step, AgentStep)
        assert step.expandFields is True
    finally:
        workflow_file.unlink()


def test_parse_workflows_rejects_agent_expand_fields_string() -> None:
    workflow_file = Path("tests/workflows/agent_expand_fields_invalid.yml")
    workflow_file.parent.mkdir(parents=True, exist_ok=True)
    workflow_file.write_text(
        """\
workflows:
  - id: wf_expand_fields_reject
    name: Expand Fields Reject
    steps:
      - type: agent
        prompt: Test
        expandFields: "true"
""",
        encoding="utf-8",
    )
    try:
        with pytest.raises(WorkflowParseError):
            parse_workflows(workflow_file)
    finally:
        workflow_file.unlink()


def test_generated_harness_expands_model_when_expand_fields_enabled(tmp_path: Path) -> None:
    """End-to-end test: model $MODEL is substituted at runtime."""
    workflow_file = tmp_path / "workflows.yml"
    workflow_file.write_text(
        """
workflows:
  - id: wf_model_expand
    name: Model Expand
    steps:
      - type: vars
        name: Set model
        values:
          MODEL: printf 'openai/gpt-4o'
      - type: agent
        name: Run agent
        expandFields: true
        model: $MODEL
        prompt: Work on my issue.
""".lstrip(),
        encoding="utf-8",
    )
    generated = subprocess.run(
        [sys.executable, "-m", "flowsh_cli", str(workflow_file)],
        check=False,
        capture_output=True,
        text=True,
        cwd=tmp_path,
    )
    assert generated.returncode == 0, generated.stderr

    bin_dir = tmp_path / "bin"
    bin_dir.mkdir()
    fake_opencode = bin_dir / "opencode"
    fake_opencode.write_text(
        """#!/usr/bin/env bash
printf '%s\\n' "$@" > "$OPENCODE_ARGS_CAPTURE"
printf '{"ok":true}\\n'
""",
        encoding="utf-8",
    )
    fake_opencode.chmod(0o700)
    args_capture = tmp_path / "opencode-args.txt"

    executed = subprocess.run(
        ["bash", str(tmp_path / "model_expand.sh")],
        check=False,
        capture_output=True,
        text=True,
        cwd=tmp_path,
        env={
            **os.environ,
            "PATH": f"{bin_dir}{os.pathsep}{os.environ['PATH']}",
            "OPENCODE_ARGS_CAPTURE": str(args_capture),
        },
    )
    assert executed.returncode == 0, executed.stderr
    args = args_capture.read_text(encoding="utf-8").splitlines()
    # Should contain --model openai/gpt-4o in the args
    assert "--model" in args
    model_idx = args.index("--model")
    assert args[model_idx + 1] == "openai/gpt-4o"

---

Patterns to Follow

From codebase — mirror these exactly:

# SOURCE: tests/test_workflow_to_harness.py:1003-1023
# Pattern for testing substitution line emission in rendered harness
def test_render_harness_safe_variable_substitution_when_expand_prompt_enabled() -> None:
    ...
    assert '_p=\'${ISSUE_NUMBER}\'; prompt="${prompt//"$_p"/"$ISSUE_NUMBER"}"' in script
    assert '_p=\'$ISSUE_NUMBER\'; prompt="${prompt//"$_p"/"$ISSUE_NUMBER"}"' in script

# SOURCE: src/flowsh_cli/render.py:377-386
# Pattern for variable detection and substitution line emission
_prompt_no_code = re.sub(r"```.*?```", "", step.prompt, flags=re.DOTALL)
braced = re.findall(r"\$\{([A-Z_][A-Z0-9_]*)\}", _prompt_no_code)
bare = re.findall(r"\$([A-Z_][A-Z0-9_]*)(?!\w)", _prompt_no_code)
seen: dict[str, None] = {}
for var in braced + bare:
    seen[var] = None
for var in seen:
    lines.append(f'  _p=\'${{{var}}}\'; prompt="${{prompt//"$_p"/"${var}"}}"')
    lines.append(f'  _p=\'${var}\'; prompt="${{prompt//"$_p"/"${var}"}}"')

---

Edge Cases & Risks

Risk/Edge Case	Mitigation
agent field contains $ but expandFields is false	Validator will reject it (unchanged from today)
expandFields: true with no vars steps or params before it	Substitution is a no-op — the variable patterns remain literal (safe)
Variable appears in model but not prompt	Substitution line is still emitted for all fields (harmless; no match = no-op)
Variable name collisions across fields	Deduplication via seen dict ensures each variable is processed once
expandFields: true combined with expandPrompt: true	Both behave identically — no conflict; expandFields implies prompt expansion
model/agent/command have $ that is not a valid variable (e.g., $123)	Regex [A-Z_][A-Z0-9_]* won't match — no substitution emitted (safe)
Existing expandPrompt behavior changes	expandPrompt continues to work identically — fields_to_expand includes prompt when either flag is set

---

Validation

Automated Checks

make qa

This runs the test suite and linting. Key test patterns to verify:

# Verify existing expandPrompt tests still pass
pytest tests/test_workflow_to_harness.py -k "expand_prompt" -v

# Verify new expandFields tests pass
pytest tests/test_workflow_to_harness.py -k "expand_fields" -v

Manual Verification

1. Create a workflow with expandFields: true and model: $MODEL, run it with MODEL set via a vars step or param, verify --model flag receives the substituted value
2. Run existing expandPrompt: true workflows to confirm no regression

---

Scope Boundaries

IN SCOPE:

• Add expandFields boolean field to AgentStep model
• Conditionally relax agent field validation when expandFields is true
• Extend the Python-side substitution emission to model, agent, command variables
• Keep existing expandPrompt flag working identically

OUT OF SCOPE (do not touch):

• Do not change the run_agent bash function — it already uses shell variables correctly
• Do not add per-field flags (expandModel, expandAgent, etc.) — use single expandFields flag
• Do not modify BashStep, VarsStep, ForStep, or other step types
• Do not modify the substitution regex or safety properties
• Do not change the generated harness structure beyond the agent step body

---

Metadata

• Investigated by: GHAR
• Timestamp: 2026-06-28T00:00:00Z
• Artifact: .ghar/issues/issue-34.md

[github-actions]

🕰️ This issue has been inactive for over 3 days.

@tbrandenburg - Could you provide an update on this issue?

If this issue is resolved or no longer relevant, please consider closing it.

[github-actions]

🕰️ This pull request has been inactive for over 3 days.

@tbrandenburg - Could you please review this PR or provide an update on its status?

If this PR is no longer needed, please consider closing it to keep the repository clean.

[github-actions]

🕰️ This issue has been inactive for over 3 days.

@tbrandenburg - Could you provide an update on this issue?

If this issue is resolved or no longer relevant, please consider closing it.