refactor(core): Align Instance AI prompts and skill guidance (no-changelog) - #37910
refactor(core): Align Instance AI prompts and skill guidance (no-changelog)#37910Swiftyos wants to merge 4 commits into
Conversation
PR review overviewBased on ownership of the 49 changed files in this PR:
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Instance AI Discovery Eval ✅Branch: Eval output |
|
Addressed all four review findings in 98c740a. Also updated the two stale prompt assertions reported by CI. The verdict test now checks evidence scope and untested paths. The credential test still checks all supported secret-entry routes. Validation: all 4,835 Instance AI tests pass across 304 files. Package lint and typecheck pass. These changes stay within prompt text and prompt tests. |
There was a problem hiding this comment.
0 issues found across 9 files (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Shadow auto-approve: would require human review. This aligns AI prompts and routing, changing decision behavior without live LLM validation. Product and operational tradeoffs require human sign-off.
Re-trigger cubic
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Instance AI Workflow EvalImportant This eval does not re-run on new commits. To test your latest push, re-run it against the PR head: gh workflow run ci-instance-ai-evals.yml -f pr=37910…or use the Run workflow button and set pr = Warning 🟡 All 25 units green over 3 runs, but 1 barely passed (failed most runs) — see Failures below. Gate: pass@k = 100% (every unit passes at least once across k runs) — 94.7% pass (71/75 trials over 25 units · k=3) pass@3 100.0% · pass^3 90.5% · LangSmith experiment Failures (3)
Per-test-case results (8)
Workflow checksScored over 24 successful build(s). N/A = check did not apply to that workflow. Error = check could not be measured (e.g. judge timeout).
All workflow checks (5 failing of 35 checks)
|
|
Addressed the workflow-eval findings in 18ab7ee. The shared question rule, ask-user description, and builder skill now allow missing input-schema questions before building and prohibit invented incoming field names. Plan revisions after rejection must pass approval before building. The builder must trace the saved graph from each trigger through its required actions. Validation: 4,836 Instance AI tests pass. Package lint and typecheck pass. Requested a new workflow eval run against this head. The previous discovery run passed all 51 trials, but GitHub marked the job cancelled; that job was also retried. No unresolved review threads remain. |
There was a problem hiding this comment.
0 issues found across 6 files (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Shadow auto-approve: would require human review. Refactors Instance AI prompts and skills, changing routing, credential handling, and one-off execution guidance; behavioral policy shifts and no live LLM validation require human sign-off.
Re-trigger cubic
Summary
Instance AI receives conflicting instructions about setup, verification, and completion. A successful save or partial check can also lead to an unsupported success claim.
This change:
Production changes affect prompt text and prompt composition only. Tool schemas, execution logic, approval enforcement, and memory storage are unchanged.
How to test
Use an instance with the AI Assistant model and sandbox configured. Enable the Agents module for the embedded builder cases.
Validation:
--incremental falseafter stale incremental diagnostics.Live LLM evaluations were not run. Unit tests check prompt composition and reference loading, not model adherence or abandonment rates.
Related Linear tickets, Github issues, and Community forum posts
https://linear.app/n8n/issue/INS-1361
Review / Merge checklist
Backport to Beta,Backport to Stable, orBackport to v1(if the PR is an urgent fix that needs to be backported)🤖 PR Summary generated by AI