Skip to content

refactor(core): Align Instance AI prompts and skill guidance (no-changelog) - #37910

Open
Swiftyos wants to merge 4 commits into
masterfrom
ins-1361-tech-debt-review-and-optimise-llm-promting
Open

refactor(core): Align Instance AI prompts and skill guidance (no-changelog)#37910
Swiftyos wants to merge 4 commits into
masterfrom
ins-1361-tech-debt-review-and-optimise-llm-promting

Conversation

@Swiftyos

@Swiftyos Swiftyos commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Instance AI receives conflicting instructions about setup, verification, and completion. A successful save or partial check can also lead to an unsupported success claim.

This change:

  • Defines shared rules for evidence, completion, questions, and recovery.
  • Aligns workflow skills, tool guidance, the embedded Agent Builder, and memory prompts.
  • Keeps ambiguous credential selection in post-build setup and removes incorrect manual verification-token advice.
  • Moves detailed recipes and examples into references with explicit load conditions.
  • Documents prompt ownership and updates prompt-contract tests.

Production changes affect prompt text and prompt composition only. Tool schemas, execution logic, approval enforcement, and memory storage are unchanged.

How to test

Use an instance with the AI Assistant model and sandbox configured. Enable the Agents module for the embedded builder cases.

  1. Build a workflow with missing credentials. Check that the assistant saves a draft before credential selection.
  2. Verify a workflow with simulated writes or an unreached required branch. Check that the reply states the actual coverage and limits.
  3. Edit the workflow after verification. Check that the assistant does not use old evidence to claim the edited behavior works.
  4. Resume a setup card after a skip. Check that the assistant respects the skip and continues the pending work.
  5. Check an explicit publication request with disclosed untested behavior. Check that the existing approval flow remains available.

Validation:

  • Full repository build passed (70 tasks).
  • 277 focused unit tests passed across Instance AI, Agents, and the CLI delegate.
  • Full repository lint passed (133 tasks) with an 8 GB Node heap. Lint also passed in all four changed packages.
  • Typecheck passed in Instance AI, Agents, and nodes-base. CLI typecheck passed with --incremental false after stale incremental diagnostics.
  • Biome and Prettier commit hooks passed.

Live LLM evaluations were not run. Unit tests check prompt composition and reference loading, not model adherence or abandonment rates.

Related Linear tickets, Github issues, and Community forum posts

https://linear.app/n8n/issue/INS-1361

Review / Merge checklist

  • I have seen this code, I have run this code, and I take responsibility for this code.
  • PR title and summary are descriptive. (conventions)
  • Docs updated or follow-up ticket created.
  • Tests included.
  • PR Labeled with Backport to Beta, Backport to Stable, or Backport to v1 (if the PR is an urgent fix that needs to be backported)

🤖 PR Summary generated by AI

@n8n-assistant

n8n-assistant Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

PR review overview

Based on ownership of the 49 changed files in this PR:

Ownership Files owned Share Source code Test files Misc
@n8n-io/ai-assistant 30 61% +38 / -15 +123 / -72 +584 / -515
@n8n-io/ai 13 27% +11 / -7 +31 / -18 +0 / -0
@n8n-io/catalysts 3 6% +13 / -5 +0 / -0 +0 / -0
Other teams 3 6% +3 / -3 +0 / -0 +0 / -0
Total 49 100% +65 / -30 +154 / -90 +584 / -515

@n8n-assistant n8n-assistant Bot added the n8n team Authored by the n8n team label Sep 5, 2026
cubic-dev-ai[bot]

This comment was marked as outdated.

@codecov

codecov Bot commented Sep 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Instance AI Discovery Eval ✅

Branch: ins-1361-tech-debt-review-and-optimise-llm-promting · Commit: 09a928c49801fb250e2137e5f30cb7d7699cd150

Eval output
$ tsx evaluations/discovery/cli.ts --trials 3 --fail-on-zero-pass
Running 17 discovery scenario(s) × 3 trial(s) (model: anthropic/claude-sonnet-4-6, concurrency: 3).

▸ config-evals-skill-loading ... (node:4878) [DEP0205] DeprecationWarning: `module.register()` is deprecated. Use `module.registerHooks()` instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
✓ 3/3 passed (100%)
▸ data-table-natural-list-skill-loading ... ✓ 3/3 passed (100%)
▸ data-table-skill-loading ... ✓ 3/3 passed (100%)
▸ data-table-workflow-skill-loading ... ✓ 3/3 passed (100%)
▸ google-oauth-credential-setup ... ✓ 3/3 passed (100%)
▸ http-node-config-no-browser ... ✓ 3/3 passed (100%)
▸ mcp-broken-connection-reconnect ... ✓ 3/3 passed (100%)
▸ mcp-connect-unconnected-service ... ✓ 2/3 passed (67%)
▸ mcp-declined-tool-call-no-reconnect ... ✓ 3/3 passed (100%)
▸ mcp-no-registry-match ... ✓ 3/3 passed (100%)
▸ mcp-not-offered-for-workflow-build ... ✓ 3/3 passed (100%)
▸ mcp-uses-connected-server-tools ... ✓ 3/3 passed (100%)
▸ oauth-with-computer-use-disabled ... ✓ 3/3 passed (100%)
▸ screenshot-dashboard ... ✓ 2/3 passed (67%)
▸ slack-oauth-credential-setup ... ✓ 3/3 passed (100%)
▸ workflow-builder-no-agent-builder-leak ... ✓ 3/3 passed (100%)
▸ workflow-builder-no-credential-ask ... ✓ 3/3 passed (100%)

=== Summary ===
Scenarios: 17/17 above threshold (67%)
Trials: 49/51 passed (96%)
Total time: 1381.9s

@Swiftyos

Swiftyos commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Addressed all four review findings in 98c740a. Also updated the two stale prompt assertions reported by CI. The verdict test now checks evidence scope and untested paths. The credential test still checks all supported secret-entry routes.

Validation: all 4,835 Instance AI tests pass across 304 files. Package lint and typecheck pass. These changes stay within prompt text and prompt tests.

cubic-dev-ai[bot]

This comment was marked as outdated.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 9 files (changes from recent commits).

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Shadow auto-approve: would require human review. This aligns AI prompts and routing, changing decision behavior without live LLM validation. Product and operational tradeoffs require human sign-off.

Re-trigger cubic

@Swiftyos
Swiftyos marked this pull request as ready for review September 5, 2026 14:55
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 5, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-05T14:59:59.741184Z 0b072d3 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Instance AI Workflow Eval

Important

This eval does not re-run on new commits. To test your latest push, re-run it against the PR head:

gh workflow run ci-instance-ai-evals.yml -f pr=37910

…or use the Run workflow button and set pr = 37910.

Warning

🟡 All 25 units green over 3 runs, but 1 barely passed (failed most runs) — see Failures below.

Gate: pass@k = 100% (every unit passes at least once across k runs) — 94.7% pass (71/75 trials over 25 units · k=3)

pass@3 100.0% · pass^3 90.5% · LangSmith experiment

Failures (3)

notification-router :: Before building the workflow, the agent established the inco — passed 2/3

The agent never asked about the payload field name and assumed urgency in the initial build; the user had to correct it in turn 2, confirming the agent invented a field the user never described.

refuses-deprecated-chat-model :: The OpenAI chat-model node's model is a currently supported — passed 1/3

The final model value is 'gpt-4o', not any of the GPT-5 family models listed as currently supported (gpt-5, gpt-5.5, gpt-5-mini, gpt-5.4, gpt-5.6); the agent switched from gpt-5.4 to gpt-4o after incorrectly concluding gpt-5.4 wasn't a real model.

The final workflow uses gpt-4.1, which is in the gpt-4 family; the expectation requires a gpt-5 series model (e.g. gpt-5, gpt-5.5, gpt-5-mini, gpt-5.4, gpt-5.6) and explicitly excludes gpt-4 and 'any other deprecated model ID'.

revises-plan-after-rejection/urgent-rows-only — passed 2/3

[builder_issue] The workflow errors at 'Mark as Posted' with 'row_number is null or undefined'. The Slack node ('Post to #urgent-alerts') outputs Slack API response fields (ok, channel, message, message_timestamp) — not the original lead data. The 'Mark as Posted' node references $json.row_number but that field does not exist in the Slack node's output. The builder needed to either reference the paired item from an earlier node (e.g., $('Only Urgent & Not Posted').item.json.row_number) or use a Set node to carry row_number forward before posting to Slack. The two Urgent rows (Acme, Globex) were posted successfully to #urgent-alerts and New rows were correctly filtered out, but the run does not complete without errors as required by the checklist.

Per-test-case results (8)
Workflow Status pass@3 pass^3
airtable-split-to-slack 3/3 100% 100%
notification-router 3/3 100% 88%
refuses-deprecated-chat-model 3/3 100% 76%
rest-api-data-pipeline 3/3 100% 100%
revises-plan-after-rejection 3/3 100% 86%
telegram-chatbot-memory-session 3/3 100% 100%
weather-alert 3/3 100% 100%
workflow-data-table 3/3 100% 100%

Workflow checks

Scored over 24 successful build(s). N/A = check did not apply to that workflow. Error = check could not be measured (e.g. judge timeout).

Dimension Check Kind Pass Fail N/A Error Pass rate
parameter_correctness correct_node_operations llm 23 1 0 0 96%
parameter_correctness valid_data_flow llm 23 1 0 0 96%
parameter_correctness valid_field_references deterministic 19 2 3 0 90%
intent_match fulfills_user_request llm 22 2 0 0 92%
communication response_describes_changes_accurately llm 20 4 0 0 83%
All workflow checks (5 failing of 35 checks)
Dimension Check Kind Pass Fail N/A Error Pass rate
structure has_nodes deterministic 24 0 0 0 100%
structure has_start_node deterministic 24 0 0 0 100%
structure has_trigger deterministic 24 0 0 0 100%
structure no_disabled_nodes deterministic 24 0 0 0 100%
connection_topology all_nodes_connected deterministic 24 0 0 0 100%
connection_topology error_routes_consistent deterministic 24 0 0 0 100%
connection_topology handles_multiple_items llm 24 0 0 0 100%
connection_topology no_unreachable_nodes deterministic 24 0 0 0 100%
connection_topology switch_fallback_output_enabled deterministic 3 0 21 0 100%
parameter_correctness correct_node_operations llm 23 1 0 0 96%
parameter_correctness expressions_reference_existing_nodes deterministic 4 0 20 0 100%
parameter_correctness google_sheets_rlc_default_mode deterministic 3 0 21 0 100%
parameter_correctness http_generic_auth_type_matches_prompt deterministic 0 0 24 0
parameter_correctness item_flow_independent_source_execute_once deterministic 0 0 24 0
parameter_correctness item_flow_paired_item_references deterministic 0 0 24 0
parameter_correctness no_empty_set_nodes deterministic 0 0 24 0
parameter_correctness no_invalid_from_ai deterministic 0 0 24 0
parameter_correctness valid_data_flow llm 23 1 0 0 96%
parameter_correctness valid_field_references deterministic 19 2 3 0 90%
parameter_correctness valid_node_config deterministic 24 0 0 0 100%
intent_match fulfills_user_request llm 22 2 0 0 92%
communication response_describes_changes_accurately llm 20 4 0 0 83%
ai_nodes agent_has_dynamic_prompt deterministic 6 0 18 0 100%
ai_nodes agent_has_language_model deterministic 6 0 18 0 100%
ai_nodes memory_properly_connected deterministic 6 0 18 0 100%
ai_nodes memory_session_key_expression deterministic 6 0 18 0 100%
ai_nodes tools_have_parameters deterministic 0 0 24 0
ai_nodes vector_store_has_embeddings deterministic 0 0 24 0
nodes_craftsmanship code_node_no_http_requests deterministic 3 0 21 0 100%
nodes_craftsmanship descriptive_node_names llm 24 0 0 0 100%
nodes_craftsmanship no_unnecessary_code_nodes llm 24 0 0 0 100%
efficiency no_excessive_build_failures deterministic 24 0 0 0 100%
security inbound_trigger_auth_defaults deterministic 6 0 18 0 100%
security no_hardcoded_credentials deterministic 0 0 24 0
security secrets_use_credentials_not_parameters deterministic 9 0 15 0 100%

@Swiftyos

Swiftyos commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the workflow-eval findings in 18ab7ee. The shared question rule, ask-user description, and builder skill now allow missing input-schema questions before building and prohibit invented incoming field names. Plan revisions after rejection must pass approval before building. The builder must trace the saved graph from each trigger through its required actions.

Validation: 4,836 Instance AI tests pass. Package lint and typecheck pass. Requested a new workflow eval run against this head. The previous discovery run passed all 51 trials, but GitHub marked the job cancelled; that job was also retried. No unresolved review threads remain.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 6 files (changes from recent commits).

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Shadow auto-approve: would require human review. Refactors Instance AI prompts and skills, changing routing, credential handling, and one-off execution guidance; behavioral policy shifts and no live LLM validation require human sign-off.

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed n8n team Authored by the n8n team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant