feat: Add AI PR analysis configuration and workflow#28024
feat: Add AI PR analysis configuration and workflow#28024
Conversation
|
CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes. |
github-actions
bot
added
size-L
risk-low
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
github-actions
bot
added
risk-low
- Added a step to checkout the AI Analyzer repository as a subdirectory. - Updated the workflow to use the local path for the AI Analyzer action instead of the remote reference.
github-actions
bot
added
risk-low
- Removed the local checkout step for the AI Analyzer repository. - Updated the workflow to use the remote reference for the AI Analyzer action instead of the local path.
github-actions
bot
added
risk-low
- Added a step to checkout the AI Analyzer repository as a subdirectory. - Updated the workflow to use the local path for the AI Analyzer action instead of the remote reference.
github-actions
bot
added
risk-low
…yzer action - Changed the checkout reference for the AI Analyzer action from 'main' to 'fix/install-all-deps' to ensure the workflow uses the correct branch for dependency installation.
github-actions
bot
added
risk-low
- Removed unnecessary environment variables and steps for determining PR number and base branch. - Updated the workflow to directly use inputs for mode, PR number, and provider, simplifying the process. - Adjusted permissions for pull-requests from read to write to enhance functionality.
github-actions
bot
added
risk-low
AI PR Analysis⚪ **** | Score: /100 | Confidence: % This PR refactors CI/test infrastructure to use an external AI analyzer action for E2E test selection and PR risk analysis. No production code is modified. The changes are purely workflow/configuration related with no impact on app functionality, controllers, or state management. The only risk is CI pipeline execution if the external action has issues, but this is mitigated by continue-on-error flags. |
- Introduced a new mode for selecting E2E and performance test tags based on regression risk. - Added configuration files for fallback scenarios, finalize schema, and hard rules for tag selection. - Updated the main configuration to include the new select-tags mode. - Enhanced the AI analysis workflow to utilize the new mode for improved test selection accuracy.
🔍 Smart E2E Test Selection
click to see 🤖 AI reasoning detailsE2E Test Selection: Performance Test Selection: |
| RISK_LEVEL=$(echo "$RESULT_JSON" | jq -r '.results["pr-risk-analysis"].risk_level // .results["select-tags"].risk_level // .risk_level // empty') | ||
| PERF_TAGS=$(echo "$RESULT_JSON" | jq -c '.results["select-tags"].performance_tests.selected_tags // .performance_tests.selected_tags // []') | ||
| REASONING=$(echo "$RESULT_JSON" | jq -r '.results["select-tags"].reasoning // .reasoning // "AI analysis completed without detailed reasoning."') | ||
| PERF_REASONING=$(echo "$RESULT_JSON" | jq -r '.results["select-tags"].performance_tests.reasoning // .performance_tests.reasoning // "No performance impact detected"') |
There was a problem hiding this comment.
Shell quoting breaks on AI reasoning with apostrophes
High Severity
RESULT_JSON is assigned via single-quoted bash string with inline ${{ }} expansion. GitHub Actions substitutes the raw JSON before bash parses the script. If the AI-generated reasoning field contains a single quote (e.g., "it's"), the bash single-quoted string is prematurely terminated, causing a syntax error. The step fails silently and all AI results are discarded in favor of the ["ALL"] fallback.
| - Signature requests | ||
| - Smart transactions | ||
|
|
||
| **Typical Test Coverage:** SmokeConfirmationsRedesigned, SmokeWalletPlatform |
There was a problem hiding this comment.
Skill file references non-existent test tag name
Low Severity
The skill file references SmokeConfirmationsRedesigned on lines 470 and 583, but this tag does not exist. The valid tag catalog in prompt-context.md and tests/tags.js defines only SmokeConfirmations. This could mislead the AI model into suggesting or referencing a non-existent tag during analysis.
Additional Locations (1)
…ndency installation - Changed the checkout reference for the AI Analyzer action from 'main' to 'fix/install-all-deps' to ensure the workflow uses the correct branch for dependency installation.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
There are 4 total unresolved issues (including 2 from previous reviews).
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
| uses: actions/checkout@v4 | ||
| with: | ||
| repository: MetaMask/ai-analyzer | ||
| ref: fix/install-all-deps |
There was a problem hiding this comment.
CI pinned to ephemeral feature branch reference
Medium Severity
Both the composite action and the standalone workflow pin the MetaMask/ai-analyzer checkout to ref: fix/install-all-deps, which is a feature branch. Once that branch is merged and deleted in the upstream repo, the actions/checkout step will fail, breaking the smart E2E selection in CI and the standalone analysis workflow. A stable ref (tag or main) is needed to avoid this fragile dependency.
Additional Locations (1)
| provider: ${{ inputs.provider }} | ||
| claude-api-key: ${{ secrets.E2E_CLAUDE_API_KEY }} | ||
| openai-api-key: ${{ secrets.E2E_OPENAI_API_KEY }} | ||
| google-api-key: ${{ secrets.E2E_GEMINI_API_KEY }} |
There was a problem hiding this comment.
Workflow missing config-path, github-token, and base-branch parameters
Medium Severity
The ai-pr-analysis.yml workflow invokes .ai-analyzer-action without passing config-path, github-token, or base-branch — all of which the working invocation in action.yml provides. Without config-path: .ai-pr-analyzer, the analyzer cannot find the prompt files, schemas, and hard rules added in this very PR. Without github-token, it likely cannot access the PR API for diff data. This workflow will fail or produce empty/incorrect results when triggered.
Additional Locations (1)
|
✅ E2E Fixture Validation — Schema is up to date |
|
2 participants
Description
Changelog
CHANGELOG entry:
Related issues
Fixes:
Manual testing steps
Screenshots/Recordings
Before
After
Pre-merge author checklist
Pre-merge reviewer checklist
Note
Medium Risk
Changes CI orchestration and test selection logic (including external action checkout and fallback behavior), which could incorrectly skip or broaden E2E/perf coverage if misconfigured.
Overview
Adds a new
.ai-pr-analyzer/configuration bundle (modes forpr-risk-analysisandselect-tags, hard rules, schemas, prompts, and fallbacks) to drive AI-based regression risk scoring and E2E/performance test tag selection.Updates the
smart-e2e-selectioncomposite action andci.ymlto runMetaMask/ai-analyzer(checked out via token) and derive outputs/comment content from itsresult_json, removing the prior local Node-based analyzer runner (.github/scripts/e2e-smart-selection.mjs). Also introduces an on-demandAI PR Analysisworkflow for running the analyzer directly, and refreshes docs/references (tests/tags.js, analyzer README) to point to the new config source.Written by Cursor Bugbot for commit 014d185. This will update automatically on new commits. Configure here.