Agent Automation #15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Agent Automation | |
| on: | |
| schedule: | |
| - cron: "0 6 * * 1-5" | |
| workflow_dispatch: | |
| permissions: | |
| contents: read | |
| concurrency: | |
| group: provider-gap-audit-scheduled-${{ github.ref }} | |
| cancel-in-progress: false | |
| jobs: | |
| audit-sdk-coverage: | |
| name: Audit SDK Coverage (${{ matrix.repo }}) | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 45 | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| include: | |
| - owner: braintrustdata | |
| repo: braintrust-sdk-javascript | |
| max_issues: 3 | |
| - owner: braintrustdata | |
| repo: braintrust-sdk-python | |
| max_issues: 3 | |
| - owner: braintrustdata | |
| repo: braintrust-sdk-go | |
| max_issues: 3 | |
| - owner: braintrustdata | |
| repo: braintrust-sdk-ruby | |
| max_issues: 3 | |
| - owner: braintrustdata | |
| repo: braintrust-sdk-rust | |
| max_issues: 3 | |
| - owner: braintrustdata | |
| repo: braintrust-sdk-java | |
| max_issues: 3 | |
| - owner: braintrustdata | |
| repo: braintrust-sdk-dotnet | |
| max_issues: 3 | |
| steps: | |
| - name: Generate GitHub App token | |
| id: app-token | |
| uses: actions/create-github-app-token@f8d387b68d61c58ab83c6c016672934102569859 # v3.0.0 | |
| with: | |
| app-id: ${{ secrets.BRAINTRUST_BOT_APP_ID }} | |
| private-key: ${{ secrets.BRAINTRUST_BOT_PRIVATE_KEY }} | |
| owner: ${{ matrix.owner }} | |
| repositories: | | |
| ${{ matrix.repo }} | |
| permission-contents: read | |
| permission-issues: write | |
| - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 | |
| with: | |
| repository: ${{ matrix.owner }}/${{ matrix.repo }} | |
| token: ${{ steps.app-token.outputs.token }} | |
| persist-credentials: false | |
| - name: Run Claude provider gap audit | |
| uses: anthropics/claude-code-action@df37d2f0760a4b5683a6e617c9325bc1a36443f6 # v1.0.75 | |
| with: | |
| anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} | |
| github_token: ${{ steps.app-token.outputs.token }} | |
| show_full_output: "true" | |
| display_report: "true" | |
| plugin_marketplaces: | | |
| https://github.com/braintrustdata/braintrust-claude-plugin.git | |
| plugins: | | |
| trace-claude-code@braintrust-claude-plugin | |
| settings: | | |
| { | |
| "env": { | |
| "TARGET_REPO_OWNER": "${{ matrix.owner }}", | |
| "TARGET_REPO_NAME": "${{ matrix.repo }}", | |
| "TRACE_TO_BRAINTRUST": "true", | |
| "BRAINTRUST_CC_PROJECT": "${{ vars.BRAINTRUST_CC_PROJECT }}", | |
| "BRAINTRUST_API_KEY": "${{ secrets.BRAINTRUST_API_KEY }}" | |
| } | |
| } | |
| prompt: | | |
| # Goal | |
| Find important instrumentation gaps in this repository's generative-AI and embeddings integrations. | |
| A gap means the upstream SDK or framework clearly supports something that this repository does not yet instrument, or instruments with materially less detail. | |
| # Scope | |
| - Infer the relevant surfaces from the checked-out repository itself. | |
| - Focus only on repo-owned generative or embeddings execution instrumentation surfaces with meaningful upstream API, docs, and release surfaces. | |
| - In-scope examples include generations/completions/responses APIs, embeddings APIs, streaming generation flows, tool-calling during model execution, and agentic run/tracing surfaces. | |
| - Explicitly exclude CRUD-style parity work (resource create/list/retrieve/update/delete APIs, admin/configuration endpoints, and similar non-generative management APIs). | |
| - Ignore generic runtime or infrastructure integrations that do not have an upstream AI API surface to compare against. | |
| - Do not open parity issues for deprecated or no-op surfaces unless you find real deprecation drift or docs drift worth reporting. | |
| # Process | |
| 1. Inspect local code, tests, docs, examples, and e2e scenarios to understand what is already instrumented. | |
| 2. For each relevant surface, independently discover the current official upstream docs and recent official releases or changelogs. | |
| 3. Compare current upstream generative and embeddings execution capabilities to current Braintrust instrumentation in this repo. | |
| 4. Always check the latest Braintrust docs at https://www.braintrust.dev/docs before deciding how to describe a gap. | |
| 5. Search existing GitHub issues for duplicates before creating anything. | |
| - Only search in `TARGET_REPO_OWNER/TARGET_REPO_NAME`. | |
| - For every GitHub MCP tool call (`mcp__github__search_issues`, `mcp__github__list_issues`, `mcp__github__get_issue`, `mcp__github__get_issue_comments`, `mcp__github__create_issue`), explicitly pass owner=`TARGET_REPO_OWNER` and repo=`TARGET_REPO_NAME`. `TARGET_REPO_OWNER` and `TARGET_REPO_NAME` are environment variables. | |
| - Never query or create issues in the repository running this workflow. | |
| 6. Only act on high-confidence, concrete gaps tied to missing generative or embeddings execution APIs, unsupported model execution call patterns, or missing execution instrumentation detail. | |
| # Examples | |
| ## Good | |
| - The upstream SDK now has a stable `responses.stream()` helper, but this repo does not instrument it at all, or instruments it without final result metadata that other similar APIs already capture here. | |
| - This repo instruments a provider's basic text generation API, but not its newer tool-calling or agent tracing API even though that API is now official and documented. | |
| - The upstream SDK has an official embeddings API surface (for example batch embeddings options), but this repo does not instrument those embeddings calls with the same depth as comparable model execution APIs. | |
| ## Bad | |
| - A vague suspicion that "something in streaming may be missing" without a concrete upstream API and a concrete repo gap. | |
| - Opening a separate issue for every release note bullet when they all describe the same missing instrumentation area. | |
| - Filing an issue for CRUD-only endpoint coverage (for example files/threads/assistants management APIs) when no generative or embeddings execution instrumentation gap is involved. | |
| # If You Find Actionable Non-Duplicate Gaps | |
| - Create at most ${{ matrix.max_issues }} issues in this run. | |
| - Create one issue per distinct gap. | |
| - Keep each issue concise, concrete, and source-backed. | |
| - Prefix every created issue title with `[BOT ISSUE] `. | |
| - If the repository supports issue types, choose whichever type fits best (`Bug`, `Feature`, or `Task`). If issue types are unavailable, create the issue without type. | |
| - Include a hidden marker comment near the top of the issue body in this exact form: | |
| ```html | |
| <!-- provider-gap-audit: <gap_id> --> | |
| ``` | |
| Each issue should clearly include: | |
| - what instrumentation is missing | |
| - whether Braintrust docs suggest the capability is `supported`, `unclear`, or `not_found` | |
| - exact upstream sources | |
| - exact Braintrust docs source or sources | |
| - exact local repo files you inspected | |
| # Duplicate Handling | |
| - Do not create an issue if an open issue already covers the same gap. | |
| - Treat a matching hidden marker comment or a clearly equivalent open issue as a duplicate. | |
| - If duplicate checking is inconclusive, do not create the issue. | |
| # Constraints | |
| - Discover source URLs yourself. Do not rely on a preset list. | |
| - Prefer official docs and official release sources. | |
| - Do not create comments. | |
| - Do not update, close, or label existing issues. | |
| - Do not create pull requests. | |
| - If there are no high-confidence non-duplicate gaps, do nothing. | |
| claude_args: | | |
| --model claude-opus-4-6 | |
| --max-turns 100 | |
| --allowedTools "Read,Glob,Grep,LS,WebSearch,WebFetch,mcp__github__get_issue,mcp__github__get_issue_comments,mcp__github__search_issues,mcp__github__list_issues,mcp__github__create_issue" | |
| --disallowedTools "Bash,Edit,MultiEdit,Write,Replace,NotebookEditCell,mcp__github__create_issue_comment,mcp__github__update_issue,mcp__github__create_pr,mcp__github__create_or_update_file,mcp__github__delete_file,mcp__github_file_ops__commit_files,mcp__github_file_ops__delete_files" |