release: v1.108.177 — an omitted match is not an absent match #126
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Replay | |
| # Retrieval-quality regression gate. Indexes this repo, replays a fixed set of | |
| # golden queries through search_symbols, and fails if nDCG / MRR / Recall@k drop | |
| # more than 2% (relative) below the committed golden baseline. This is the gate | |
| # that lets ranking-affecting changes (fusion weights, BM25 normalization, parser | |
| # extraction) land with a proof they did not regress retrieval quality. | |
| on: | |
| push: | |
| branches: [main] | |
| pull_request: | |
| permissions: | |
| contents: read | |
| jobs: | |
| replay: | |
| name: Retrieval-quality gate | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5 | |
| - name: Set up uv | |
| uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7 | |
| with: | |
| python-version: "3.12" | |
| - name: Install | |
| run: uv sync --group dev | |
| - name: Index self + run replay gate | |
| run: | | |
| set -euo pipefail | |
| STORE="${RUNNER_TEMP}/replay-index" | |
| # The repo id is a hash of the absolute index path, which differs on a | |
| # CI runner, so capture the actual id and pass it through --repo. | |
| REPO_ID=$(uv run python -c "from jcodemunch_mcp.tools.index_folder import index_folder; print(index_folder(path='.', use_ai_summaries=False, incremental=False, storage_path='${STORE}')['repo'])") | |
| echo "indexed repo: ${REPO_ID}" | |
| uv run python benchmarks/replay/run_replay.py \ | |
| --fixture benchmarks/replay/fixtures/self_v1_75_0.json \ | |
| --repo "${REPO_ID}" \ | |
| --storage-path "${STORE}" \ | |
| --baseline-file benchmarks/replay/results/self_v1_75_0-golden.json \ | |
| --gate 0.02 |