Remove the agent-capabilities (cohort comparison) feature - #1288
Open
kyle-compute wants to merge 1 commit into
Open
Remove the agent-capabilities (cohort comparison) feature#1288kyle-compute wants to merge 1 commit into
kyle-compute wants to merge 1 commit into
Conversation
Deletes the successful-vs-failing cohort comparison: its backend service and cohort blocks, both API endpoints (+ the deprecated /cohort-comparison alias), the public share endpoint, the frontend pane/routes/hooks, and every test whose subject was one of those files. A queued ANALYZER job with payload.mode = "agent_capabilities" now fails permanently instead of running. Preserved for the feature's planned return as a 'capabilities' analysis trial: the output schema (AgentCapabilitiesOutput and sub-models) moved into oddish.analyze.models, and analyze/prompts/agent_capabilities.txt kept in place. No migration: analyzer_blocks rows are untouched and remain readable. Claude-Session: https://claude.ai/code/session_01ACF6SUXdbLFarpj3qwF1Ki
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Oddish previewCommit:
Vercel deployment URL: https://oddish-ae2mp5tde.oddish.app Plan:
This comment is updated by the PR Preview workflow. |
This was referenced Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR deletes the agent capabilities feature.
It is PR B of six (A–F) rebuilding analysis so every analysis job runs as a normal trial. This is the rebuild of [#1122](#1122). PR A is [#1287](#1287).
B does not depend on A. Both branch from
stagingand can merge in either order.What is being deleted
Agent capabilities took one task version, compared its successful trials with its failing trials, and asked an LLM to describe the behavioral differences across categories such as planning, debugging, and scope adherence.
That feature currently consists of:
blocks/analyzer/cohort/prompt/schema/workspace packageThis PR removes all of that.
In total:
The task drawer now has only:
instead of:
What is intentionally kept
Capabilities will return later as a normal
'capabilities'analysis trial once PRs C–D add the new analysis pipeline.To avoid rebuilding its data contract later, this PR keeps:
AgentCapabilitiesOutputand its sub-modelsBehaviorCategorytaxonomyoddish/src/oddish/analyze/prompts/agent_capabilities.txtThe models move unchanged into:
What happens to already-queued jobs
A worker may still receive an old:
job after this deploy.
Instead of crashing or retrying forever, that job is now permanently failed once with:
This is the same cleanup pattern PR A uses for removed report jobs.
Database impact
There is no migration.
Existing capabilities results live in
analyzer_blocks. Those rows and the table are left untouched. PR E removes the table later.frontend/src/lib/experiment-agent-grouping.tsis also intentionally untouched. Despite the similar name, it is shared agent-grouping code used by unrelated parts of the frontend.What still works
Verification
The existing test suites are already red on a fresh database on unmodified
staging, so the meaningful check is whether this branch introduces new failures.I ran both branches against the same local Postgres 16 harness and compared the failure sets.
Result:
oddish: 89 failures, all 89 also fail on
stagingbackend: 56 failures vs. 57 on
stagingtest_agent_capabilities_endpoint.py, which this PR deletesbranch-only failures: 0
backend boots successfully with 176 routes
no remaining route matches
capabilitiesorcohortpnpm buildexits 0Tests
No new tests are added.
This is a deletion PR: tests whose only subject was the deleted feature are deleted with it. The preserved schema move was verified by importing it locally rather than adding a permanent test whose only purpose is to prove that deleted code is absent.