[eval] Store evaluator output directly in FineStore - #9177
Draft
loom-oa-dev[bot] wants to merge 15 commits into
Draft
loom-oa-dev[bot] wants to merge 15 commits into
loom-oa-dev[bot] wants to merge 15 commits into
Conversation
Ship the schema-v4 evaluation records, lm-eval row mapper, and EvaluationStore in marin-finestore. Harbor already depends on this API for its per-trial END hook, and Evalchemy needs the same API to write normalized samples without importing marin-core. Move all Marin and Evaldash consumers to the packaged module. Changes to its source now select the python-libs release family, so merging this commit publishes a wheel instead of waiting for the daily release. The mapper expands Evalchemy's one-record-per-document filter cohorts into distinct (task, doc_id, trial_id, filter) table rows.
Keep package-local imports in their configured section and remove a private docstring that repeated the helper name.
Pass each run archive directly to Evalchemy and Harbor, then summarize their preserved sources for coverage without rebuilding normalized tables in Marin. Keep aggregate records and retry behavior unchanged.
Share sample coverage accumulation between legacy and native sources, normalize object-key checks, and remove Harbor fields that no longer feed the aggregate after native archive writing.
Publish the normalized evaluation schemas and generic archive operations without interpreting evaluator-native records. Evalchemy owns lm-eval row conversion, and Harbor owns trajectory flattening.\n\nKeep Marin's lm-eval conversion only for rebuilding historical archives. Move metric and filter selection into Marin because those priorities are dashboard and measurement policy.
Describe Marin's lm-eval converter as the compatibility path for outputs that lack native FineStore writes. Use the schema's canonical steps table name in its archive round-trip test.
Reconcile native evaluator output with the corrected FineStore boundary. Evalchemy and Harbor own native conversion; Marin keeps compatibility and migration conversion outside the FineStore package.
Read Evalchemy’s normalized FineStore table for coverage instead of reconverting its native JSONL in Marin. Refresh the evaluation docs and remove legacy-only test setup from the native path.
loom-oa-dev
Bot
added
agent-generated
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Route native evaluator output through FineStore end to end. Marin passes each run's archive root to Evalchemy and configures Harbor's archive hook, but it does not own either evaluator's conversion policy.
Evalchemy now writes its aggregate JSON, per-task sample JSONL, and normalized sample rows directly to FineStore. The Marin client keeps only a temporary local aggregate file for completion checking, uploads no second results tree, and reads headline metrics from Evalchemy's FineStore source artifacts. Harbor writes typed trial results, trajectories, the job config, lock, and result files, normalized samples, and normalized steps through its native archive hook; its ordinary job tree remains checkpoint state.
The shared schema and storage API shipped in #9158. Before this PR is marked ready, the external source pins will advance to the merged producer changes in marin-community/evalchemy#126 and marin-community/harbor#152.
Repository lint passed after merging current
main. The affected test runner passed 1,872 tests, skipped 4, and reported 5 expected failures; the focused evaluator and archive tests passed all 81 tests.