Conversation
Checkpoint the single-step spike with 41 source specifications and 81 Harbor exports. Separate semantic environment requirements, answer extraction, and private verifier resources so the same task can run with different submission conventions and compatible execution environments. Include TaskTrove, GSM8K, and R2E-Gym importers, structured rejections, reproducible examples, and Harbor adapters. Published examples use local Docker image IDs; rebuilding images is required on another machine. Training integration and target-model suitability studies remain deferred.
Task specifications now carry reviewed coverage tags for the bounded 50-task example suite. Rendered tasks retain those labels and add result encoding tags for JSON, XML, and file submission without representing formatting as a competency. The labels classify difficulty for an approximately 2B-active Snowball model and are included in the schema, Parquet records, and example builders.
Add a Dataset Viewer sampler that pins source revisions and selects two deterministic rows from every non-blend NeMo Gym corpus. The accompanying review contract requires safe standardization or an explicit source-specific rejection or deferral.
Record the two-row review of 29 NeMo Gym task corpora and standardize the code and instruction-following rows whose private verifier contracts can be preserved. Hub-backed imports retain the pinned revision, split, and sampled offset without exposing source verifier data to the model.\n\nAction-prediction rows that depend on private tool results or authentication state stay deferred, alongside sources without a supported verifier or environment contract.
Generalize the NeMo Workplace importer for two pinned Hub rows while retaining private source provenance and the shared seeded provider contract. Export the rows with provider-tool Harbor bindings and update the sweep ledger.\n\nCalendar and SWE rows remain out of scope because their source contracts do not verify all public task requirements or provide a reproducible workspace oracle.
Add eleven pinned answer-only NeMo rows with shared MCQA, symbolic-math, and reference-answer contracts. Public prompts omit source delivery and grading language; source rows, answers, references, and judge policy remain private. The build exports 44 Chat/NoEnvironment lowerings across plain text, boxed LaTex, JSON, and XML. The NeMo sweep ledger now records 17 accepted rows.
Run representative TaskCompendium packages through Harbor's native Trial lifecycle and select them with the harbor_conformance marker. Update stale test exports to separate semantic bindings from launch configuration. Malformed provider submissions now record extraction errors. Unexpected verifier failures record infra_error without a reward. The selector covers Docker, ShellSim, final-state, native-action, and multi-step provider cases.
Define the current semantic schema, public rendering, Harbor lowering, and execution-artifact ownership. Describe the Harbor-backed SkyRL path, a later direct-chat generator, and planned MCP, browser, provider, and interactive-environment extensions without presenting them as supported interfaces.
TaskCompendium now serializes TaskSpec as its only semantic record. Harbor lowerings render model-visible instructions and agent resources directly, retain result tags in their manifest, and no longer write a task.json projection. Schema 0.8 removes the public Task records and legacy schemas, simplifies Harbor tool bindings, and rebuilds the checked-in sample artifacts. No-tool lowerings no longer create a filesystem input directory.
Document the target binding as the target-owned compatibility contract in a lowering. It defines the environment, exposed tools, and target setup while leaving task semantics and rollout policy in their respective layers.
Define sparse semantic coverage labels and a copy-ready labeling prompt. The guide separates TaskSpec semantics from runtime, verification, and lowering result encoding, and calibrates difficulty to Snowball’s roughly 2B active model.
Make the Luna prompt self-contained by including the standard semantic labels and their selection rules.
Apply sparse semantic coverage labels to the 47 POC and six first-wave specifications. The reviewed labels identify meaningful workspace and provider state and repository context while retaining existing label mappings.
Review the static and provider-backed NeMo specifications with the coverage taxonomy, including persistent provider state for Workplace rows.
Link the TaskCompendium documentation to the schema 0.8 Hugging Face publication and its current semantic and lowering counts.
Tell labelers to use dotted domain subtags for recurring functionality, such as javascript.react and calculus.integration, while avoiding one-off prompt-specific labels.
Replace the coverage domain namespace with sparse subject tags, permit dotted recurring subtags, and regenerate the checked-in semantic samples.
Allow artifact and context tags to use normalized MIME types, and migrate the structured-document examples to canonical MIME values.
Store only the most specific subject tag, use dotted-prefix grouping for parent facets, and permit sparse sibling subtags when both are material.
# Conflicts: # config/external/MarinSkyRL/uv.lock # lib/marin/src/marin/external_dependencies.py
dlwh
added
agent-generated
Contributor
IaC preview
marinLogged in to runnervmlun5p as runner (gs://marin-iac-state)
Previewing update (marin):
@ previewing update....
pulumi:pulumi:Stack: (same)
[urn=urn:pulumi:marin::marin-iac::pulumi:pulumi:Stack::marin-iac-marin]
@ previewing update....
! ~ gcp:iap/webBackendServiceIamBinding:WebBackendServiceIamBinding: (update)
[id=projects/hai-gcp-models/iap_web/compute/services/iris-marin-be/roles/iap.httpsResourceAccessor]
[urn=urn:pulumi:marin::marin-iac::marin:gcp:GcpIam$gcp:iap/webBackendServiceIamBinding:WebBackendServiceIamBinding::iap-backend-service-iris-marin-be-roles-iap-httpsresourceaccessor]
[provider=urn:pulumi:marin::marin-iac::pulumi:providers:gcp::gcp::8dde97ef-6668-4e99-9612-7f94be24d979]
! ~ members: [
- - [18]: "user:[redacted]"
]
Resources:
! ~ 1 to update
250 unchanged |
Run mixed lowerings through the pinned TrajectoryRunner interface with per-task launches. Archive semantic failures with absent rewards and reject numeric batches containing them. The consumer example reconstructs tokens from messages and tool schemas. Production runner registration and native mini-SWE trace conversion remain follow-up work.
Collaborator
|
The rough idea sounds good to me!
"""
|
Expose the mixed Harbor trajectory runner as installable package code. Model-caused extraction failures retain their null semantic rewards while entering numeric training as zero; verifier and infrastructure failures still reject the batch.
Write each semantic attempt batch to an optional fsspec destination before trainer admission. This keeps null extraction and infrastructure outcomes available when local rollout directories disappear.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TaskCompendium separates task semantics and private verification from submission formatting, environment bindings, and harness selection. This lets agentic and nonagentic data share a representation while preserving multiple compatible lowerings.
Please review the design overview. The specification has the field definitions and extension paths.
This draft includes the implementation spike and Harbor integration. A focused implementation review fixed compatibility, conversation retention, runtime cleanup, and grading failures. Broader code and source-task review is still pending. The review requested here is for the design; training, direct-chat lowering, MCP, browser, and reactive user simulators remain follow-up work.
The published dataset spike contains semantic specifications and Harbor lowerings. The boundary fixes pass 317 safe TaskCompendium tests, 220 affected Marin tests, and changed-file pre-commit. Live Docker validation remains pending.