feat: kernel-grounded sorry/axiom auditor (lake exe atlas_audit) + first CI - #2
Open
mvanhorn wants to merge 1 commit into
Open
feat: kernel-grounded sorry/axiom auditor (lake exe atlas_audit) + first CI#2mvanhorn wants to merge 1 commit into
mvanhorn wants to merge 1 commit into
Conversation
…+ CI Adds a standalone Lean executable that walks each Atlas declaration's transitive axiom dependencies and classifies it as proved (benign axioms only), sorry (reaches sorryAx), or other-axiom. Writes audit/proved.json plus a per-book summary. The auditor imports only Lean, so it compiles without building Mathlib or Atlas; the library is loaded at runtime. Also adds the repo's first GitHub Actions workflow: a fast classifier fixture test plus the full audit (Mathlib via cache, Atlas built, audit run, report uploaded as an artifact).
Contributor
|
Hi Matt! Thanks for the PR, it is a good point and a feature we agree we should have, let me check on the implementation. |
Author
|
Thanks @niketp03, glad it resonates, and no rush on my end. A couple of things that might help the review: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ATLAS now has a kernel-grounded way to tell which of its 46,203 declarations are actually proved.
lake exe atlas_auditwalks each Atlas declaration's transitive axiom dependencies and classifies it asproved(benign axioms only),sorry(reachessorryAx), orother-axiom, then writesaudit/proved.jsonplus a per-book summary. This PR also adds the repo's first GitHub Actions workflow.Why this matters
Each book's
report.jsoncarries aproof_integrityscore, but that score is produced by an LLM judge, not the Lean kernel. There is currently no in-repo way to ask the kernel which declarations are genuinely proved:sorrys live across 874 files inside the default-builtAtlaslibrary, with no machine-checkable boundary. TheUnprovedlean_libdeclared inlakefile.tomlhas no source files yet, so proved and unproved declarations are interleaved in the same target.atlas_auditreuses the sameLean.collectAxiomstraversal that#print axiomsuses, so its verdict is ground truth, and it surfaces any declaration whose axiom set is neither benign (propext,Classical.choice,Quot.sound) norsorryAxfor review.provedflag and google-deepmind/formal-conjectures emitshasSorryFreeProof. ATLAS had neither, and no CI to verify the library builds against the pinned Mathlib.Demo
Simulated Demo. A real full-corpus run builds ~484K lines of Lean, so the figures shown here are ATLAS's own published statistics used illustratively; the classifier logic itself is covered by the fixture test below.
Changes
Audit/Audit.leanis the auditor. It imports onlyLean, solake build atlas_auditcompiles without building Mathlib or Atlas; the library is loaded at runtime viawithImportModulesand walked module-by-module, restricted toAtlas.*modules.Audit/Test.leanis a self-contained fixture that asserts the classifier labels a genuinely-proved theorem and asorry-backed one correctly. It runs without building the corpus..github/workflows/audit.ymladds CI as two jobs: a fastclassifier-test(runs the fixture on every PR, no Mathlib build) andaudit(Mathlib fromlake exe cache get, builds Atlas, runs the auditor, uploadsproved.json).lakefile.tomldeclares theatlas_auditlean_exe.Testing
Audit/Audit.leanandAudit/Test.leancompile cleanly against the pinnedleanprover/lean4:v4.29.0: no errors, no warnings, 80-char / 2-space style.sorryAx; thesorrytheorem does.lake exe atlas_auditrun executes in theauditCI job and uploads the report.AI was used for assistance.