Roadmap: close the remaining gaps

[avayaaggarwal]

Summary

Erasus already covers a wide span of unlearning methods, but to be the clear state-of-the-art package for unlearning research it needs a tighter combination of reproducibility, benchmark parity, scalable training support, and stronger packaging ergonomics.

This issue tracks the highest-leverage work needed to make Erasus the default open-source package for unlearning experiments rather than just a broad collection of implementations.

Why this matters

Researchers and practitioners evaluate unlearning frameworks on four things:

• breadth of methods
• credibility of benchmark results
• ease of reproducing real papers on real models
• reliability of the package in messy real environments

Erasus is already strong on breadth. The next gap is turning that breadth into reproducible, benchmark-backed, easy-to-run depth.

Roadmap

1. Reproducible real-benchmark harnesses

• Standardize real benchmark entrypoints for TOFU, MUSE, WMDP, and lm-eval tasks behind one common CLI contract
• Add config presets for paper-faithful runs on GPT-2, Zephyr-7B, and at least one PEFT-based 7B path
• Save machine-readable result bundles with metrics, configs, seed, git SHA, model revision, and dataset revision
• Add a benchmark manifest schema so leaderboard outputs are reproducible and comparable across runs

2. Baseline parity with major unlearning papers

• Audit implemented strategies against the latest LLM unlearning papers and document missing training details or approximations
• Add paper-parity benchmark scripts for NPO, SimNPO, RMU, FLAT, UNDIAL, activation steering, DExperts, and delta-unlearning
• Publish baseline result tables in-repo for at least TOFU, WMDP, and post-unlearning capability tasks
• Add a clear "implemented", "approximated", and "paper-faithful" status tag in strategy docs

3. Scalable training and inference support

• Add first-class PEFT support across unlearners with LoRA/QLoRA adapters
• Add accelerate integration for multi-GPU and gradient accumulation workflows
• Support 4-bit / 8-bit loading paths for both unlearning and post-unlearning evaluation
• Add checkpoint resume support for long-running benchmark jobs

4. Stronger verification and privacy evaluation

• Expand prompt-extraction suites into reusable attack packs with jailbreak, multilingual, paraphrase, and long-context retrieval modes
• Add calibration and confidence reports to MIA outputs, not just scalar attack scores
• Add corpus-level memorization reports for forget sets with attribution to exact samples/documents
• Add a unified privacy report spanning MIA, memorization, extraction, relearning, and RAG leakage

5. Dataset and deletion workflow quality

• Add canonical dataset wrappers for popular unlearning datasets with version pinning and preprocessing provenance
• Add machine-readable forget request manifests for sample-, user-, concept-, and document-level deletion settings
• Add validation utilities that detect overlap/leakage between forget, retain, and eval splits
• Add documentation for supported deletion granularities and expected benchmark mappings

6. Package ergonomics and modularity

• Introduce optional dependency extras by surface area such as llm, vision, audio, benchmarks, and ui
• Make package-level imports consistently lazy so optional integrations do not make import erasus brittle
• Add a plugin/discovery mechanism for third-party strategies, selectors, metrics, and benchmark adapters
• Standardize structured result objects and serialization across unlearning, evaluation, verification, and benchmarks

7. Quality gates and developer trust

• Re-enable the full main test suite in CI and keep it green with coverage enforcement
• Add smoke tests for package import surfaces under reduced optional-dependency environments
• Add benchmark regression tests that check output schema, not just execution
• Add docs pages mapping claims to tests, benchmarks, and implemented modules

Suggested acceptance criteria

• A new user can reproduce at least one real TOFU, MUSE, and WMDP run from documented commands without editing source files
• Strategy docs clearly state whether each implementation is approximate or paper-faithful
• Result bundles are reproducible across seeds and include all provenance needed for comparison
• import erasus and core workflows succeed cleanly in minimal environments with optional features gated behind extras
• Verification outputs are consolidated into a single report that is suitable for papers and model-card publication

Nice-to-have follow-ups

• Public benchmark artifact hosting for result bundles and plots
• Example notebooks for end-to-end LLM unlearning on real models
• Automatic model-card generation from benchmark + verification reports

[OnePunchMonk]

8. Hybrid & Agentic Unlearning (Advanced Modalities)

• Cross-Modal Consistency: Implement alignment checks for VLMs to ensure unlearning a text concept (e.g., a specific person) also suppresses its visual representation in the image encoder.
• RAG-Sync Utilities: Add hooks to synchronize model unlearning with Vector Database deletions, preventing "ghost hallucinations" where the model recalls data no longer in the retrieval context.
• System-Prompt Resilience: Evaluate if unlearned concepts can be "resurrected" via specific system-prompt injections or persona-shifting.
• Instruction-Tuning Protection: Add specialized loss functions to ensure the model’s instruction-following capabilities remain intact post-unlearning.

9. Utility Guardrails & Drift Control

• Reference-Model Monitoring: Integrated KL-divergence trackers to monitor the semantic drift between the "unlearning" model and a frozen "reference" model in real-time.
• Automated Early-Stopping: Implement a "Utility-Floor" trigger that halts unlearning if general benchmarks (e.g., MMLU or GSM8K) drop below a user-defined percentage.
• Activation Steering / Abliteration: Add "training-free" unlearning paths using weight orthogonalization to suppress concepts without full gradient updates (SOTA for efficiency).
• Second-Order Optimizers: Support Woodbury Matrix Identity or Hessian-Free approximations for more precise "influence-based" data removal.

10. Legal, Compliance & Auditability

• Certificates of Erasure: Auto-generate signed JSON/PDF reports summarizing the "forgetting" efficacy (MIA scores, p-values, and entropy deltas) for legal audit trails.
• Differential Privacy (DP) Integration: Add support for $\epsilon$-differential privacy bounds to provide "Certified Removal" guarantees for sensitive user data.
• Efficiency Telemetry: Capture and report FLOPs, peak VRAM, and wall-clock time in every benchmark bundle to measure the "cost-to-forget."
• Hugging Face Hub Integration: Add a one-click utility to push "Unlearned Adapters" to the Hub with automated metadata tags indicating what was removed.

11. Documentation & Community DX

• The "Unlearning Playbook": A CLI-guided wizard (erasus-wizard) that recommends strategies and hyperparameters based on the user's compute budget and data type.
• Visual Impact Reports: Generate automated "Before vs. After" heatmaps of model activations to visualize which neurons/layers were most affected by the unlearning process.
• Glossary & Paper Mapping: A comprehensive cross-reference in the docs linking Erasus modules directly to their foundational ArXiv papers.

[OnePunchMonk]

Roadmap: Towards a SOTA Unlearning Framework (Erasus 2026)

Summary

Erasus already covers a wide span of unlearning methods, but to be the clear state-of-the-art (SOTA) package for unlearning research, it needs a tighter combination of reproducibility, benchmark parity, scalable training support, and stronger packaging ergonomics.

This roadmap tracks the highest-leverage work needed to transform Erasus into the default open-source standard for unlearning experiments, moving it from a broad collection of implementations to a rigorous research and production framework.

Why This Matters

As of 2026, researchers and practitioners evaluate unlearning frameworks on four specific pillars:

1. Breadth of methods: Native support for LLMs, VLMs, and Diffusion.
2. Credibility of benchmarks: Trustworthy, verifiable, and standardized metrics.
3. Reproducibility: The ability to replicate paper-level results on real models with zero friction.
4. Reliability: Robust performance in messy, real-world hardware environments.

---

Roadmap

1. Reproducible Real-Benchmark Harnesses

• Standardize real benchmark entrypoints for TOFU, MUSE, WMDP, and lm-eval tasks behind a unified CLI contract.
• Add config presets for paper-faithful runs on GPT-2, Zephyr-7B, and at least one PEFT-based 7B path.
• Implement machine-readable result bundles that save metrics, configs, seed, git SHA, and model/dataset revisions.
• Add a benchmark manifest schema to ensure leaderboard outputs are reproducible across different compute environments.

2. Baseline Parity with Major Unlearning Papers

• Audit all current strategies against the latest LLM unlearning papers; document any missing training details or approximations.
• Add paper-parity benchmark scripts for NPO, SimNPO, RMU, FLAT, UNDIAL, activation steering, and delta-unlearning.
• Publish baseline result tables in-repo for at least TOFU and WMDP tasks.
• Implement status tags in strategy documentation: [Implemented], [Approximated], or [Paper-Faithful].

3. Scalable Training and Inference Support

• First-class PEFT support: Full integration across all unlearners with LoRA/QLoRA adapters.
• Distributed Workflows: Add accelerate integration for multi-GPU setups and gradient accumulation.
• Precision Loading: Support 4-bit / 8-bit loading paths for both unlearning and post-unlearning evaluation.
• Add checkpoint resume support for long-running benchmark jobs.

4. Stronger Verification and Privacy Evaluation

• Attack Packs: Expand prompt-extraction suites into reusable packs with jailbreak, multilingual, and long-context retrieval modes.
• Advanced MIA: Add calibration and confidence reports to Membership Inference Attack outputs (e.g., LiRA), moving beyond simple scalar scores.
• Memorization Reports: Add corpus-level reports for forget sets with attribution to exact samples/documents.
• Unified Privacy Report: Generate a consolidated report spanning MIA, memorization, extraction, relearning, and RAG leakage.

5. Hybrid & Agentic Unlearning (2026 Frontiers)

• Cross-Modal Consistency: Ensure unlearning a text concept in VLMs also suppresses its visual representation in the vision encoder.
• RAG-Sync Utilities: Add hooks to synchronize model unlearning with Vector Database deletions to prevent "ghost hallucinations."
• System-Prompt Resilience: Evaluate if unlearned concepts can be "resurrected" via specific system-prompt injections or persona-shifting.
• Utility Guardrails: Integrated KL-divergence trackers to monitor semantic drift against a frozen "reference" model in real-time.

6. Package Ergonomics and Modularity

• Introduce optional dependency extras: pip install erasus[llm,vision,benchmarks].
• Implement lazy imports to ensure the core package remains lightweight and non-brittle.
• Add a plugin/discovery mechanism for community-contributed strategies, selectors, and metrics.
• The "Unlearning Playbook": A CLI-guided wizard (erasus-wizard) to recommend strategies based on the user's compute budget and data type.

7. Quality Gates and Developer Trust

• Re-enable the full main test suite in CI with 90%+ coverage enforcement.
• Add smoke tests for package import surfaces under minimal dependency environments.
• Add benchmark regression tests that validate output schema consistency, not just execution.

---

Suggested Acceptance Criteria

• One-Click Reproducibility: A new user can reproduce a standard TOFU or WMDP run from documented commands without editing source files.
• Paper Fidelity: All strategy documentation explicitly states if an implementation is bit-for-bit faithful to the original paper.
• Utility Retention: Unlearning the "forget set" must result in < 2% degradation on general capability benchmarks (e.g., MMLU, HumanEval).
• Efficiency: The framework must support unlearning on a 7B model using a single consumer GPU (24GB VRAM) via 4-bit PEFT.
• Auditability: Every run produces a machine-readable "Verification Bundle" (MIA p-values, entropy metrics, and provenance) suitable for model-card publication.

Nice-to-have Follow-ups

• Certificates of Erasure: Auto-generated PDF summaries for legal/compliance audits.
• Public benchmark artifact hosting for result bundles and community plots.
• Automatic model-card generation from benchmark + verification reports.