[Architecture]: Massive code duplication creates merge conflict surface and policy drift

[morluto]

Summary

Jacobian has three major code duplication zones that together produce the largest merge conflict surface in the repository, drive inconsistent policy enforcement, and create a maintenance burden far exceeding the duplicated code's value.

Root cause

1. verifier_support.py duplicated 233 times (#885)

• 233 physical copies of the same 466-line file across benchmark datasets
• Only 9 unique content hashes — 194 are byte-identical copies
• 114,619 total lines of duplicated code
• Appears in 394 merge conflict resolutions — the Adopt the v0.1 generic verification-kernel contract #1 conflict file
• One merge commit touched 220 copies in a single merge
• Duplication enforced by Harbor's separate-verifier build context (no symlinks, no parent COPY)

2. ~30,000 lines of duplicated verifier and spike code (#717)

• Benchmark verifier code is hand-duplicated across dataset families
• Provider feasibility spike utilities duplicated across multiple verifiers
• Each copy can drift independently, creating policy inconsistencies

3. Duplicated capability helper code across domain contracts (#695, #696, #699)

• Polynomial capability helpers duplicated, violating kernel-separation rules
• Rational digit-bound validation duplicated across 5 contract modules
• Hand-rolled linear algebra and combinatorial algorithms with maintained backends available

Evidence

Duplication	Scale	Conflict rate
verifier_support.py	233 copies, 114K lines	394 conflict resolutions
Verifier + spike code	~30,000 lines	High merge conflict
Polynomial helpers	5+ modules	Moderate
Rational digit-bound	5 contract modules	Low

Impact

• Adopt the v0.1 generic verification-kernel contract #1 merge conflict source: 394 conflict resolutions involving verifier_support.py
• Repository bloat: ~114K lines of duplicated code
• Slow PR review: 200+ file diffs in a single logical change
• Policy drift: Each copy can diverge, creating inconsistent verifier behavior
• Merge conflict amplification: N concurrent benchmark PRs all conflict against the same infrastructure change

Anti-patterns

• Copy-paste reuse: Duplicating shared code instead of sharing it through a package or generated artifact
• Enforced duplication: Validation gate requires physical copies, forbidding symlinks or shared imports
• Drift-prone duplication: Each copy can independently diverge, creating policy inconsistencies
• Hand-rolled reimplementations: Duplicating algorithms that maintained backends already provide

Best practices

• Single source of truth: One human-edited source deterministically generates task-local runtime copies
• Generated artifacts, not committed copies: CI verifies generated output matches source; it does not silently rewrite files
• Shared packages: Extract shared code into a package that is vendored or available hermetically inside the build context
• Content-addressable evidence: Use content-addressable evidence where the checksum is derived from the evidence itself, not from a pre-computed baseline
• Reuse maintained backends: Do not reimplement linear algebra, combinatorics, or graph algorithms when SymPy, NetworkX, or other maintained backends provide them

Recommended architecture

1. Short-term: Create a sync script that automatically propagates code changes to all verifier_support.py copies
2. Medium-term: Investigate whether Docker multi-stage builds or build-time ADD <url> can fetch shared verifier code from a content-addressable location
3. Long-term: Create a benchmark scaffolding tool that generates task directories from a template, stamping the correct verifier_support.py at generation time
4. For capability helpers: Extract shared validation and conversion logic into domain-owned utility modules with explicit __all__ exports
5. For reimplemented algorithms: Replace hand-rolled implementations with calls to maintained backends (SymPy, NetworkX, etc.)

Related

• 233 copies of verifier_support.py create massive merge conflict surface #885 (233 copies of verifier_support.py)
• [Tech debt]: ~30,000 lines of duplicated verifier and spike code across benchmark datasets #717 (~30K lines of duplicated verifier/spike code)
• [Tech debt]: Stop committing 26 generated verifier copies — generate at build time or share #728 (stop committing 26 generated verifier copies — generate at build time)
• [Tech debt]: Extract verifier_support.py into a shared package across dataset families #729 (extract verifier_support.py into shared package)
• [Tech debt]: Duplicated polynomial capability helpers violate AGENTS.md kernel-sharing rule #695 (duplicated polynomial capability helpers)
• [Tech debt]: Duplicated rational digit-bound validation across 5 contract modules #696 (duplicated rational digit-bound validation)
• [Tech debt]: Hand-rolled linear algebra and combinatorial algorithms with maintained-library equivalents #699 (hand-rolled linear algebra with maintained backends available)
• 133 verifier checksum refresh commits indicate tight coupling between benchmarks and verification #891 (133 checksum refresh commits — consequence of duplication)

[morluto]

Closing as not planned because “duplication” here combines unrelated causes and proposes incompatible remedies:

• Harbor's physical task-local copies and verifier-policy drift are owned by [Architecture]: Make Harbor verifier policy manifest-driven and generate hermetic task-local runtimes #1011, which preserves the hermetic build-context requirement and uses deterministic generation—not runtime ADD <url>.
• Feature-level domain/checker collision is owned by [Superseded]: Colocate feature code where useful; do not add CapabilitySlice manifests or registries #1056 through typed vertical slices.
• Exact-rational validation duplication is owned by [Design]: Make integer ratios the exact-rational interoperability boundary #768.
• Individual maintained-backend substitution decisions remain domain-specific and cannot be inferred merely from duplicate line counts.

A single duplication ticket would obscure the different trust and packaging constraints.

[morluto]

2026-08-10 handrolling audit map

A fresh current-main pass focused specifically on infrastructure Jacobian appears to own unnecessarily. The important result is not “replace custom code with frameworks everywhere”; several existing custom layers are justified, and several tempting replacements would add more complexity than they remove.

The current systemic map is:

Hand-rolled area	Current owner / disposition	Simpler boundary
In-process atomic capability request schemas + manual dict conversions	#1107	Pydantic/TypeAdapter as the runtime type and JSON-Schema source of truth; keep raw jsonschema only for external/schema-only contracts
Clean-room checker copies of producer wire grammar	#1110	generated passive checker contract bundles; retain independently implemented mathematics
Artifact provenance inferred by recursively scanning arbitrary output strings	#1108	explicit domain-owned typed artifact references + authoritative verification-record/manifests
Product + repository subprocess engines	#1099	one neutral structured process owner with thin policy adapters
/proc parsing, descendant reconstruction, RSS/PSS observation inside process engine	#1109, child of #1099	psutil for generic process-tree/memory introspection; retain killpg, hard limits, and security policy
Worker JSON envelope parsing	#647 / #1052 / #1102	one strict decoder + typed shared worker envelope; no per-worker key-set parser
Capability discovery tokenizer/ranker	#1059	versioned hybrid retrieval/index instead of repeated synonym/tag patches
Python static architecture/policy scanners	#1101	declarative import/dependency contracts plus a narrow residual custom-policy layer
Lean text/token/front-end scanners	#1103–#1105	Lean-owned parser/elaboration artifacts where semantics matter; keep only bounded protocol framing locally
Source-only Python importer, package walker, implementation digest protocol	#1112	verified installed distributions + PyPA entry points + fresh worker import
Fully trusted in-process capability adapter extension authority	#1106	explicitly classify as deployment code or replace with isolated least-authority plugin protocol
module:attribute discovery for any retained trusted in-process extension	#1113, conditional child of #1106	PyPA entry-point group and owning distribution metadata; do not duplicate #1112's isolated plugin/checker trust path
Managed agent skill as handwritten shadow contract	#1058	generate bounded skill/reference projection from canonical installed contracts
Host-specific trajectory/event scraping	#1087	one typed normalized trajectory/event model with surface adapters
Benchmark verifier-support physical duplication/policy drift	#1011 / #1030 and children	one declarative verifier protocol source, deterministic hermetic task-local generated runtime
Hand-rolled mathematical algorithms with maintained backend equivalents	#699, closed completed	prior identified RREF/Bareiss/union-find/cartesian-product cases were removed/replaced; current search no longer finds those named implementations

Explicit non-issues from this pass

I do not recommend opening generic replacement issues for these:

1. JSON Schema validation — capability_validation.py and schema_validation.py already delegate standards semantics to jsonschema.Draft202012Validator; the remaining problem is duplicated schema authorship ([Architecture]: Make typed request/result models the source of truth for atomic service capability schemas #1107/[Architecture]: Stop reimplementing producer wire contracts inside clean-room checkers #1110), not a home-grown validator engine.
2. Canonical JSON — canonical.py already delegates generic JSON canonicalization to RFC 8785. Jacobian's extra exact-integer/rational guards are mathematical wire semantics, not a reason to swap in another serialization framework.
3. SQLite persistence/migrations — the current direct sqlite3 migration layer is small and explicit. Introducing SQLAlchemy/Alembic solely to remove a few migration helpers would likely increase the trusted/operational surface. [Superseded]: Do not build mixed-version writer-fencing infrastructure for deleted core state #1111 owns the actual mixed-version writer-fencing problem.
4. Process containment — psutil must not replace POSIX process groups, killpg, RLIMIT/prlimit, or fail-closed ownership. [Deletion]: Remove unused full-/proc process-tree RSS polling from bounded_process #1109 replaces observation plumbing only.
5. Checker mathematical replay — generated/shared wire contracts must not turn an independent checker into a producer import. [Architecture]: Stop reimplementing producer wire contracts inside clean-room checkers #1110 deliberately preserves independent mathematical implementations.

Architectural rule that falls out of the audit

A useful repository-wide ratchet is:

Jacobian should own mathematical semantics, trust/authority policy, exact resource bounds, and evidence binding. Prefer maintained standards/libraries for generic packaging, parsing, schema generation/validation, process introspection, and algorithm implementations unless a documented Jacobian-specific invariant makes the maintained primitive insufficient.

For any remaining hand-written replacement of a maintained primitive, require a short source comment/test stating the missing property (for example checker independence, process-group containment, exact canonical representation, or hermetic build isolation). That makes exceptions intentional rather than accidental infrastructure forks.

[morluto]

2026-08-10 handrolling/duplication audit update

Static audit baseline: main at 7d277e169c809455d1a171b3a96aaa0d915471b7.

I completed a producer/checker/tooling sweep and separated five categories that were previously conflated under “hand-rolled code.” The new canonical children are:

Child	Root defect	Maintained/simpler direction
#1110	Clean-room checkers rewrite producer field sets, scalar grammars, bounds, version literals, and canonicalization before doing independent mathematics	Generate a passive, content-addressed checker contract bundle; retain independent mathematical replay
#1119	PR #983 implements graph6 decoding in both producer and checker despite NetworkX already providing the codec	Producer uses nx.from_graph6_bytes plus a canonical round trip and explicit order bound; checker may remain independent
#1123	PR #907 introduces a private GF(p^m) arithmetic engine in the first finite-field operation and a second engine in the checker	Add an exact finite-field domain/provider; evaluate pinned galois first, retain a separate checker path
#1126	Topology producer uses SymPy DomainMatrix for rank/RREF but also owns _IncrementalVectorBasis	Use DomainMatrix nullspace and pivot selection for image and quotient bases while preserving original-vector provenance
#1127	Certified-SNF support delegates decomposition to SymPy but hand-writes exact Gauss–Jordan inversion for already-unimodular transformations	Use DomainMatrix.inv_den(), require reduced denominator 1, and verify both inverse identities

Additional systemic owners found during the same audit:

• [Superseded]: Keep architecture checks small and direct; do not add a rule engine or policy-manifest system #1047 — the architecture checker itself is a monolithic hand-built rule/configuration/dispatch hub; decompose it into statically registered rules and strict owner-scoped policy manifests.
• [Superseded]: Colocate feature code where useful; do not add CapabilitySlice manifests or registries #1056 — capability/checker additions are forced through shared domain/checker hubs; package them as independent vertical slices.
• [Architecture]: Worker protocol parsing hand-rolls JSON validation without hashability, duplicate-key, or type guards #1052 / [Architecture]: Trust boundary validation gaps for plugin/checker responses — no schema enforcement, os.environ mutation #1063 / [Protocol][Security]: Reject duplicate JSON object names and non-standard numeric values before MCP/Pydantic decoding #1102 — worker/plugin/raw-JSON protocol parsing and strict trust-boundary decoding.
• [Architecture][Deletion]: Consolidate each external tool’s duplicated pins using existing native files; do not build a provider-manifest system #1075 — provider version/source/feature probes should be generated from one content-addressed provider contract rather than hand-maintained output scrapers and constants.
• [Architecture][Security]: Replace hand-written Lean source scanning with Lean-native term parsing and structured axiom inspection #1104 — Lean source policy must use Lean-native parsing rather than Python regex/token scanners.
• [Architecture]: Make Harbor verifier policy manifest-driven and generate hermetic task-local runtimes #1011 / [Architecture]: Make Harbor verifier policy manifest-driven and generate hermetic task-local runtimes #1039 — Harbor policy should be declarative and deterministically compiled into hermetic task-local runtimes rather than independently hand-coded per verifier.
• [Superseded]: Do not replace DomainBundle with provider inventories and component graphs #1010 / [Architecture]: Model fallible resource shutdown as an explicit terminal state machine #1021 — provider, checker, tenant, worker, and shutdown lifecycle state machines need explicit component/resource ownership rather than service-local copies.

Cases deliberately not filed as handrolling defects

The audit rejected several superficially plausible replacements because they would weaken correctness or independence:

• Exact spanning-tree count: NetworkX 3.6.1's number_of_spanning_trees() uses NumPy's floating determinant. Jacobian's exact SymPy cofactor is therefore justified, not needless reinvention.
• Planar geometry: the producer already delegates lines, projections, circles, polygon area/containment, and convex hull to SymPy. The remaining segment/contact and simple-polygon loops encode Jacobian's exact witness/contact contract and are bounded domain logic.
• Finite posets: the producer already uses NetworkX transitive closure/reduction, topological sorting, and bipartite maximum matching. The local bounded linear-extension DP and Möbius recurrence are the mathematical operation itself.
• Canonical JSON and large integers: canonical.py already uses RFC 8785 and strict stdlib decoder hooks. Its chunked decimal conversion exists specifically to avoid Python's process-global integer-string digit limit; no equivalent zero-cost drop-in was found.
• Independent checker mathematics: small stdlib replays can be valuable implementation diversity. The defect is duplicating passive serialization/contracts ([Architecture]: Stop reimplementing producer wire contracts inside clean-room checkers #1110), not the mere existence of a second mathematical algorithm.
• Held-out tar extraction: the tooling already combines explicit path/type checks with Python's tarfile filter="data"; replacing it with a less explicit helper would not improve the boundary.

Disposition of the old #699 audit

#699 was directionally correct but closed after partial remediation. Its unresolved producer-side items now have focused owners (#1126 and #1127), while checker schema duplication is owned by #1110. Future closure should verify each concrete reproduction rather than treating one small replacement as completion of the umbrella.

This comment is the current root-cause map for the handrolling portion of #1030. The priority order I recommend is:

1. [Architecture]: Stop reimplementing producer wire contracts inside clean-room checkers #1110, because every new verifier otherwise multiplies the duplicated contract surface;
2. [Tech debt]: Replace operation-local finite-field arithmetic with a maintained finite-field provider #1123, before finite-field arithmetic becomes a private dependency of more capabilities;
3. [Superseded]: Keep architecture checks small and direct; do not add a rule engine or policy-manifest system #1047 and [Superseded]: Colocate feature code where useful; do not add CapabilitySlice manifests or registries #1056, to stop shared-hub and policy-registry recurrence;
4. [Tech debt]: Use NetworkX’s maintained graph6 codec instead of duplicating the parser in the producer #1119, [Tech debt]: Remove topology’s private finite-field basis engine and use DomainMatrix pivots #1126, and [Tech debt]: Replace hand-written unimodular inversion with DomainMatrix.inv_den #1127 as bounded, low-ambiguity removals;
5. the broader provider/lifecycle/Harbor roots according to their existing plans.

[morluto]

Fresh conflict evidence from the current-main reconciliation of PRs #1300 (#1300), #1301 (#1301), and #1311 (#1311) against origin/main at da58870:

• feat(probability): add verified finite-table mutual information #1300 and feat(number-theory): add verified finite abelian factorizations #1301 both conflicted in src/jacobian/checker_operations.py and src/jacobian/portfolio/core_installation.py.
• feat(number-theory): add verified finite abelian factorizations #1301 additionally conflicted in src/jacobian/exact_domain_checkers.py, src/jacobian/domains/number_theory/checkers.py, and tests/component/checkers/test_exact_domain_checker_installation.py.
• The conflicts mixed shared runtime/install changes with feature declarations, so resolution required retaining current-main infrastructure and reapplying only feature-local additions.

This is a current, smaller-scale instance of the shared-hub duplication and policy-drift pattern tracked here. The PRs are now reconciled, but the same hubs remain collision-prone for the next capability slice.

[morluto]

Current-main triage (ef298fc245c9d0a0a8156a2b64d048f8a90aaa87): the product-side capability/plugin/artifact duplication described here was removed by the stateless direct-tool cutover (#1513/#1514). The only substantial surviving duplication family is Harbor task-local verifier code and policy, already owned by #631 and #1011. Closing this umbrella as duplicate so it does not become a mandate to build a new shared framework around deleted systems.