refactor(forks): nest forks/ under new spec/ subpackage (leanEthereum#788)

* refactor(forks): nest forks/ under new spec/ subpackage

Group per-fork consensus rules under a dedicated lean_spec.spec namespace
so the protocol specification has a clear home as more spec content
(SSZ, chain, validator, etc.) potentially moves alongside it.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(crypto): collect koalabear and poseidon under spec/crypto/

Move the two cryptographic primitive subspecs into the spec/ subpackage
created in the previous commit:

- subspecs/koalabear/field.py    -> spec/crypto/koalabear.py
- subspecs/poseidon1/{constants,permutation}.py -> spec/crypto/poseidon.py

Each primitive collapses into a single module: koalabear loses its
re-exporting __init__, and poseidon merges its round-constants table
with the permutation engine. The Poseidon1 class is renamed to Poseidon
(and Poseidon1Params -> PoseidonParams) since the file no longer carries
the variant number; PARAMS_16/PARAMS_24 keep their names.

All import sites in src/, tests/, and packages/testing/ are updated.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(crypto): collapse subspecs/ssz/ into spec/crypto/merkleization.py

Merge the four-file SSZ hashing subspec into a single module that lives
alongside the other primitives under spec/crypto/. The hash-tree-root
dispatch, the binary-tree merkleizer, the length-mix helper, and the
two chunk-width constants now all sit in one place; the cross-file
imports between hash.py, merkleization.py, and constants.py disappear.

All import sites in src/, tests/, and packages/testing/ are updated to
import from spec.crypto.merkleization.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(crypto): collapse zero-hash cache into a single expression

Replace the three-piece machinery (named max-depth constant, builder
function, module-level call) with one itertools.accumulate that folds
the recurrence h_{d+1} = sha256(h_d || h_d) over the all-zero seed.

The cache now covers depth 64 unconditionally, so _zero_tree_root drops
the past-cache fallback loop and becomes a direct table lookup. The two
tests that exercised the fallback are deleted with the dead code.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(crypto): bullet-format the zero-hash table and depth-mapping comments

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor: move xmss under spec/crypto, rename subspecs -> node

Two related reorgs that finish carving out spec/ vs runtime/:

- XMSS is a cryptographic primitive specification, structurally identical
  to the koalabear/poseidon/merkleization siblings, so move the entire
  subpackage to lean_spec.spec.crypto.xmss alongside them.

- Everything left under subspecs/ is reference-node runtime (api,
  networking, storage, sync, validator, observability, metrics, plus the
  chain/genesis fuzzy middle ground and the Node orchestrator itself).
  Rename it to lean_spec.node so the package name reflects what it is.

The old inner subspecs/node/ subpackage is dissolved in the same move:
its node.py and anchor.py lift up by one level so the new layout exposes
lean_spec.node.{Node, NodeConfig, anchor.*} directly instead of nesting
them inside lean_spec.node.node.*.

All import sites in src/, tests/, and packages/testing/ are updated, the
per-file ruff ignore in pyproject.toml moves with xmss, and the xmss
internal relative imports are repointed to the new neighbors.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(node): move snappy/ under node/

Snappy framing is a wire-format runtime concern used by gossip and
reqresp, not a piece of the protocol specification. Group it with the
other runtime services under lean_spec.node.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(ssz): carve SSZ primitives into spec/ssz/, lift base.py to lean_spec.base

The 8 SSZ-primitive modules under lean_spec.types (bitfields, boolean,
byte_arrays, collections, container, exceptions, ssz_base, uint) move to
lean_spec.spec.ssz alongside the other protocol-spec subpackages. What
stays under lean_spec.types is the domain layer: Slot, Checkpoint,
ValidatorIndex, SubnetId, AggregationBits, the RLP helpers, and the
participation bits — types built on top of SSZ rather than defining it.

base.py (CamelModel, StrictBaseModel) is not SSZ-specific and is used by
both SSZ models and non-SSZ Pydantic models. Moving it would create a
spec/ssz/ssz_base.py → types.base → types/__init__.py → spec/ssz cycle,
so it lifts one level up to lean_spec.base, sitting next to config.py
and log.py as generic Pydantic infrastructure.

All 135 SSZ-touching import sites are split where they mix SSZ and
domain symbols, and 21 sites importing StrictBaseModel/CamelModel are
repointed at lean_spec.base. No backward-compat re-exports.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(tests): mirror src/lean_spec/ layout under tests/

The unit test tree was lagging behind the recent source reorgs. Bring
tests/lean_spec/ into 1:1 correspondence with src/lean_spec/:

- tests/lean_spec/types/{test_bitfields,test_boolean,test_byte_arrays,
  test_collections,test_container,test_ssz_base,test_uint}.py
  -> tests/lean_spec/spec/ssz/
- tests/lean_spec/types/test_base.py -> tests/lean_spec/test_base.py
- tests/lean_spec/forks/ -> tests/lean_spec/spec/forks/
- tests/lean_spec/subspecs/{api,chain,genesis,metrics,networking,
  observability,storage,sync,validator}/ -> tests/lean_spec/node/
- tests/lean_spec/subspecs/node/{test_anchor,test_node}.py dissolved up
  into tests/lean_spec/node/
- tests/lean_spec/subspecs/koalabear/test_field.py
  -> tests/lean_spec/spec/crypto/test_koalabear.py
- tests/lean_spec/subspecs/poseidon1/test_permutation.py
  -> tests/lean_spec/spec/crypto/test_poseidon.py
- tests/lean_spec/subspecs/ssz/{test_hash,test_merkleization}.py
  -> tests/lean_spec/spec/crypto/
- tests/lean_spec/subspecs/xmss/ -> tests/lean_spec/spec/crypto/xmss/
- tests/lean_spec/subspecs/containers/test_attestation_aggregation.py
  -> tests/lean_spec/spec/forks/lstar/
- tests/lean_spec/snappy/ -> tests/lean_spec/node/snappy/
- tests/consensus/{lstar,devnet}/poseidon1 -> .../poseidon
- tests/lean_spec/subspecs/conftest.py -> tests/lean_spec/node/conftest.py

Justfile paths updated: codespell --skip for the snappy testdata
fixture, and the test-consensus recipe now points at the new layout.

Side fix: the node/__init__.py eager re-export of Node/NodeConfig was
turning any import of lean_spec.node.<anything> into a load of the
whole api -> networking -> reqresp.handler chain, which then tries to
import SignedBlock from lean_spec.spec.forks while spec.forks is
mid-init. The three call sites switch to the explicit module path
lean_spec.node.node, and the __init__.py drops the re-export.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

Author: tcoratger

.claude/rules/ssz-patterns.md

@@ -1,6 +1,6 @@
 ---
 paths:
-  - "src/lean_spec/forks/*/containers/**/*.py"
+  - "src/lean_spec/spec/forks/*/containers/**/*.py"
   - "src/lean_spec/types/**/*.py"
 ---
 
@@ -36,7 +36,7 @@ When creating SSZ types, follow these established patterns:
 Containers should be organized into modules with clear separation:
 
 ```
-src/lean_spec/forks/<fork>/containers/
+src/lean_spec/spec/forks/<fork>/containers/
 ├── state/
 │   ├── __init__.py      # Exports State and related types
 │   ├── state.py         # Main State container class

Justfile

@@ -40,7 +40,7 @@ typecheck *args:
 # Spell check source, tests, packages, and docs
 [group('quality')]
 spellcheck *args:
-    uv run --group lint codespell src tests packages docs README.md CLAUDE.md --skip="*.lock,*.svg,.git,__pycache__,.pytest_cache,tests/lean_spec/snappy/testdata" --ignore-words=.codespell-ignore-words.txt "$@"
+    uv run --group lint codespell src tests packages docs README.md CLAUDE.md --skip="*.lock,*.svg,.git,__pycache__,.pytest_cache,tests/lean_spec/node/snappy/testdata" --ignore-words=.codespell-ignore-words.txt "$@"
 
 # Verify markdown formatting in docs/
 [group('quality')]
@@ -78,7 +78,7 @@ test-cov-gate *args:
 # Run consensus-only unit tests (containers, forkchoice, networking)
 [group('tests')]
 test-consensus *args:
-    uv run --group test pytest -n auto --maxprocesses=10 --durations=10 --dist=worksteal tests/lean_spec/subspecs/containers tests/lean_spec/subspecs/forkchoice tests/lean_spec/subspecs/networking "$@"
+    uv run --group test pytest -n auto --maxprocesses=10 --durations=10 --dist=worksteal tests/lean_spec/spec/forks tests/lean_spec/node/networking "$@"
 
 # Canonical CI fixture run; contributors should use `uv run fill` directly.
 [group('tests'), private]

packages/testing/src/consensus_testing/forks/forks.py

@@ -2,7 +2,7 @@
 
 from framework.forks import BaseFork
 
-from lean_spec.forks.lstar.spec import LstarSpec
+from lean_spec.spec.forks.lstar.spec import LstarSpec
 
 
 class Lstar(BaseFork):

packages/testing/src/consensus_testing/genesis.py

@@ -1,16 +1,17 @@
 """Consensus layer pre-state generation."""
 
-from lean_spec.forks.lstar.containers import (
+from lean_spec.spec.crypto.merkleization import hash_tree_root
+from lean_spec.spec.forks.lstar.containers import (
     AggregatedAttestations,
     Block,
     BlockBody,
     State,
     Validator,
     Validators,
 )
-from lean_spec.forks.lstar.spec import LstarSpec
-from lean_spec.subspecs.ssz.hash import hash_tree_root
-from lean_spec.types import Bytes52, Slot, Uint64, ValidatorIndex
+from lean_spec.spec.forks.lstar.spec import LstarSpec
+from lean_spec.spec.ssz import Bytes52, Uint64
+from lean_spec.types import Slot, ValidatorIndex
 
 from .keys import XmssKeyManager
 

packages/testing/src/consensus_testing/keys.py

@@ -41,34 +41,30 @@
 from typing import ClassVar, Literal
 
 from lean_spec.config import LEAN_ENV
-from lean_spec.forks.lstar.containers import AggregatedAttestations, AttestationData
-from lean_spec.subspecs.koalabear import Fp
-from lean_spec.subspecs.ssz.hash import hash_tree_root
-from lean_spec.subspecs.xmss.aggregation import TypeOneMultiSignature
-from lean_spec.subspecs.xmss.constants import TARGET_CONFIG
-from lean_spec.subspecs.xmss.containers import (
+from lean_spec.spec.crypto.koalabear import Fp
+from lean_spec.spec.crypto.merkleization import hash_tree_root
+from lean_spec.spec.crypto.xmss.aggregation import TypeOneMultiSignature
+from lean_spec.spec.crypto.xmss.constants import TARGET_CONFIG
+from lean_spec.spec.crypto.xmss.containers import (
     PublicKey,
     SecretKey,
     Signature,
     ValidatorKeyPair,
 )
-from lean_spec.subspecs.xmss.interface import (
+from lean_spec.spec.crypto.xmss.interface import (
     PROD_SIGNATURE_SCHEME,
     TEST_SIGNATURE_SCHEME,
     GeneralizedXmssScheme,
 )
-from lean_spec.subspecs.xmss.types import (
+from lean_spec.spec.crypto.xmss.types import (
     HashDigestList,
     HashDigestVector,
     HashTreeOpening,
     Randomness,
 )
-from lean_spec.types import (
-    Bytes32,
-    Slot,
-    Uint64,
-    ValidatorIndex,
-)
+from lean_spec.spec.forks.lstar.containers import AggregatedAttestations, AttestationData
+from lean_spec.spec.ssz import Bytes32, Uint64
+from lean_spec.types import Slot, ValidatorIndex
 
 KeyRole = Literal["attestation", "proposal"]
 """Discriminator for which signing role's key to load from a validator key pair."""

packages/testing/src/consensus_testing/test_fixtures/api_endpoint.py

@@ -3,11 +3,12 @@
 from collections.abc import Callable
 from typing import Any, ClassVar
 
-from lean_spec.forks.lstar import Store
-from lean_spec.forks.lstar.containers import AggregatedAttestations, Block, BlockBody, State
-from lean_spec.forks.lstar.spec import LstarSpec
-from lean_spec.subspecs.ssz.hash import hash_tree_root
-from lean_spec.types import Bytes32, Slot, Uint64, ValidatorIndex
+from lean_spec.spec.crypto.merkleization import hash_tree_root
+from lean_spec.spec.forks.lstar import Store
+from lean_spec.spec.forks.lstar.containers import AggregatedAttestations, Block, BlockBody, State
+from lean_spec.spec.forks.lstar.spec import LstarSpec
+from lean_spec.spec.ssz import Bytes32, Uint64
+from lean_spec.types import Slot, ValidatorIndex
 
 from ..genesis import build_anchor, generate_pre_state
 from .base import BaseConsensusFixture
@@ -150,7 +151,7 @@ def _metrics_response(_store: Store, _fixture: "ApiEndpointTest") -> dict[str, A
     so the fixture pins only the stable contract: status, content-type,
     and the full list of metric names clients must expose.
     """
-    from lean_spec.subspecs.metrics.registry import registry as metrics_registry
+    from lean_spec.node.metrics.registry import registry as metrics_registry
 
     # Names enumerated from the leanMetrics spec. Any change to this list
     # is a cross-client-visible metrics surface change and should be

packages/testing/src/consensus_testing/test_fixtures/fork_choice.py

@@ -11,17 +11,18 @@
 
 from pydantic import Field, model_validator
 
-from lean_spec.forks.lstar.containers import (
+from lean_spec.node.chain.clock import Interval
+from lean_spec.spec.crypto.merkleization import hash_tree_root
+from lean_spec.spec.forks.lstar.containers import (
     AggregatedAttestations,
     Block,
     BlockBody,
     State,
     Validators,
 )
-from lean_spec.forks.lstar.spec import LstarSpec
-from lean_spec.subspecs.chain.clock import Interval
-from lean_spec.subspecs.ssz import hash_tree_root
-from lean_spec.types import Slot, Uint64, ValidatorIndex
+from lean_spec.spec.forks.lstar.spec import LstarSpec
+from lean_spec.spec.ssz import Uint64
+from lean_spec.types import Slot, ValidatorIndex
 
 from ..keys import (
     XmssKeyManager,

packages/testing/src/consensus_testing/test_fixtures/gossipsub_handler.py

@@ -13,11 +13,11 @@
 from typing import Any, ClassVar
 from unittest.mock import patch
 
-from lean_spec.subspecs.networking import PeerId
-from lean_spec.subspecs.networking.gossipsub.behavior import GossipsubBehavior, PeerState
-from lean_spec.subspecs.networking.gossipsub.message import GossipsubMessage
-from lean_spec.subspecs.networking.gossipsub.parameters import GossipsubParameters
-from lean_spec.subspecs.networking.gossipsub.rpc import (
+from lean_spec.node.networking import PeerId
+from lean_spec.node.networking.gossipsub.behavior import GossipsubBehavior, PeerState
+from lean_spec.node.networking.gossipsub.message import GossipsubMessage
+from lean_spec.node.networking.gossipsub.parameters import GossipsubParameters
+from lean_spec.node.networking.gossipsub.rpc import (
     RPC,
     ControlGraft,
     ControlIDontWant,
@@ -27,7 +27,7 @@
     ControlPrune,
     Message,
 )
-from lean_spec.subspecs.networking.gossipsub.types import MessageId, Timestamp, TopicId
+from lean_spec.node.networking.gossipsub.types import MessageId, Timestamp, TopicId
 
 from .base import BaseConsensusFixture
 

packages/testing/src/consensus_testing/test_fixtures/networking_codec.py

@@ -2,10 +2,9 @@
 
 from typing import Any, ClassVar
 
-from lean_spec.snappy import compress, decompress, frame_compress, frame_decompress
-from lean_spec.subspecs.networking.enr.enr import ENR
-from lean_spec.subspecs.networking.gossipsub.message import GossipsubMessage
-from lean_spec.subspecs.networking.gossipsub.rpc import (
+from lean_spec.node.networking.enr.enr import ENR
+from lean_spec.node.networking.gossipsub.message import GossipsubMessage
+from lean_spec.node.networking.gossipsub.rpc import (
     RPC,
     ControlGraft,
     ControlIDontWant,
@@ -16,15 +15,16 @@
     Message,
     SubOpts,
 )
-from lean_spec.subspecs.networking.gossipsub.topic import GossipTopic, TopicKind
-from lean_spec.subspecs.networking.gossipsub.types import TopicId
-from lean_spec.subspecs.networking.reqresp.codec import (
+from lean_spec.node.networking.gossipsub.topic import GossipTopic, TopicKind
+from lean_spec.node.networking.gossipsub.types import TopicId
+from lean_spec.node.networking.reqresp.codec import (
     ResponseCode,
     decode_request,
     encode_request,
 )
-from lean_spec.subspecs.networking.transport.peer_id import KeyType, PeerId, PublicKeyProto
-from lean_spec.subspecs.networking.varint import decode_varint, encode_varint
+from lean_spec.node.networking.transport.peer_id import KeyType, PeerId, PublicKeyProto
+from lean_spec.node.networking.varint import decode_varint, encode_varint
+from lean_spec.node.snappy import compress, decompress, frame_compress, frame_decompress
 from lean_spec.types import SubnetId
 
 from .base import BaseConsensusFixture

packages/testing/src/consensus_testing/test_fixtures/poseidon_permutation.py

@@ -1,20 +1,20 @@
-"""Poseidon1 permutation test fixture.
+"""Poseidon permutation test fixture.
 
-Generates JSON test vectors for the Poseidon1 permutation over the
+Generates JSON test vectors for the Poseidon permutation over the
 KoalaBear field at widths 16 and 24. Clients must produce identical
 output states bit-for-bit for every input state.
 """
 
 from typing import Any, ClassVar
 
-from lean_spec.subspecs.koalabear.field import Fp
-from lean_spec.subspecs.poseidon1 import PARAMS_16, PARAMS_24, Poseidon1
+from lean_spec.spec.crypto.koalabear import Fp
+from lean_spec.spec.crypto.poseidon import PARAMS_16, PARAMS_24, Poseidon
 
 from .base import BaseConsensusFixture
 
 
 class PoseidonPermutationTest(BaseConsensusFixture):
-    """Fixture for Poseidon1 permutation conformance.
+    """Fixture for Poseidon permutation conformance.
 
     Each vector names the permutation width and supplies an input state
     as decimal strings. The fixture runs the spec's permutation engine
@@ -24,7 +24,7 @@ class PoseidonPermutationTest(BaseConsensusFixture):
     """
 
     format_name: ClassVar[str] = "poseidon_permutation"
-    description: ClassVar[str] = "Tests Poseidon1 permutation at widths 16 and 24"
+    description: ClassVar[str] = "Tests Poseidon permutation at widths 16 and 24"
 
     width: int
     """State width. Must be 16 or 24."""
@@ -36,7 +36,7 @@ class PoseidonPermutationTest(BaseConsensusFixture):
     """Computed output state. Filled by make_fixture."""
 
     def make_fixture(self) -> "PoseidonPermutationTest":
-        """Run the Poseidon1 permutation and produce the output state.
+        """Run the Poseidon permutation and produce the output state.
 
         Returns:
             A copy of this fixture with output populated.
@@ -45,11 +45,11 @@ def make_fixture(self) -> "PoseidonPermutationTest":
             ValueError: If the width is unsupported.
         """
         if self.width == 16:
-            engine = Poseidon1(PARAMS_16)
+            engine = Poseidon(PARAMS_16)
         elif self.width == 24:
-            engine = Poseidon1(PARAMS_24)
+            engine = Poseidon(PARAMS_24)
         else:
-            raise ValueError(f"Unsupported Poseidon1 width: {self.width}")
+            raise ValueError(f"Unsupported Poseidon width: {self.width}")
 
         state_ints = [int(x) for x in self.input["inputState"]]
         if len(state_ints) != self.width: