Integrate xmss (leanEthereum#117)

* feat: integrate xmss signatures

* fix: non deterministic tests

* fix: address comments

* fix: address comments

* feat: add xmss caching

* fix: lint

* feat: parallelize fill test fixtures

Author: anshalshukla

.github/workflows/ci.yml

@@ -76,4 +76,4 @@ jobs:
         run: uv sync --no-progress
 
       - name: Fill test fixtures
-        run: uv run fill --fork=Devnet --clean
+        run: uv run fill --fork=Devnet --clean -n auto

CLAUDE.md

@@ -16,15 +16,17 @@ subspecifications that the Lean Ethereum protocol relies on.
 ## Development Workflow
 
 ### Running Tests
+
 ```bash
 uv sync                           # Install dependencies
 uv run pytest                     # Run unit tests
-uv run fill --fork=devnet --clean # Generate test vectors
+uv run fill --fork=devnet --clean -n auto # Generate test vectors
 # Note: execution layer support is planned for future, infrastructure is ready
 # for now, `--layer=consensus` is default and the only value used.
 ```
 
 ### Code Quality
+
 ```bash
 uv run ruff format src tests packages      # Format code
 uv run ruff check --fix src tests packages # Lint and fix
@@ -34,13 +36,15 @@ uvx tox                                    # Everything (checks + tests + docs)
 ```
 
 ### Common Tasks
+
 - **Main specs**: `src/lean_spec/`
 - **Subspecs**: `src/lean_spec/subspecs/{subspec}/`
 - **Unit tests**: `tests/lean_spec/` (mirrors source structure)
 - **Consensus spec tests**: `tests/consensus/` (generates test vectors)
 - **Execution spec tests**: `tests/execution/` (future - infrastructure ready)
 
 ## Code Style
+
 - Line length: 100 characters, type hints everywhere
 - Google docstring style (no docstrings for `__init__`)
 - Test files/functions must start with `test_`
@@ -54,15 +58,17 @@ uvx tox                                    # Everything (checks + tests + docs)
    - *Note: `tests/execution/` infrastructure is ready for future execution layer work*
 
 **Test Filling Framework:**
+
 - Layer-agnostic pytest plugin in `packages/testing/src/framework/pytest_plugins/filler.py`
 - Layer-specific packages: `consensus_testing` (active) and `execution_testing` (future)
 - Write consensus spec tests using `state_transition_test` or `fork_choice_test` fixtures
 - These fixtures are type aliases that create test vectors when called
-- Run `uv run fill --fork=Devnet --clean` to generate consensus fixtures
+- Run `uv run fill --fork=Devnet --clean -n auto` to generate consensus fixtures
 - Use `--layer=execution` flag when execution layer is implemented
 - Output goes to `fixtures/{layer}/{format}/{test_path}/...`
 
 **Example spec test:**
+
 ```python
 def test_block(state_transition_test: StateTransitionTestFiller) -> None:
     state_transition_test(
@@ -73,6 +79,7 @@ def test_block(state_transition_test: StateTransitionTestFiller) -> None:
 ```
 
 **How it works:**
+
 1. Test function receives a fixture class (not instance) as parameter
 2. Calling it creates a `FixtureWrapper` that runs `make_fixture()`
 3. `make_fixture()` executes the spec code (state transitions, fork choice steps)
@@ -81,12 +88,14 @@ def test_block(state_transition_test: StateTransitionTestFiller) -> None:
 6. Writes fixtures at session end to `fixtures/{layer}/{format}/{test_path}/...`
 
 **Layer-specific architecture:**
+
 - `framework/` - Shared infrastructure (base classes, pytest plugin, CLI)
 - `consensus_testing/` - Consensus layer fixtures, forks, builders
 - `execution_testing/` - Execution layer fixtures, forks, builders
 - Regular pytest runs (`uv run pytest`) ignore spec tests - they only run via `fill` command
 
 **Serialization requirements:**
+
 - All spec types (State, Block, Uint64, etc.) must be Pydantic models
 - Custom types need `@field_serializer` or `model_serializer` for JSON output
 - SSZ types typically serialize to hex strings (e.g., `"0x1234..."`)
@@ -97,6 +106,7 @@ def test_block(state_transition_test: StateTransitionTestFiller) -> None:
 - Test the serialization: `fixture.model_dump(mode="json")` must produce valid JSON
 
 **Key fixture types:**
+
 - `StateTransitionTest` - Tests state transitions with blocks
 - `ForkChoiceTest` - Tests fork choice with steps (tick/block/attestation)
 - Selective validation via `StateExpectation` and `StoreChecks` (only validates fields you specify)
@@ -113,18 +123,21 @@ def test_block(state_transition_test: StateTransitionTestFiller) -> None:
 When creating SSZ types, follow these established patterns:
 
 ### Domain-Specific Types (Preferred)
+
 - Use meaningful names that describe the purpose: `JustificationValidators`, `HistoricalBlockHashes`, `Attestations`
 - Define domain-specific types in modular structure (see Architecture section below)
 - Avoid generic names with numbers like `Bitlist68719476736` or `SignedAttestationList4096`
 
 ### SSZType vs SSZModel Design Decision
 
 **SSZType (IS-A pattern)**: Use for types that *are* data
+
 - Primitive scalars: `Uint64`, `Boolean`, `Bytes32`
 - These inherit directly from their underlying Python types
 - Example: `Uint64(42)` *is* the integer 42 with SSZ serialization
 
 **SSZModel (HAS-A pattern)**: Use for types that *have* data
+
 - Collections: `SSZList`, `SSZVector`, bitfields
 - Containers: `State`, `Block`, etc.
 - These use Pydantic models with a `data` field for contents
@@ -150,6 +163,7 @@ src/lean_spec/subspecs/containers/
 ```
 
 **Key principles:**
+
 - **Base types** (BaseBitlist, SSZList, etc.) stay in general scope (`src/lean_spec/types/`)
 - **Spec-specific types** go in their respective modules (`state/types.py`, `block/types.py`)
 - **Public API** exposed through `__init__.py` files for backward compatibility
@@ -158,6 +172,7 @@ src/lean_spec/subspecs/containers/
 ### Examples
 
 **Good domain-specific types:**
+
 ```python
 # In state/types.py
 HISTORICAL_ROOTS_LIMIT = 262144
@@ -174,6 +189,7 @@ class Attestations(SSZList):
 ```
 
 **Avoid generic types:**
+
 ```python
 # Don't do this:
 class Bitlist68719476736(BaseBitlist): ...
@@ -183,5 +199,6 @@ class SignedAttestationList4096(SSZList): ...
 ### API Compatibility
 
 When refactoring, maintain backward compatibility:
+
 - Keep existing import paths working through `__init__.py` exports
 - Preserve method signatures and behavior

packages/testing/src/consensus_testing/keys.py

@@ -0,0 +1,103 @@
+"""XMSS key management utilities for testing."""
+
+from typing import Optional
+
+from lean_spec.subspecs.containers import Attestation, Signature
+from lean_spec.subspecs.containers.slot import Slot
+from lean_spec.subspecs.ssz.hash import hash_tree_root
+from lean_spec.subspecs.xmss.containers import PublicKey, SecretKey
+from lean_spec.subspecs.xmss.interface import DEFAULT_SIGNATURE_SCHEME
+from lean_spec.types import ValidatorIndex
+
+
+class XmssKeyManager:
+    """
+    Manages XMSS keys for test validators.
+
+    Generates and manages XMSS key pairs for validators on demand.
+    Keys are generated to be valid up to the specified max_slot.
+    """
+
+    DEFAULT_MAX_SLOT = Slot(100)
+    """Default maximum slot for key generation."""
+
+    max_slot: Slot
+    public_keys: dict[ValidatorIndex, PublicKey]
+    secret_keys: dict[ValidatorIndex, SecretKey]
+
+    def __init__(
+        self,
+        max_slot: Optional[Slot] = None,
+    ) -> None:
+        """
+        Initialize the XMSS key manager.
+
+        Args:
+            max_slot: Maximum slot for which keys should be valid. Keys will be
+                generated with enough capacity to sign messages up to this slot.
+                Defaults to 100 slots.
+        """
+        self.max_slot = max_slot if max_slot is not None else self.DEFAULT_MAX_SLOT
+        self.public_keys: dict[ValidatorIndex, PublicKey] = {}
+        self.secret_keys: dict[ValidatorIndex, SecretKey] = {}
+
+    def create_and_store_key_pair(
+        self, validator_index: ValidatorIndex
+    ) -> tuple[PublicKey, SecretKey]:
+        """
+        Create an XMSS key pair for the given validator index.
+
+        Args:
+            validator_index: The index of the validator to create a key for.
+
+        Returns:
+            A tuple containing the public and secret keys for the given validator index.
+        """
+        if validator_index not in self.public_keys:
+            # Use max_slot + 1 as num_active_epochs since slots are used as epochs in the spec.
+            # +1 to include genesis slot
+            num_active_epochs = self.max_slot.as_int() + 1
+            self.public_keys[validator_index], self.secret_keys[validator_index] = (
+                DEFAULT_SIGNATURE_SCHEME.key_gen(0, num_active_epochs)
+            )
+        return self.public_keys[validator_index], self.secret_keys[validator_index]
+
+    def sign_attestation(self, attestation: Attestation) -> Signature:
+        """
+        Sign an attestation with the given validator index.
+
+        Args:
+            attestation: The attestation to sign.
+
+        Returns:
+            A signature for the given attestation.
+        """
+        validator_id = attestation.validator_id
+
+        sk = self.secret_keys[validator_id]
+        message = bytes(hash_tree_root(attestation))
+        epoch = int(attestation.data.slot)
+        xmss_sig = DEFAULT_SIGNATURE_SCHEME.sign(sk, epoch, message)
+
+        signature_bytes = xmss_sig.to_bytes(DEFAULT_SIGNATURE_SCHEME.config)
+        # Pad to 3100 bytes (Signature.LENGTH) with zeros on the right
+        # Padding only occurs with TEST_CONFIG(796 bytes) and not PROD_CONFIG(3100 bytes).
+        padded_bytes = signature_bytes.ljust(Signature.LENGTH, b"\x00")
+        signature = Signature(padded_bytes)
+        return signature
+
+    def __contains__(self, validator_index: ValidatorIndex) -> bool:
+        """
+        Check if a validator has a registered key.
+
+        Args:
+            validator_index: The index of the validator to check.
+
+        Returns:
+            True if the validator has a registered key, False otherwise.
+        """
+        return validator_index in self.secret_keys
+
+    def __len__(self) -> int:
+        """Return the number of registered keys."""
+        return len(self.secret_keys)