Aggregated block proof - devnet5 (leanEthereum#717)

* dummy type 1 and type 2 aggregation with block proofs

* integrate bindings, fix tests

* update bindings

* update bindings

* cleanup

* fix lint

* limit max attestation data to 8

* update CI params, remove parallelization in fill

* fix reaggregation logic, cleanup

* reduce max proof size to 500KiB

* switch few CI jobs to macos

* fix failing tc post main merge, minor refactor

* deconstruct even in case of proposer, not just for aggregators

* address comments

* block deconstruction

* address review feedback

- Fix the broken sentence in LstarSpec.verify_signatures' docstring
  (dangling colon now reads as a coherent description of what the
  Type-2 proof binds).
- Delete test_noop_when_not_a_validator: the validator-id early-return
  it exercised was removed so non-validator nodes also accrue
  block-imported attestation weight.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* address second-pass review

Fixes flagged in the second round of consensus and Python architecture
review:

- sync/service.py: widen the decode_bytes catch to (SSZError,
  ValueError, IndexError). Pre-fix narrowing missed the SSZ exception
  family raised by Container deserialization, so a malformed proof
  would crash _process_block_wrapper instead of being demoted to a
  debug log.
- sync/service.py: align the per-attestation loop variable name with
  the second loop.
- validator/service.py: validate each participant index against the
  active validator set before indexing validators[vid] in _sign_block,
  raising a clear ValueError instead of a deep KeyError if a stale
  partial aggregate is passed in.
- spec.py: document the one-slot deferral of fork-choice weight from
  block-imported attestations.  The empty-set seeding plus the
  subsequent acceptance-tick migration is intentional and not a
  missed update_head call.
- test_aggregation.py: add five Type-2 unit tests covering the
  aggregate-rejects paths, the verify round-trip, the message-swap
  rejection (pins the per-component binding security property), and
  the length-mismatch rejection.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Thomas Coratger <60488569+tcoratger@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: 3 people

.github/workflows/ci.yml

@@ -76,9 +76,15 @@ jobs:
       - name: Run tests
         run: just test
 
+  # coverage-gate, fill-tests and interop-tests run on macOS because the
+  # lean_multisig_py prover's process-global setup is corrupted by xdist
+  # parallelism; these jobs must run serially and macOS runners were the
+  # stable choice during bring-up. This is an upstream limitation, not a
+  # preference: revisit (and prefer ubuntu for cost) once the prover setup
+  # is per-process safe. See the matching note in the Justfile fill-ci.
   coverage-gate:
     name: Coverage gate - Python 3.12
-    runs-on: ubuntu-latest
+    runs-on: macos-latest
     steps:
       - name: Checkout leanSpec
         uses: actions/checkout@v4
@@ -104,7 +110,7 @@ jobs:
 
   fill-tests:
     name: Fill test fixtures - Python 3.14
-    runs-on: ubuntu-latest
+    runs-on: macos-latest
     steps:
       - name: Checkout leanSpec
         uses: actions/checkout@v4
@@ -130,7 +136,7 @@ jobs:
 
   interop-tests:
     name: Interop tests - Multi-node consensus
-    runs-on: ubuntu-latest
+    runs-on: macos-latest
     timeout-minutes: 10
     steps:
       - name: Checkout leanSpec

Justfile

@@ -81,9 +81,13 @@ test-consensus *args:
     uv run --group test pytest -n auto --maxprocesses=10 --durations=10 --dist=worksteal tests/lean_spec/subspecs/containers tests/lean_spec/subspecs/forkchoice tests/lean_spec/subspecs/networking "$@"
 
 # Canonical CI fixture run; contributors should use `uv run fill` directly
+# Runs serially (no -n auto): xdist workers race on the lean_multisig_py
+# prover's process-global setup, which corrupts proofs intermittently.
+# This is an upstream limitation; restore -n auto once the prover setup
+# is per-process safe. See the matching note in .github/workflows/ci.yml.
 [group('tests'), private]
 fill-ci *args:
-    uv run --group test fill --fork=Lstar --clean -n auto "$@"
+    uv run --group test fill --fork=Lstar --clean "$@"
 
 # Run API conformance tests against an external client
 [group('tests')]

packages/testing/src/consensus_testing/keys.py

@@ -42,13 +42,10 @@
 
 from lean_spec.config import LEAN_ENV
 from lean_spec.forks.lstar.containers import AttestationData
-from lean_spec.forks.lstar.containers.block.types import (
-    AggregatedAttestations,
-    AttestationSignatures,
-)
+from lean_spec.forks.lstar.containers.block.types import AggregatedAttestations
 from lean_spec.subspecs.koalabear import Fp
 from lean_spec.subspecs.ssz.hash import hash_tree_root
-from lean_spec.subspecs.xmss.aggregation import AggregatedSignatureProof
+from lean_spec.subspecs.xmss.aggregation import TypeOneMultiSignature
 from lean_spec.subspecs.xmss.constants import TARGET_CONFIG
 from lean_spec.subspecs.xmss.containers import (
     PublicKey,
@@ -67,7 +64,13 @@
     HashTreeOpening,
     Randomness,
 )
-from lean_spec.types import Bytes32, Slot, Uint64, ValidatorIndex, ValidatorIndices
+from lean_spec.types import (
+    Bytes32,
+    Slot,
+    Uint64,
+    ValidatorIndex,
+    ValidatorIndices,
+)
 
 KeyRole = Literal["attestation", "proposal"]
 """Discriminator for which signing role's key to load from a validator key pair."""
@@ -514,18 +517,21 @@ def sign_and_aggregate(
         self,
         validator_ids: list[ValidatorIndex],
         attestation_data: AttestationData,
-    ) -> AggregatedSignatureProof:
+    ) -> TypeOneMultiSignature:
         """
-        Sign attestation data with each validator and aggregate into a single proof.
+        Sign attestation data with each validator and aggregate into a Type-1 proof.
 
-        Convenience method for the common sign-each-validator-then-aggregate pattern.
+        Each validator's XMSS attestation key signs the attestation data
+        root. The signatures are then handed to the multi-signature
+        binding to produce a single cryptographically valid Type-1 proof
+        binding all participants to (data, slot).
 
         Args:
             validator_ids: Validators to sign with.
             attestation_data: The attestation data to sign.
 
         Returns:
-            Aggregated signature proof combining all validators' signatures.
+            Cryptographically valid Type-1 proof covering validator_ids.
         """
         raw_xmss = [
             (
@@ -534,46 +540,44 @@ def sign_and_aggregate(
             )
             for vid in validator_ids
         ]
-
-        xmss_participants = ValidatorIndices(data=validator_ids).to_aggregation_bits()
-
-        return AggregatedSignatureProof.aggregate(
-            xmss_participants=xmss_participants,
+        return TypeOneMultiSignature.aggregate(
+            xmss_participants=ValidatorIndices(data=validator_ids).to_aggregation_bits(),
             children=[],
             raw_xmss=raw_xmss,
             message=hash_tree_root(attestation_data),
             slot=attestation_data.slot,
         )
 
-    def build_attestation_signatures(
+    def build_attestation_proofs(
         self,
         aggregated_attestations: AggregatedAttestations,
         signature_lookup: Mapping[AttestationData, Mapping[ValidatorIndex, Signature]]
         | None = None,
-    ) -> AttestationSignatures:
+    ) -> list[TypeOneMultiSignature]:
         """
-        Produce aggregated signature proofs for a list of attestations.
+        Produce Type-1 proofs aligned with the given attestations.
 
         For each aggregated attestation:
 
-        1. Identify participating validators from the aggregation bitfield
-        2. Collect each participant's public key and individual signature
-        3. Combine them into a single aggregated proof for the leanVM verifier
+        1. Identify participating validators from the aggregation bitfield.
+        2. Collect each participant's attestation public key and signature.
+        3. Combine them into a single Type-1 single-message proof via the
+           multi-signature binding.
 
         Pre-computed signatures can be supplied via the lookup to avoid
-        redundant signing. Missing signatures are computed on the fly.
+        redundant signing. Missing entries are signed on the fly.
 
         Args:
             aggregated_attestations: Attestations with aggregation bitfields set.
             signature_lookup: Optional pre-computed signatures keyed by
                 attestation data then validator index.
 
         Returns:
-            One aggregated signature proof per attestation.
+            One Type-1 single-message proof per attestation, parallel to the input.
         """
         lookup = signature_lookup or {}
 
-        proofs: list[AggregatedSignatureProof] = []
+        proofs: list[TypeOneMultiSignature] = []
         for agg in aggregated_attestations:
             # Decode which validators participated from the bitfield.
             validator_ids = agg.aggregation_bits.to_validator_indices()
@@ -582,7 +586,7 @@ def build_attestation_signatures(
             # Fall back to signing on the fly for any missing entries.
             sigs_for_data = lookup.get(agg.data, {})
 
-            # Collect the attestation public key for each participant.
+            # Collect the attestation public keys for each participant.
             public_keys = [self.get_public_keys(vid)[0] for vid in validator_ids]
 
             # Gather individual signatures, computing any that are missing.
@@ -593,16 +597,17 @@ def build_attestation_signatures(
 
             # Produce a single aggregated proof that the leanVM can verify
             # in one pass over all participants.
-            proof = AggregatedSignatureProof.aggregate(
-                xmss_participants=agg.aggregation_bits,
-                children=[],
-                raw_xmss=list(zip(public_keys, signatures, strict=True)),
-                message=hash_tree_root(agg.data),
-                slot=agg.data.slot,
+            proofs.append(
+                TypeOneMultiSignature.aggregate(
+                    children=[],
+                    raw_xmss=list(zip(public_keys, signatures, strict=True)),
+                    xmss_participants=agg.aggregation_bits,
+                    message=hash_tree_root(agg.data),
+                    slot=agg.data.slot,
+                )
             )
-            proofs.append(proof)
 
-        return AttestationSignatures(data=proofs)
+        return proofs
 
 
 def _generate_single_keypair(

packages/testing/src/consensus_testing/test_fixtures/fork_choice.py

@@ -300,7 +300,7 @@ def make_fixture(self) -> Self:
                     case BlockStep():
                         # Build a complete signed block from the lightweight spec.
                         # The spec contains minimal fields; we fill the rest.
-                        signed_block = step.block.build_signed_block_with_store(
+                        signed_block, store = step.block.build_signed_block_with_store(
                             store, self._block_registry, key_manager, self.lean_env
                         )
 

packages/testing/src/consensus_testing/test_fixtures/state_transition.py

@@ -10,7 +10,7 @@
 from lean_spec.forks.lstar.containers.state import State
 from lean_spec.forks.lstar.spec import LstarSpec
 from lean_spec.subspecs.ssz.hash import hash_tree_root
-from lean_spec.subspecs.xmss.aggregation import AggregatedSignatureProof
+from lean_spec.subspecs.xmss.aggregation import TypeOneMultiSignature
 from lean_spec.types import Bytes32, ValidatorIndices
 
 from ..keys import XmssKeyManager
@@ -250,7 +250,7 @@ def _build_block_from_spec(
 
         # Path 3: normal block construction via the spec's builder.
         else:
-            aggregated_payloads: dict[AttestationData, set[AggregatedSignatureProof]] = {}
+            aggregated_payloads: dict[AttestationData, set[TypeOneMultiSignature]] = {}
             if spec.attestations:
                 aggregated_payloads = StateTransitionTest._build_aggregated_payloads_from_spec(
                     spec.attestations, state, block_registry
@@ -304,7 +304,7 @@ def _build_aggregated_payloads_from_spec(
         attestation_specs: list[AggregatedAttestationSpec],
         state: State,
         block_registry: dict[str, Block],
-    ) -> dict[AttestationData, set[AggregatedSignatureProof]]:
+    ) -> dict[AttestationData, set[TypeOneMultiSignature]]:
         """
         Build aggregated signature payloads from attestation specifications.
 
@@ -320,7 +320,7 @@ def _build_aggregated_payloads_from_spec(
         # XMSS keys require precomputation up to the highest slot used.
         max_slot = max(spec.slot for spec in attestation_specs)
         key_manager = XmssKeyManager.shared(max_slot=max_slot)
-        payloads: dict[AttestationData, set[AggregatedSignatureProof]] = {}
+        payloads: dict[AttestationData, set[TypeOneMultiSignature]] = {}
 
         for spec in attestation_specs:
             if not spec.valid_signature:

packages/testing/src/consensus_testing/test_fixtures/verify_signatures.py

@@ -6,17 +6,20 @@
 
 from pydantic import Field
 
-from lean_spec.forks.lstar.containers.attestation import AggregatedAttestation
-from lean_spec.forks.lstar.containers.block import (
-    SignedBlock,
-)
-from lean_spec.forks.lstar.containers.block.types import (
-    AggregatedAttestations,
-    AttestationSignatures,
-)
+from lean_spec.forks.lstar.containers.attestation import AggregatedAttestation, AttestationData
+from lean_spec.forks.lstar.containers.block import SignedBlock
+from lean_spec.forks.lstar.containers.block.types import AggregatedAttestations
 from lean_spec.forks.lstar.containers.state import State
 from lean_spec.forks.lstar.spec import LstarSpec
-from lean_spec.types import AggregationBits, Boolean, ValidatorIndex
+from lean_spec.types import (
+    AggregationBits,
+    Boolean,
+    ByteList512KiB,
+    Bytes32,
+    Checkpoint,
+    Slot,
+    ValidatorIndex,
+)
 
 from ..keys import XmssKeyManager
 from ..test_types import BlockSpec
@@ -60,10 +63,6 @@ class VerifySignaturesTest(BaseConsensusFixture):
 
     Supported operations:
 
-    - `{"operation": "drop_last_signature"}`: Remove the last entry
-      from the block's attestation_signatures list. Produces a signed
-      block whose signature-group count is one less than its
-      attestation count.
     - `{"operation": "set_proposer_index", "value": int}`: Rewrite
       the block's proposer_index field. Use this to exercise the
       validator-bounds check that the builder skips because its round-
@@ -72,6 +71,15 @@ class VerifySignaturesTest(BaseConsensusFixture):
       first body attestation with one whose aggregation_bits carry no
       set bit. Exercises the empty-participants check inside
       signature verification.
+    - `{"operation": "corrupt_proof"}`: Replace the merged proof with
+      a short non-decodable blob. Exercises the Type-2 decode check.
+    - `{"operation": "append_phantom_attestation"}`: Add a body
+      attestation with no matching proof component. Exercises the
+      component count check between the body and the merged proof.
+    - `{"operation": "mutate_state_root"}`: Change a block field after
+      signing so the block root differs. Exercises the per-component
+      message binding that prevents reusing an honest proof under a
+      different message.
 
     Tampered blocks bypass the builder's structural invariants. The
     resulting fixture pins the exact rejection a client must raise when
@@ -153,16 +161,6 @@ def _apply_tamper(self, signed_block: SignedBlock) -> SignedBlock:
         assert self.tamper is not None
         operation = self.tamper.get("operation")
 
-        if operation == "drop_last_signature":
-            original = signed_block.signature.attestation_signatures.data
-            if not original:
-                raise ValueError("drop_last_signature requires at least one attestation signature")
-            truncated = AttestationSignatures(data=list(original[:-1]))
-            tampered_signatures = signed_block.signature.model_copy(
-                update={"attestation_signatures": truncated}
-            )
-            return signed_block.model_copy(update={"signature": tampered_signatures})
-
         if operation == "set_proposer_index":
             value = self.tamper.get("value")
             if value is None:
@@ -185,4 +183,43 @@ def _apply_tamper(self, signed_block: SignedBlock) -> SignedBlock:
             new_block = signed_block.block.model_copy(update={"body": new_body})
             return signed_block.model_copy(update={"block": new_block})
 
+        if operation == "corrupt_proof":
+            # Replace the merged proof with a short non-decodable blob.
+            # Decoding the Type-2 envelope must fail before verification.
+            return signed_block.model_copy(
+                update={"proof": ByteList512KiB(data=b"\x00\x01\x02\x03")}
+            )
+
+        if operation == "append_phantom_attestation":
+            # Add a body attestation with no matching proof component.
+            # The proof binds one component per original attestation plus
+            # the proposer, so the body now claims more components than the
+            # proof carries.
+            body = signed_block.block.body
+            phantom_data = AttestationData(
+                slot=Slot(0),
+                head=Checkpoint(root=Bytes32(b"\x00" * 32), slot=Slot(0)),
+                target=Checkpoint(root=Bytes32(b"\x00" * 32), slot=Slot(0)),
+                source=Checkpoint(root=Bytes32(b"\x00" * 32), slot=Slot(0)),
+            )
+            phantom = AggregatedAttestation(
+                aggregation_bits=AggregationBits(data=[Boolean(True)]),
+                data=phantom_data,
+            )
+            new_attestations = AggregatedAttestations(data=[*body.attestations.data, phantom])
+            new_body = body.model_copy(update={"attestations": new_attestations})
+            new_block = signed_block.block.model_copy(update={"body": new_body})
+            return signed_block.model_copy(update={"block": new_block})
+
+        if operation == "mutate_state_root":
+            # Change a block field after signing so the block root differs.
+            # The proposer component's bound message no longer matches the
+            # recomputed block root, even though the signature is honest.
+            # This is the repackaging vector: an honest proof reused under
+            # a different message.
+            tampered_block = signed_block.block.model_copy(
+                update={"state_root": Bytes32(b"\xff" * 32)}
+            )
+            return signed_block.model_copy(update={"block": tampered_block})
+
         raise ValueError(f"Unknown tamper operation: {operation!r}")