bug: BIP-340 Specification Violation: schnorr.ParseSignature accepts non-canonical signatures

[RafilxTenfen]

BIP-340 Specification Violation: schnorr.ParseSignature accepts non-canonical signatures with S > curve order, enabling signature malleability

Immunefi #54049

Description

Brief/Intro

The integrating btcsuite schnorr.ParseSignature() function call from the babylon implementation violates the BIP-340 specification by accepting signatures with S values ≥ curve order, when it should strictly require S ∈ [0, n-1].

This creates a signature malleability vulnerability where two different signature byte representations can be mathematically valid for the same transaction when low S values occur naturally.

This affects signature parsing and validation throughout the codebase (such as staking) wherever schnorr.ParseSignature() and then signature.Verify() is called for signature validation. If exploited, this could enable transaction replay attacks, double-spending scenarios, or protocol confusion where the same staking transaction could have multiple valid signature representations, potentially bypassing slashing mechanisms or creating inconsistent transaction states across the network.

Vulnerability Details

Root Cause

The vulnerability exists due to the non validated inputs (e.g. not checking the S component) when the Babylon implementation calls the btcsuite schnorr.ParseSignature() [1] implementation. This call sequence fails to validate that the S component of Schnorr signatures falls within the required range specified by BIP-340 [2].

According to the BIP-340 [2] specification, signature parsing must "Fail if s ≥ n" where n is the curve order, but the current implementation [1] only validates the R component:

func ParseSignature(sig []byte) (*Signature, error) {
// ... length checks ...
var r btcec.FieldVal
if overflow := r.SetByteSlice(sig[0:32]); overflow {
str := "invalid signature: r >= field prime"
return nil, signatureError(ecdsa_schnorr.ErrSigRTooBig, str)
}
var s btcec.ModNScalar
s.SetByteSlice(sig[32:64]) // MISSING OVERFLOW CHECK HERE
return NewSignature(&r, &s), nil
}

The R component correctly checks for overflow and rejects invalid values, but the S component uses ModNScalar.SetByteSlice()[3] which silently performs modular reduction without reporting when the input value exceeds the curve order.

This exact path happens during the critical verification call of VerifyTransactionSigWithOutput()[4] with no further input validation on call sites for the S component; which is used for staking and slashing signature verification.

Signature Malleability Mechanism

When a signature naturally has a low S value (S < 2^256 - n), an attacker can create a malleated version by computing S' = S + n. Both signatures are mathematically equivalent because:

Original: (R, S) where S < n
Malleated: (R, S + n) where S + n still fits in 256 bits
Both signatures will verify successfully because (S + n) mod n ≡ S mod n, making them cryptographically equivalent despite having different byte representations.

Attack Vector

Natural occurrence: When signing produces a signature with S < (2^256 - n)
Malleation: Attacker computes S_malleated = S + curve_order
Bypass parsing: Both signatures pass ParseSignature() validation (with different values of S)
Verification success: Both signatures verify against the same message and public key
Protocol confusion: Same transaction now has two valid signature representations

Feasibility Limitations

This report would like to highlight the practical exploitation of this vulnerability is severely constrained by the low probability of naturally occurring signatures with S values small enough to enable malleability (S < 2^256 - n).

Combined with deterministic signing preventing attackers from generating multiple signature attempts, making this primarily a specification compliance issue for BIP-340 [2] rather than a practical security threat under normal conditions.

Impact Details

The BIP-340 specification violation in schnorr signature parsing creates signature malleability risks that undermine transaction uniqueness guarantees in the Babylon staking protocol:

• High: "Babylon node recognizes Bitcoin Staking protocol transactions and/or signatures as valid when they are invalid" - When low-S signatures naturally occur, attackers can create malleated versions where S' = S + curve_order that remain mathematically valid but violate BIP-340 specification requirements. Both the original canonical signature (S < curve_order) and the malleated non-canonical signature (S ≥ curve_order) will pass ParseSignature() validation and verify successfully against the same transaction. This allows the same staking transaction to exist with multiple valid signature representations, creating protocol confusion where slashing mechanisms might fail to correlate malleated signatures with their canonical counterparts. The parsing layer incorrectly validates signatures that should be rejected per BIP-340 specification, enabling potential bypasses of security mechanisms that rely on signature uniqueness for proper staking transaction identification and validator accountability.
• Medium: "Generation of staking transactions that cannot be confirmed by the Bitcoin ledger" - The acceptance of non-canonical signatures creates ambiguity in transaction propagation and state tracking. When malleated signatures are broadcast alongside their canonical counterparts, Bitcoin network nodes implementing strict BIP-340 compliance may reject the non-canonical versions while accepting the canonical ones, leading to inconsistent transaction states across different network participants. This specification violation undermines the cryptographic assumptions that staking protocol components rely upon for transaction uniqueness and could result in failed transaction confirmations when non-canonical signatures are submitted to BIP-340 compliant Bitcoin nodes.

Suggested Fix

Validate the S signature component falls within the range [0, n] before calling schnorr.ParseSignature()

References

[1] ParseSignature() implementation missing S overflow https://github.com/btcsuite/btcd/blob/e8097a1b044c3c0b3843b2aba377e87eb5fc4f9f/btcec/schnorr/signature.go#L70-L97
[2] BIP-340 https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki#verification
[3] ModNScalar.SetByteSlice() implementation https://github.com/decred/dcrd/blob/95a36a36ed6f1361d904283357537a7a8bf644c3/dcrec/secp256k1/modnscalar.go#L349-L368
[4] VerifyTransactionSigWithOutput() implementation

babylon/btcstaking/staking.go

Lines 688 to 694 in 46f6210

	parsedSig, err := schnorr.ParseSignature(signature)
	if err != nil {
	return err
	}
	valid := parsedSig.Verify(sigHash, pubKey)

Proof of concept

To test specifically the integrating library calls from btcsuite, the following test case can be added to the btcstaking/staking_test.go [1]:

func TestSchnorrParseSignatureBIP340Violation(t *testing.T) {
// Test demonstrating that schnorr.ParseSignature accepts
// non-canonical signatures with S >= curve order, violating BIP-340

curveOrder := btcec.S256().N

// Generate keys and message for verification test
privKey, err := btcec.NewPrivateKey()
require.NoError(t, err)
pubKey := privKey.PubKey()
msgHash := make([]byte, 32)
msgHash[0] = 0x01

// Create a signature with valid R but invalid S (S = curve_order + 1)
validR := make([]byte, 32)
validR[31] = 0x01

invalidS := new(big.Int).Add(curveOrder, big.NewInt(1))
invalidSBytes := make([]byte, 32)
invalidS.FillBytes(invalidSBytes)

// Construct non-canonical signature: R || S where S >= curve_order
nonCanonicalSig := make([]byte, 64)
copy(nonCanonicalSig[:32], validR)
copy(nonCanonicalSig[32:], invalidSBytes)

// Verify S is indeed >= curve order
sBigInt := new(big.Int).SetBytes(invalidSBytes)
require.True(t, sBigInt.Cmp(curveOrder) >= 0, "S should be >= curve order")

// BIP-340 requires: "Fail if s >= n"
// ParseSignature should reject this signature but doesn't
parsedSig, err := schnorr.ParseSignature(nonCanonicalSig)
require.NoError(t, err, "Test failed: ParseSignature correctly rejected invalid S (bug has been fixed!)")
require.NotNil(t, parsedSig, "Test failed: ParseSignature returned nil despite no error")

// Verification should fail because the signature is invalid
isValid := parsedSig.Verify(msgHash, pubKey)

// NOTE: In this case the signature is rejected because we picked a nonsense S
// Computing an intentionally low S value to generate malleability is of a significantly low probability, however not impossible
// If a low S value ever appears due to chance then this allows producing a malleated form of this signature i.e. S in the range [N, 2^256)
require.False(t, isValid, "Verification should reject non-canonical signature")
}
```