btcec/schnorr: document signing and verifying divergences against BIP340 and libsecp256k1 behavior

[allocz]

Change Description

Sign and Verify: document the requirement of 32 byte messages, as BIP340 requires and libsecp256k1 supports variable length messages.

Sign: since RFC6979 is used when CustomNonce is not passed, which is a deviation from BIP340 and secp256k1, a not was added explaining that passing an empty 32 byte array with CustomNonce is the same as the default behavior of libsecp256k1 when producing signatures.

Several leakages of sensitive material were fixed.

I would like to make the default behavior BIP340 compliant, matching libsecp256k1, but as this would introduce different behavior, even with the same API, the recommendation was to keep the current behavior but document it better.

Closes #2501 #2542 #2543

Steps to Test

cd ./btcec
go test . -count=1

Pull Request Checklist

Testing

• Your PR passes all CI checks.
• Tests covering the positive and negative (error paths) are included.
• Bug fixes contain tests triggering the bug to prevent regressions.

Code Style and Documentation

• The change is not insubstantial. Typo fixes are not accepted to fight bot spam.
• The change obeys the Code Documentation and Commenting guidelines, and lines wrap at 80.
• Commits follow the Ideal Git Commit Structure.
• Any new logging statements use an appropriate subsystem and logging level.

📝 Please see our Contribution Guidelines for further guidance.

[Copilot]

Pull request overview

This PR updates the btcec/schnorr BIP340 signing and verification implementation to match the current BIP340/libsecp256k1 behavior, particularly around variable-length messages and default nonce derivation when auxiliary randomness is not provided.

Changes:

• Allow Schnorr signing and verification to accept variable-length messages (no 32-byte length enforcement).
• Change signing behavior so omitting CustomNonce is treated equivalently to providing 32 zero bytes (matching libsecp256k1/BIP340 behavior, removing the RFC6979 fallback).
• Update/extend test vectors to cover omitted auxRand behavior and non-32-byte message cases.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
btcec/schnorr/signature.go	Adjusts Sign/Verify behavior to accept variable-length messages and align default auxRand handling with BIP340/libsecp256k1; includes comment/doc updates needed.
btcec/schnorr/signature_test.go	Updates/extends BIP340 test vectors and signing options setup to validate new behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[GustavoStingelin]

While this converges toward the secp approach, does it also introduce any nonce reuse risk?

If a bug causes signatures to be produced with a zeroed nonce, the private key could be extracted. Given that, I think falling back to RFC6979 would provide an extra safety layer.

Rejecting a zeroed nonce outright is another option, but that seems to conflict with the goal of this PR, which is to match secp signing behavior.

At minimum, I think it would be helpful to document why secp considers this safe with a zeroed nonce, and why this code historically used RFC6979 instead.

[GustavoStingelin]

We should also zero t. With a zero nonce, the XOR operation leaves a recoverable private key in that variable.

[allocz]

The data derived from auxRand (t) is hashed with the public key and the message to generate rand, so, there's no risk of nonce reusage by using a zeroed auxRand. Also, the name CustomNonce is misleading, the correct name should be AuxRand, for this reason I marked CustomNonce as deprecated and added AuxRand in replacement.

Now, zeroing both t and privBytes.

[allocz]

This implementation was written in the commit d6d38ad, which states:

The signing implementation by default, deviates from BIP-340 as it opts
to use rfc6979 deterministic signatures by default, which means callers
don't need to always pass in their own `auxNonce` randomness. A set of
functional arguments allows callers to pass in their own value, which is
the way all the included test vectors function.

Refreshing it again, BIP-340 allows the auxRand to be all zeroes, which is the case for multiple of the test vectors. The generated rand is always safe to use, different messages will result in different nonces, even with auxRand being all zeroes, so there's no safety added by using RFC6979 here, only additional compute cost.

libsecp256k1 states that passing null (which in this case is not passing AuxRand) is the same as passing the zeroed 32 byte array. Adopting this enables faster signature generation and a behavior that matches both libsecp256k1 and BIP-340 without compromising security.

Would be nice to see @Roasbeef take on this.

[allocz]

SignDefault, which is the path that previously used RFC6979 is now ~5% faster, with a drop of 50% in the total allocations.

goos: linux
goarch: amd64
pkg: github.com/btcsuite/btcd/btcec/v2/schnorr
cpu: Intel(R) Core(TM) i5-10310U CPU @ 1.70GHz
              │ /tmp/before │             /tmp/after             │
              │   sec/op    │   sec/op     vs base               │
SigVerify-8     169.4µ ± 0%   160.8µ ± 0%  -5.09% (n=60)
Sign-8          70.07µ ± 0%   69.88µ ± 0%  -0.27% (p=0.004 n=60)
SignDefault-8   74.23µ ± 0%   69.90µ ± 0%  -5.84% (n=60)
geomean         95.86µ        92.25µ       -3.76%

              │ /tmp/before │              /tmp/after              │
              │    B/op     │    B/op     vs base                  │
SigVerify-8      240.0 ± 0%   240.0 ± 0%        ~ (p=1.000 n=60) ¹
Sign-8           496.0 ± 0%   496.0 ± 0%        ~ (p=1.000 n=60) ¹
SignDefault-8   1216.0 ± 0%   464.0 ± 0%  -61.84% (n=60)
geomean          525.1        380.8       -27.47%
¹ all samples are equal

              │ /tmp/before │              /tmp/after              │
              │  allocs/op  │ allocs/op   vs base                  │
SigVerify-8      5.000 ± 0%   5.000 ± 0%        ~ (p=1.000 n=60) ¹
Sign-8           12.00 ± 0%   12.00 ± 0%        ~ (p=1.000 n=60) ¹
SignDefault-8    22.00 ± 0%   11.00 ± 0%  -50.00% (n=60)
geomean          10.97        8.707       -20.63%
¹ all samples are equal

[GustavoStingelin]

Cool, LGTM!

I noticed a few linting issues, but it looks like linting is not running in CI yet. I opened an issue to track that separately: #2549

[Roasbeef]

Sign, and Verify: accept variable length messages, as required by BIP340 and implemented by libsecp256k1.

So this was a purposeful decision w.r.t the API. Do you have a concrete use case of signing arbitrary length messages?

Sign: don't use RFC6979 when auxRand bytes are not passed by the caller, instead, match BIP340 spec and libsecp256k1 implementation by making not passing CustomNonce equivalent to passing a zeroed 32 byte array.

Again a divergence from the original design goals of the API. RFC6979 is a safer default, as it makes nonce re-use harder for callers.

[allocz]

So this was a purposeful decision w.r.t the API. Do you have a concrete use case of signing arbitrary length messages?

For now, the objective is to pass the BIP-340 test vectors.

Again a divergence from the original design goals of the API. RFC6979 is a safer default, as it makes nonce re-use harder for callers.

The generated rand is always safe to use, different messages will result in different nonces, even with auxRand being all zeroes, so there's no safety added by using RFC6979 here, only additional compute cost.

[TechLateef]

Since the nonce derivation in BIP340 includes the private key and message, zeroed auxRand doesn't introduce nonce reuse risk. That said, @Roasbeef concern about API safety for callers who don't understand the internals is worth considering maybe a doc comment explicitly warning that passing no AuxRand is equivalent to zeroed bytes (not random) would help callers make an informed decision.

[kcalvinalvin]

Thanks for the work! Some thoughts:

1: I think we should stick to the original design with the 32 byte check since we're changing the public API and the behavior. If passing the BIP340 tests are the goal, testing the unexported functions schnorrSign and schnorrVerify is better imo. Best not to sliently change what's happening under the hood. Later down the road if someone wants to sign arbitrary length messages, we can have a separate function for that instead of modifying Sign.

2: I went over RFC6979 vs 32 zeros and came to the conclusion that zeroing out doesn't really affect safety. For single sig it doesn't matter anyways since the r value is different for each message. For multi-sig, it doesn't matter if you're using 0s or RFC6979 since they're both deterministic. But again, since this will cause the API to break, I'm leaning towards keeping it the same.

[allocz]

As the deviation from BIP340 does not introduce vulnerabilities, I've changed the commit to keep the current behavior (as @kcalvinalvin suggested), but also added more documentation covering that BIP340 signatures can be generated by using a zero 32 byte array with CustomNonce and also about arbitrary length messages not being supported.

Also fixed the docs and some private key and nonce leaks.