[2/2] Add Output Descriptor library - #2568
Conversation
|
Me looking at the number of lines changed 
. Would be awesome to have though! I'll start slowly going through... |
Yeah, it's a junky boyyy... A lot of the added lines are test vectors. But lots of Go code as well, I agree. |
351d6df to
3f5e7b6
Compare
|
I ran a full sweep with Kimi K3 that found 19 issues (1 critical, 6 high, 5 medium and 7 low) which are now fixed. The two READMEs now point out exactly what the supported features and known divergences are against Core and/or |
|
It’s a nice feature! Do you think it would be feasible to split this PR into a few smaller ones without losing too much context? That way we could review it in individual chunks. The new GitHub stacked PRs feature also seems like a good fit for this kind of change. |
Yes, you're right, splitting up the PR does make sense. I'll give it a try. |
|
I noticed |
guggero
changed the title
guggero
changed the title
Okay, PRs are now split. Only two parts, but way smaller commits each. I wanted to avoid needing to close this one, so I didn't end up using the stacked PR feature directly. But then I made a mistake which required me to close part 1 and re-open again anyway, so it wouldn't have mattered in the first place 😅 |
Yes, the feature parity with |
A descriptor names its keys through key expressions: a raw public key, a WIF private key, or an extended key with an optional origin and a derivation path that may end in a wildcard, and that may hold a multipath element. This parses them and derives the concrete key of a path, which is the part of a descriptor that turns one string into the many scripts of a wallet.
Output script descriptors carry an optional checksum which catches the transcription errors a raw descriptor string is prone to: any single character substitution, and a swap of characters that are not too far apart. BIP380 specifies it. This is the first piece of the descriptors package, and the only one that needs nothing else.
The descriptor grammar of BIP380 and its script expressions: the key based ones (pk, pkh, wpkh), the wrappers (sh, wsh), the multisigs (multi, sortedmulti) and a miniscript expression as the script of a wsh. Which expression may appear where is part of the grammar, so the parser tracks the position it is in and rejects, say, a wsh inside a wsh. The result is a tree of nodes with the keys of the descriptor collected in the order they appear. Nothing is derived from it yet.
A tr() descriptor has an internal key and an optional script tree, written as a nested pair of branches with a script expression at every leaf. BIP386 defines it. Parsing the tree is what the rest of this commit is about, including the two limits that keep a pathological descriptor from exhausting the process: the depth of the tree, which BIP341 caps at 128 anyway, and the depth of the nesting of the descriptor itself.
With the tree parsed and the keys derivable, a descriptor can produce what a wallet needs from it: the address of the output at a derivation index, and the script code that a signature over that input commits to. Both are per multipath and derivation index, and both compile a miniscript expression through the package of the previous commits where the descriptor holds one.
What a spend of a descriptor's output costs, in weight units: the witness or scriptSig that satisfies it, sized without the signatures being available. The estimate is the one a fee calculation needs, and for a taproot descriptor it is the cheapest of the key path and the leaves of the script tree, since that is the path a spender would take.
The redeem script of a P2SH output is pushed as a single element of the spending scriptSig, so the 520-byte limit on a script element applies to it: an output whose redeem script is larger can never be spent, and a descriptor that describes one is a mistake worth rejecting at parse time. The same holds for the number of keys of a bare multisig, which the standardness rules cap at three.
The normative vectors of BIP380 to BIP386: the descriptors those documents list as valid or invalid, with the scripts and addresses they must produce. The features this package does not support are classified explicitly rather than skipped silently, so that the gaps are visible in the test output instead of hiding in a skip.
The semantic policy of a whole descriptor rather than of a single miniscript: a taproot descriptor lifts to the choice between its key path and its leaves, a multisig to a threshold, and a wsh to the policy of its script. It is what a wallet shows a user as the spending conditions of an address.
A plan is the spending path a descriptor takes given what a signer holds: which signatures, preimages and time locks are available decides which branch of the descriptor can be satisfied, and of the branches that can be, the cheapest one is chosen. The plan carries the weight of that satisfaction and knows how to complete it once the real signatures arrive, which is the two-phase flow a wallet needs to build a transaction, size its fee and only then sign it.
For a taproot descriptor the plan chooses between the key path and every leaf of the script tree, which requires the control block of a leaf and therefore its merkle proof. The proofs are collected in one walk over the tree, and the cheapest of the candidates wins, as a spender would have it.
The differential vectors come from rust-miniscript and descriptors-go: for about 500 descriptors they compare the canonical string, the type, the keys, the addresses of a grid of derivation indices, the script code, the satisfaction weight, the lifted policy and the plan. Everything this package computes about a descriptor is in there, which is what makes the comparison worth having.
The descriptor parser takes strings from the outside as well, so it is fuzzed like the miniscript one, with the invariant that everything a descriptor derives stays consistent with the string it round trips to. The benchmarks cover parsing, address derivation and planning, and the recursion tests pin that a deeply nested descriptor is rejected rather than crashing the process.
A reader judging this package needs to know what it does, what it does not, and where it deliberately differs from Bitcoin Core and rust-miniscript. That is what these two documents are: the supported expressions, the unsupported ones, the divergences with the commit of each implementation they were compared against, and the behaviours that are worth knowing before using the API.
f4cc1de to
59961ba
Compare
4 participants
I wasn't able to reopen the old PR, so I'm resurrecting it here.
After using https://github.com/benma/descriptors-go in a mobile project, the limitations of using a WASM interpreter showed to be too restricting. So I'm giving it another try, this time with quite a bit of agentic help (I gave up on #1987 because I'm not smart enough for compiler-level stuff, but apparently Claude is quite capable).
Description
Depends on #2592.
Part 1 added a
miniscriptlibrary (originally authored by @benma in https://github.com/benma/miniscript-go) and this second part now adds full Output Descriptor support on top.The code is modeled after the
rust-miniscriptimplementation and a lot of test vectors for diferential unit tests were extracted from that implementation.The
descriptorsfeatures are currently limited to feature parity with https://github.com/benma/descriptors-go.Implementation disclaimer
The code in this PR is more or less a supervised re-implementation of
rust-miniscript. I spent a lot of tokens on differential code review and bug hunting, including large amounts of test vectors extracted from the original Rust code.There are also seeded fuzz tests that have been running for several hours on a 12-core machine.
All commits are hand-crafted and conform to the repository's code style. But apart from that, the code is not mine (I'd guess about 60% original miniscript code by @benma and the rest added by Claude Opus 4.8).