engine: add Rest-SSZ spec

[MariusVanDerWijden]

There have been multiple attempts at this already.
Moving away from JSON-RPC to REST-SSZ.
However most kept the engine api as is.
I think we have a good shot at refactoring the engine api with this change.

This Draft does that; the move and the refactoring.

Happy for any feedback I can get!

The core of the change is:

Old method	New endpoint	Notes
engine_newPayloadV{1..5}	POST /{fork}/payloads	parentBeaconBlockRoot and executionRequests folded into the SSZ envelope; expectedBlobVersionedHashes removed; INVALID_BLOCK_HASH removed from the status enum
engine_forkchoiceUpdatedV{1..4}	POST /{fork}/forkchoice	one atomic call; carries forkchoice state, optional payload_attributes, and (Amsterdam+) optional custody_columns
engine_getPayloadV{1..6}	GET /{fork}/payloads/{id}	poll-style, same semantics as today
engine_getPayloadBodiesByHashV{1,2}	POST /{fork}/bodies/hash	{fork} selects the response schema (not the era of requested blocks); POST because hash lists are too large for URLs
engine_getPayloadBodiesByRangeV{1,2}	GET /{fork}/bodies?from=...&count=...	{fork} selects the response schema
engine_getBlobsV1	POST /blobs/v1	independently versioned; legacy version numbers carry forward
engine_getBlobsV2	POST /blobs/v2	all-or-nothing cell proofs
engine_getBlobsV3	POST /blobs/v3	partial-response cell proofs
engine_getBlobsV4	POST /blobs/v4	cell-range selection
engine_getClientVersionV1	GET /identity + X-Engine-Client-Version request header	unscoped
engine_exchangeCapabilities	GET /capabilities	unscoped
engine_exchangeTransitionConfigurationV1	removed	already deprecated since Cancun

[arnetheduck]

One thing that I think would make sense would be to reuse the base structure of the beacon api (https://github.com/ethereum/beacon-APIs/) - this includes several things:

• primitives and their json/ssz encoding - ie in general, where possible, reuse the types from https://github.com/ethereum/beacon-APIs/blob/master/types/primitive.yaml so that we don't end up with pointless minor differences in how for example a number is string-encoded (0x0 vs 0x and the like)
• execution payloads and other (consensus) spec types - the "shape" of objects in the beacon api generally follows the shape of things as they travel on the gossip network and their SSZ encoding - by reusing these types, we would reduce the maintenance overhead of having to pointlessly reorder and rename the exact same fields from the beacon api/consensus spec just to send the same info to the execution api in a slightly different shape
• use of the canonical ssz/json encodings specified here: https://github.com/ethereum/consensus-specs/blob/master/ssz/simple-serialize.md#json-mapping - this aids debugging and removes the need to double-specify things
• explicit encoding of fork in the http headers -> we can then upgrade to an new hard fork "automatically" without having to come up with V2, V3 etc

[arnetheduck]

The core of the change is:

For top-up sync we also need "current block number", similar to eth_blockNumber but limited to latest and with a well-defined behavior for when the EL does not have a state.

[developeruche]

I opened #773 a few weeks ago with a narrower scope: adding a single new endpoint (POST /new-payload-with-witness) that combines engine_newPayload and debug_executionWitness into one call and returns the witness SSZ-encoded over HTTP. The motivation was to unblock zkVM provers and zkAttestors from having to follow the chain one block behind.

Since #793 is now doing a full Engine API refactor with the same REST+SSZ foundation, I think the witness endpoint fits naturally into this design. A few thoughts:

The witness endpoint should be added to this new spec. The existing two-call flow (engine_newPayload → debug_executionWitness) has real-world latency problems at a ~500 MB witness, the JSON-RPC + JSON approach takes ~8s just to return the witness. With HTTP + SSZ and the EL pipeline optimizations I profiled (moving trie writes off the critical path, parallelizing storage trie updates), this drops to ~932ms total EL time. That's comfortably within the 8s newPayload timeout even for worst-case blocks.

Suggested endpoint: POST /{fork}/payloads/with-witness (or folded directly into POST /{fork}/payloads as an optional response field when requested via a query param or Accept header). The response would carry the PayloadStatus + ExecutionWitness SSZ-encoded, consistent with the rest of the new spec.

Benchmark data (ethrex, 203 txs, 36 Mgas, ~502 MB SSZ witness):

Approach	EL Total	Wire Size
JSON-RPC + JSON	8,131 ms	~502 MB
HTTP + SSZ	1232 ms	502 MB

Happy to close #773 in support of this PR I think it's a better, more seamless flow. Would love to discuss where the witness endpoint fits best in the new endpoint table.

cc: @MariusVanDerWijden

[flcl42]

dropping /engine/v2 prefix misleads a bit

[flcl42]

What does v2 mean? Was there v1? Why is fork necessary in the URL? Fork name seems to be a way to describe minor API version, but on other side blobs endpoints have it after resource name, not before and it's a number

[flcl42]

Also if payload did not change across forks, does it mean we will need same endpoint under different urls? Just single /vN/ looked simpler

[arnetheduck]

the consensus REST spec does not have this requirement - we use 1.1 throughout and going to 2.0 would not be viable short-term for Nimbus.