feat: BCS ABNF compile-time generation by thibault-martinez · Pull Request #1040 · iotaledger/iota-rust-sdk

thibault-martinez · 2026-03-25T19:05:39Z

Summary

This PR introduces a machine-readable BCS schema for iota-sdk-types, expressed as an ABNF grammar file, along with the tooling to generate and validate it.

What's included

crates/iota-bcs-schema — a new proc-macro crate providing a #[derive(BcsSchema)] macro. When applied to a type, it emits a bcs_schema_definition() method that returns the type's ABNF grammar fragment. The macro handles structs, enums, and common generic wrappers (Option<T>, Vec<T>, HashMap<K, V>), and supports override attributes (#[bcs_schema(name = "...", definition = "...")] on types and #[bcs_schema(as_type = "...")] on fields/variants) for cases where the derived output needs manual tuning.
crates/iota-sdk-types/bcs-schema.abnf — the auto-generated ABNF schema covering all SDK types (transactions, effects, checkpoints, crypto primitives, etc.). It is committed to the repo and regenerated on demand via BCS_SCHEMA=1 cargo check -p iota-sdk-types --features bcs-schema.
crates/iota-sdk-types/build.rs — a build script that, when BCS_SCHEMA=1 is set alongside the bcs-schema feature, deletes the old schema file and touches all .rs sources to force every #[derive(BcsSchema)] to re-run, producing a fresh schema from scratch.
Grammar-driven fuzz tests (tests/bcs_schema_fuzzing.rs) — a test suite that parses bcs-schema.abnf, randomly generates conforming byte sequences for each rule, and asserts that the BCS deserializer accepts them. This validates that the grammar is a sound description of the wire format.
CI — a new lint workflow job that regenerates the schema and fails if the committed file is out of date.

Motivation

Having a formal, human- and machine-readable description of the BCS wire format makes it easier to:

build cross-language BCS parsers/validators without reimplementing the Rust types,
catch accidental serialization-breaking changes at the schema level, and
fuzz the deserializer against grammar-valid inputs.

Usage

# Regenerate the schema file
BCS_SCHEMA=1 cargo check -p iota-sdk-types --features bcs-schema

# Normal development — no impact
cargo check --all-features

Derive attributes

Attribute	Level	Purpose
`#[bcs_schema(definition = "32OCTET")]`	Type	Override entire RHS (for `[u8; Self::LENGTH]`, etc.)
`#[bcs_schema(name = "custom-name")]`	Type	Override the ABNF rule name
`#[bcs_schema(as_type = "u64")]`	Field	Override schema type (for type aliases like `Version = u64`)
`#[bcs_schema(as_type = "call-arg")]`	Field	Reference a wire-type that differs from the Rust type
`#[bcs_schema(skip)]`	Field	Omit from schema

Known limitations

as_type and definition values are trusted — no validation against the actual serde impl
Custom Serialize impls are invisible to the macro; divergences must be annotated manually
Serde attributes (skip, rename, flatten) are not read by the macro

Test plan

cargo check -p iota-sdk-types --all-features — no regressions
BCS_SCHEMA=1 cargo check --features bcs-schema produces complete, sorted bcs-schema.abnf
Two consecutive BCS_SCHEMA=1 builds produce byte-identical output
Removing BcsSchema from a field type causes a compile error
cargo check without the feature is unaffected (fully incremental)

thibault-martinez · 2026-04-08T11:55:28Z

 #[cfg_attr(feature = "schemars", derive(schemars::JsonSchema))]
 #[cfg_attr(feature = "proptest", derive(test_strategy::Arbitrary))]
+#[cfg_attr(feature = "bcs-schema", derive(iota_bcs_schema::BcsSchema))]
+#[cfg_attr(feature = "bcs-schema", bcs_schema(definition = "%x60 96OCTET"))]


can these be merged? If yes, do it for all instances please

Thoralf-M

We should add a README.md in iota-bcs-schema/ which explains what this is good for and how to use it

…t-sdk into bcs-abnf-poc

thibault-martinez · 2026-04-09T12:22:18Z

+#[cfg_attr(
+    feature = "bcs-schema",
+    derive(iota_bcs_schema::BcsSchema),
+    bcs_schema(definition = "%x60 96OCTET")


Can you check if "OCTET" is the standard?

thibault-martinez · 2026-04-09T12:24:40Z

+#[cfg_attr(
+    feature = "bcs-schema",
+    derive(iota_bcs_schema::BcsSchema),
+    bcs_schema(definition = "%x60 96OCTET")


Can we have it in decimal?

thibault-martinez · 2026-04-09T12:28:39Z

+#[cfg_attr(
+    feature = "bcs-schema",
+    derive(iota_bcs_schema::BcsSchema),
+    bcs_schema(definition = "bytes")


What is bytes? Is it defined in the ABNF? length-prefixed?

There are now definitions added to the file for these basic types

thibault-martinez · 2026-04-09T12:34:51Z

+#[cfg_attr(
+    feature = "bcs-schema",
+    derive(iota_bcs_schema::BcsSchema),
+    bcs_schema(definition = "string")


Properly define string in the ABNF?

thibault-martinez · 2026-04-09T12:36:16Z

+#[cfg_attr(
+    feature = "bcs-schema",
+    derive(iota_bcs_schema::BcsSchema),
+    bcs_schema(name = "type-tag")


What would happen if I created a different type with the same BCS schema name? Can we make sure it fails?

There is now duplicate detection

thibault-martinez · 2026-04-09T12:36:58Z

+#[cfg_attr(
+    feature = "bcs-schema",
+    derive(iota_bcs_schema::BcsSchema),
+    bcs_schema(definition = "32OCTET")


Can we put a space? I always read it as 320 :(

I could add something like byte = OCTET so it would look like 32byte instead?

thibault-martinez · 2026-04-09T12:42:56Z


    #[derive(serde::Serialize, serde::Deserialize)]
+    #[cfg_attr(feature = "bcs-schema", derive(iota_bcs_schema::BcsSchema))]
+    #[cfg_attr(feature = "bcs-schema", bcs_schema(name = "genesis-object"))]


thibault-martinez and others added 7 commits March 25, 2026 20:03

feat: BCS ABNF PoC

1bb2139

fix TypeTag order

d6890ca

add new types

04a7d2d

execution-status order

c1f99c9

feat: Add grammar driven fuzzing

5af3bb4

dprint and clippy

0d80353

fmt

6c39a1d

thibault-martinez commented Apr 8, 2026

View reviewed changes

Comment thread crates/iota-sdk-types/src/effects/v1.rs

thibault-martinez commented Apr 8, 2026

View reviewed changes

Comment thread crates/iota-bcs-schema/Cargo.toml Outdated

DaughterOfMars added 3 commits April 8, 2026 11:43

improve definitions and tests

c0ea8bb

consistent comments

3ce4103

Remove trait

e0b64c5

DaughterOfMars marked this pull request as ready for review April 8, 2026 10:14

DaughterOfMars requested a review from a team as a code owner April 8, 2026 10:14

DaughterOfMars added 3 commits April 8, 2026 13:02

licenses and stuff

7a90190

add CI job

f3c5be6

Merge branch 'develop' into bcs-abnf-poc

d4d12ad

thibault-martinez commented Apr 8, 2026

View reviewed changes

Thoralf-M reviewed Apr 8, 2026

View reviewed changes

Comment thread crates/iota-bcs-schema/src/lib.rs Outdated

thibault-martinez changed the title ~~feat: BCS ABNF PoC~~ feat: BCS ABNF compile-time generation Apr 8, 2026

Alex6323 reviewed Apr 8, 2026

View reviewed changes

Comment thread crates/iota-bcs-schema/src/lib.rs

Alex6323 reviewed Apr 8, 2026

View reviewed changes

Comment thread crates/iota-bcs-schema/src/lib.rs

Alex6323 reviewed Apr 8, 2026

View reviewed changes

Comment thread crates/iota-bcs-schema/src/lib.rs

Alex6323 reviewed Apr 8, 2026

View reviewed changes

Comment thread crates/iota-bcs-schema/Cargo.toml

Thoralf-M reviewed Apr 8, 2026

View reviewed changes

Comment thread crates/iota-sdk-types/src/effects/v1.rs Outdated

Thoralf-M reviewed Apr 8, 2026

View reviewed changes

Comment thread crates/iota-sdk-types/src/transaction/mod.rs

Thoralf-M reviewed Apr 8, 2026

View reviewed changes

Comment thread crates/iota-sdk-types/src/type_tag/mod.rs Outdated

Thoralf-M reviewed Apr 8, 2026

View reviewed changes

Comment thread crates/iota-sdk-types/src/object.rs

Thoralf-M reviewed Apr 8, 2026

View reviewed changes

nits

c6d1387

Merge branch 'bcs-abnf-poc' of https://github.com/iotaledger/iota-rus…

bfa5002

…t-sdk into bcs-abnf-poc

thibault-martinez commented Apr 8, 2026

View reviewed changes

Comment thread crates/iota-sdk-types/bcs-schema.abnf Outdated

thibault-martinez commented Apr 8, 2026

View reviewed changes

Comment thread crates/iota-sdk-types/bcs-schema.abnf

Thoralf-M reviewed Apr 8, 2026

View reviewed changes

Comment thread crates/iota-sdk-types/bcs-schema.abnf Outdated

Thoralf-M reviewed Apr 8, 2026

View reviewed changes

Comment thread crates/iota-sdk-types/bcs-schema.abnf Outdated

DaughterOfMars added 3 commits April 9, 2026 09:03

const file name

51d5d63

remove duplicates

173738d

add env check to proc macro

e9df75a

thibault-martinez commented Apr 9, 2026

View reviewed changes

thibault-martinez linked an issue Apr 9, 2026 that may be closed by this pull request

Rethink BCS compatibility check iotaledger/iota#10513

Open

DaughterOfMars added 7 commits April 9, 2026 15:45

replace hex with decimal

12b7f9c

conform to rfc5234

87355a5

disallow duplicate names

6361fba

combine

a1fd82d

fix duplication detection

915ea7c

fmt

7907d6c

update bindings

a436701

Conversation

thibault-martinez commented Mar 25, 2026 • edited by DaughterOfMars Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's included

Motivation

Usage

Derive attributes

Known limitations

Test plan

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Thoralf-M left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

thibault-martinez commented Mar 25, 2026 •

edited by DaughterOfMars

Loading