[DNM] Speedy Rust port of the Linux-facing tools

[quic-kdybcio]

Processed schema creation and DTB validation are huge bottlenecks in the current kernel maintainer and contributor workflows. This is particularly noticeable when doing work on a machine that's not your main beefy workstation.

In an attempt to remedy these pains, I wanted to explore the room for improvement here. With "a little" help from a robot, this is the final result when validating all 3,873 DTBs and 269 DTBOs present in the Linux kernel (as of some recent -next, for architectures that LLVM supports):

+-------------------------------+------------------------+------------------------+---------+
| Tool / workflow               | Python                 | Rust                   | Speedup |
+-------------------------------+------------------------+------------------------+---------+
| dt-mk-schema -j Linux bindings| 68.18s                 | 2.49s                  | 27.4x   |
| dt-validate all arch DTBs     | 1:27:53                | 23.41s                 | 225.2x  |
| dt-doc-validate Linux bindings| 3:16.29, exit 1        | 33.05s, exit 1         | 5.9x    |
| dt-check-compatible 5k strs   | 1.45s                  | 2.57s                  | 0.6x    |
| dt-extract-example 500 files  | 1:09.92                | 1.75s                  | 40.0x   |
+-------------------------------+------------------------+------------------------+---------+

(Running on a Ryzen 6850U in WSL)

As another point of reference, validating all of the 399 qcom DTBs only takes around 2-3s.

The resulting JSON output is equivalent (not necessarily byte-for-byte identical, but equivalent) to the Python reference. The stderr output, similarly, is equivalent (modulo how rust vs python implement Display/repr for certain types).

The Rust implementation of dt-validate is meant to be run as a single process against an N-long argument list. Parallelism is taken care of through the rayon crate.

The question remains, what should we do with this - dumping 6.5k LoC at you and saying "have fun" is obviously missing the point, but particularly the dt-mk-schema and dt-validate speedups are to the point where we could integrate this into an LSP, so I really believe we should think about integrating this seriously.

cc @robherring @krzk @quic-bjorande

[robherring]

Nice!! How do I run this?

[quic-kdybcio]

cd rust && cargo build --release will build all the binaries (assuming you have a non-ancient Rust installation - get one from rustup.rs if you don't)

Each tool is a drop-in replacement for the existing ones, so you should be good to just swap out the bin.

For Linux, I used the following hunk. Unfortunately(?) we lose the as-we-go prints with the CHECK make tag, as the tool just crunches everything in a single invocation.

Also, for completeness: make ARCH=arm64 LLVM=1 DT_MK_SCHEMA=~/dt-schema/rust/target/release/dt-mk-schema DT_CHECKER=~/dt-schema/rust/target/release/dt-validate dtbs_check

diff --git a/Makefile b/Makefile
index fc2d94aafb45..47462fcd9d8d 100644
--- a/Makefile
+++ b/Makefile
@@ -1607,11 +1607,24 @@ ifneq ($(filter dtbs_check %.yaml, $(MAKECMDGOALS)),)
 export CHECK_DTBS=y
 endif
 
+ifneq ($(CHECK_DTBS),)
+DT_CHECKER ?= dt-validate
+DT_CHECKER_FLAGS ?= $(if $(DT_SCHEMA_FILES),-l $(DT_SCHEMA_FILES),-m)
+endif
+
 ifneq ($(CHECK_DTBS),)
 dtbs_prepare: dt_binding_schemas
 endif
 
+quiet_cmd_dtbs_check = CHECK   $(dtstree)
+      cmd_dtbs_check = f=$$(mktemp); \
+	grep '\.dtb$$' $(objtree)/$(dtstree)/dtbs-list > $$f; \
+	$(DT_CHECKER) $(DT_CHECKER_FLAGS) -u $(srctree)/$(dtbindingtree) \
+		-p $(objtree)/$(dtbindingtree)/processed-schema.json @$$f || true; \
+	rm -f $$f
+
 dtbs_check: dtbs
+	$(call cmd,dtbs_check)
 
 dtbs_install:
 	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.dtbinst obj=$(dtstree)
diff --git a/scripts/Makefile.dtbs b/scripts/Makefile.dtbs
index c4e466390284..251f929fbfed 100644
--- a/scripts/Makefile.dtbs
+++ b/scripts/Makefile.dtbs
@@ -78,7 +78,7 @@ dtb-check-enabled = $(if $(filter %.dtb, $@),y)
 endif
 
 quiet_dtb_check_tag = $(if $(dtb-check-enabled),[C],   )
-cmd_dtb_check = $(if $(dtb-check-enabled),; $(DT_CHECKER) $(DT_CHECKER_FLAGS) -u $(srctree)/$(DT_BINDING_DIR) -p $(DT_TMP_SCHEMA) $@ || true)
+cmd_dtb_check =
 
 # Overlay
 # ---------------------------------------------------------------------------
-- 
2.54.0

[robherring]

Humm, A large part of python's slowness is the startup time. That's somewhat mitigated on dt-doc-validate, but not the examples or dtbs. So we could speed up python quite a bit doing the same thing, but I think keeping per target rules is important.

Note I just tried out jsonschema-fast last week which uses a rust implementation for validation, but is a drop-in replacement for the python jsonschema module. It was not any faster...

[robherring]

I have 2 thoughts on the general direction for how we would merge/support this.

As the schemas and meta-schemas are the only shared part, should we split them out to a separate repo? Then the python and rust implementations can both include it. I suppose we'd have to keep something to test them.

Can we make the python executables be thin wrappers to the rust implementation? Then we're not maintaining 2 implementations. This would allow users to just keep doing pip install though you'd still be paying the python startup time penalty (should be a bit less avoiding all the module imports). The rust dependency should be pretty transparent because we already have that dependency indirectly with the referencing module. Need to look at it, but I imagine it's calling cargo from it's pip install.

On further thought, maybe better to just put all this in a new repo without any of the python baggage and effort to wrap rust code. Telling folks to run cargo instead of pip isn't that difficult. Hopefully we're beyond having rust as an obstacle. Though I did find this didn't work with debian stable which has rustc 1.85 though there is a 1.95 backport available.

[robherring]

Looks like rust startup is slower than python?

For single example dtb, rust is 4x slower:

$ time dt-validate -s .build-arm64/Documentation/devicetree/bindings/processed-schema.json .build-arm64/Documentation/devicetree/bindings/watchdog/airoha,en7581-wdt.example.dtb

real    0m2.064s
user    0m1.961s
sys     0m0.032s
$ time ~/.local/bin/dt-validate -s .build-arm64/Documentation/devicetree/bindings/processed-schema.json .build-arm64/Documentation/devicetree/bindings/watchdog/airoha,en7581-wdt.example.dtb

real    0m0.521s
user    0m0.496s
sys     0m0.012s

For single large dtbs, rust is about 2x faster with 2 vs 4 sec. There seems to be a 2 sec fixed overhead that needs to be investigated.

[robherring]

I (well, codex did) fixed the slow startup time: https://github.com/robherring/dt-schema.git rust-port

A reimplementation of the cache (aka processed-schema.json) gives another incremental improvement. Looks like dt_binding_check is 3-4x faster and arm64 dtbs_check is ~15x faster.