Spec Compliance Suite

[sinclairzx81]

Hello!

This is just a quick suggestion to consider including 3 additional benchmarks that detail specification compliance for JSON Schema and JSON Schema–adjacent validators. Such benchmarks would be helpful to independently track specification adoption and compliance work happening within the ecosystem.

Such compliance work has some relation to upstream IETF and ECMA standardization efforts.

---

Prior Art

The proposed tests would be derived from the following compliance suite.

https://github.com/sinclairzx81/json-schema-compliance-suite

---

Benchmarking

https://github.com/json-schema-org/JSON-Schema-Test-Suite

It should be possible to vendor the official test suite (link above) and enumerate each .json test document. During enumeration, the draft, schema, and data values can be extracted per test and sent to a single test function which returns a boolean result.

Example

import { Validator } from 'xyz'

// draft, schema and data originate from .json test documents
complianceTest((draft, schema, data) => {
   const validator = new Validator(draft)
   return validator.check(schema, data) // boolean
})

Notes

• A full test needs only count the correct boolean results for each invocation (across all drafts). Accumulated counts can then be used to indicate a statistical likelihood that a validator is conforming to specification (noting some potential for error given boolean false positives).
• The test suite has enough information to be able to identify which parts of the specification a validator may be failing (see prior art), so providing features that allow an implementer to drill down into the benchmark would be really helpful ... this is a nice-to-have though, not a requirement.

---

Categories

The following are the 3 candidate benchmarks for consideration:

Test	Description
Validation	JSON Schema is passed directly to the library and used for validation. Results indicate compliance with the official JSON Schema test suite.
Semantics	The JSON Schema is decoded into library types and validated using the library's internal logic. Results indicate how closely the library aligns with the formal semantics of JSON Schema.
RoundTrip	JSON Schema is decoded then re-encoded using the library's provided translation API. The translated JSON Schema is then passed to CfWorker for validation. Failed tests indicate either lossy translation or a serialization error.

See link below for a reference of how these tests could be implemented

https://github.com/sinclairzx81/json-schema-compliance-suite/blob/main/tests.ts

Updating

The official test suite is frequently updated by specification implementers and core spec contributors, so the benchmarks should consider souring the official test suite on CI nightlies to get auto-updates before running a daily test.

---

For consideration