✨ feat: replace tombi with an in-repo TOML model

[gaborbernat]

tombi 1.5 deleted the mutable red-green tree this formatter was built on. tombi-syntax is gone, clone_for_update and splice_children no longer exist, and the new SyntaxNode exposes little past kind, span and text. Staying on 1.4.0 meant tracking a tree nobody else uses; upgrading meant rewriting the engine anyway. So this PR writes the engine and drops tombi.

What replaces it

toml-doc is a new crate: a format-preserving TOML document model with a mutation API. It parses with toml_parser, the lossless lexer and push parser behind toml_edit, which tracks TOML 1.1.0 and carries forbid(unsafe_code). The model above that event stream is ours.

Comments and blank lines lead the item below them, so document.sections.reverse() carries each header's comments along. A member owns the comma that follows it and the comment that closes its line, so a reorder leaves the separators alone and no comment lands on someone else's line. Unchanged text borrows from the source, so parsing allocates for structure alone.

common now sits on that model. Its passes are layout, sections, arrays, strings, nesting, spacing, disabled, settings, shape and build, each a walk that sets fields the document already carries. Nothing clones a tree to mutate it, re-parses between passes, or counts LINE_BREAK tokens to work out which entry a comment belonged to. common/src/{array,table,string,create,util,format_options}.rs are gone, the two formatters' ~40 tool modules now take fix(document: &mut Document<'_>), and a new tox-rules crate holds the tox policy both of them share.

The model holds its own invariants rather than trusting callers. Ws takes spaces and tabs, Comment takes text that opens with # and stays on one line, and a key segment takes a name rather than any token a value could carry. A pass that goes through the model's own editors cannot write a document no reader accepts.

Bug fixes

Each one reproduces against pyproject-fmt 2.28.2 and tox-toml-fmt 1.9.3 from PyPI.

1. A requirement written with a space after its operator read as no requirement at all. >= 3.12 did not parse as a version bound, so the generated classifiers fell back to the configured floor and ceiling. pypa/build, whose file says requires-python = ">= 3.10", got a 3.9 classifier it does not support.

requires-python = ">= 3.12" # before: 3.9, 3.10, 3.11, 3.12, 3.13
# after:  3.12, 3.13

2. The classifier window worked at series precision, so a bound below a patch release lost a whole series.

requires-python = "<3.10.1" # before: 3 :: Only, 3.9
# after:  3.9, 3.10

3.10.0 satisfies <3.10.1, so 3.10 belongs. Nothing pins the interpreter to Python 3 either, so 3 :: Only goes.

3. A literal string carried no sort key, so it held its place while the same value in double quotes sorted.

dependencies = ['zz', 'aa'] # before: [ "zz", "aa" ]
# after:  [ "aa", "zz" ]

4. A comment written before a member's comma moved to a line of its own, leaving the comma stranded below it. The comma is what says which member a comment belongs to: one written before it closes that member's line, one written after it leads the next member. Both travel with the member they belong to, and neither did before.

dependencies = [
  "b"
  # about b
  , "a",
]

Before, the comment took a line of its own and the comma that ends "b" took another:

dependencies = [
  "a",
  "b"
 # about b
 ,
]

After:

dependencies = [
  "a",
  "b", # about b
]

5. Writing classifiers as a string and asking for generated classifiers replaced the string with an array. The key holds text, not a list, so there is nothing to add to; the value stays as the file wrote it.

classifiers = "License :: OSI Approved :: MIT License"
# before: replaced by a generated array, losing the license classifier
# after:  left alone

6. Folding sub-tables into their parent depended on the order the file wrote them in.

[tool.x.b] # before: a.k = 2 then b.k = 1
k = 1 # after:  b.k = 1 then a.k = 2

[tool.x.a] k = 2 ```

**7. The key order missed pyrefly's own option spellings.** pyrefly documents hyphenated names, the order listed only the underscore forms, and a file using the documented spelling fell through to alphabetical order.

```toml
[tool.pyrefly] # before: project-includes, python-platform
project-includes = ["src"] # after:  python-platform, project-includes
python-platform = "linux"

8. A dependency group sorted its include-group entries away from the requirements around them. An include-group pulls its group in where it is written, so moving it changes what the group resolves to.

dev = ["z", { include-group = "b" }, "a", { include-group = "a" }]
# before: [ "a", "z", { include-group = "a" }, { include-group = "b" } ]
# after:  the order the file wrote

9. Free-form license text came back rewritten as though it were an SPDX expression. The formatter now rewrites the value only once it parses as an SPDX expression over registered identifiers.

license = "MIT or later" # before: "MIT OR later"
# after:  left alone

10. A tox deps list holding a pip option, path or URL sorted anyway. pip reads that list the way it reads a requirements file, where a later --index-url replaces the one before it, so the order carries meaning.

deps = ["zzz", "-r requirements.txt", "aaa"]
# before: [ "-r requirements.txt", "aaa", "zzz" ]
# after:  the order the file wrote

11. A tox set_env table came back alphabetized. tox reads the table in order, so a key written after file overrides what that file said and a key written before it does not.

set_env.Z = "1"        # before: A, file, Z
set_env.file = "a.env" # after:  Z, file, A
set_env.A = "2"

12. use_develop = true beside an existing package key dropped the use_develop and kept the other mode. tox reads use_develop first and installs an editable package whatever package says.

use_develop = true # before: package = "sdist"
package = "sdist"  # after:  package = "editable"

13. An environment whose name the file quoted never matched its env_list entry. [env."3.14"] is the environment env_list names as "3.14", and comparing the spelled key against the plain name never said so, so its table fell in with the ones the list does not name at all.

env_list = [ "3.14", "docs" ]

[env.docs]
[env."3.14"]
# before: docs first, then "3.14" among the environments the list does not name
# after:  "3.14" first, then docs, the order the list gives

14. A nested array closed its last member without the comma the outer form writes.

commands = [
  [ "a", "b" ]                       # before: no comma after "b"
]                                    # after:  "b",

15. The * catch-all in a setuptools data table was matched against the wrong spelling. * is not a name TOML reads bare, so the file writes it quoted and a rule matching it has to spell it the same way. The catch-all led the table only because a quote happens to sort before a letter.

Behaviour this changes on purpose

A string now measures from the start of its key. A long key can be what pushes a value past column_width. Measuring the value alone left lines running past the column the setting asks for.

[tool.x]
a-fairly-long-key-name-that-eats-the-column = """\
  and a value that together with the key runs past one hundred and twenty columns\
  """

Deep input comes back as an error rather than a crash. Reading a value, writing it and dropping it each walk it by calling themselves, so toml-doc caps nesting at 256 and the PEP 508 marker parser caps parenthesis depth at 256. A 12,000-deep value used to end the process.

Performance

The rewrite removes the quadratic walks the old tree forced. Both runs use an optimized wheel on the same machine.

Input	released	this PR
800 tox environments, each with one sub-table	18.6 s	0.12 s
32,000 interleaved root keys under two tables	618 s	4.8 s

Scaling is now close to linear: the tox case runs 5 ms / 7 ms / 33 ms / 120 ms at 100 / 200 / 400 / 800 environments.

Verification

1,214 Rust tests, run with cargo nextest at 100% line and region coverage on each of toml-doc, common, tox-rules, pyproject-fmt and tox-toml-fmt, measured the way each CI job measures it: every crate proves its own, rather than leaning on the tests of a crate that ships it. Every test lives beside src rather than inside it and reaches the crate the way a caller does; the two formatters build an rlib beside the cdylib the wheel needs so their tests can link against them. 94 Python tests for toml-fmt-common, 75 for pyproject-fmt, 54 for tox-toml-fmt.

toml-doc round-trips the 268 valid cases of toml-test and every TOML file in this repository byte for byte, and rejects every UTF-8 case TOML 1.1.0 calls invalid, with a floor in compliance.rs to keep that from slipping.

Every change formats the 38 files behind the projects at https://bernat.tech/oss/ and the 268 valid toml-test files, and compares both against a recorded baseline. Against 2.28.2 and 1.9.3, 27 of the 38 come out byte for byte the same; the other 11 carry the changes above and nothing else.

Running toml-test caught two defects on the way in. A lone \r came back out as \r\n, and a = with no value parsed clean, because toml_parser validates when a value decodes rather than when it parses. Decoding each key, scalar and trivia run at parse time fixes both.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (5659261) to head (002dec8).

Additional details and impacted files

Flag	Coverage Δ
common	100.00% <100.00%> (?)
pyproject-fmt	100.00% <ø> (?)
toml-doc	100.00% <ø> (?)
tox-rules	100.00% <ø> (?)
tox-toml-fmt	100.00% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
common/src/arrays.rs	100.00% <100.00%> (ø)
common/src/build.rs	100.00% <100.00%> (ø)
common/src/disabled.rs	100.00% <100.00%> (ø)
common/src/group.rs	100.00% <100.00%> (ø)
common/src/layout.rs	100.00% <100.00%> (ø)
common/src/lib.rs	100.00% <100.00%> (ø)
common/src/nesting.rs	100.00% <100.00%> (ø)
common/src/pep508/marker.rs	100.00% <100.00%> (ø)
common/src/pep508/requirement.rs	100.00% <100.00%> (ø)
common/src/pep508/version_op.rs	100.00% <100.00%> (ø)
... and 57 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.