Skip to content

Test formatter changes against baseline format#89

Open
straight-shoota wants to merge 4 commits intomasterfrom
infra/test-formatter-base
Open

Test formatter changes against baseline format#89
straight-shoota wants to merge 4 commits intomasterfrom
infra/test-formatter-base

Conversation

@straight-shoota
Copy link
Copy Markdown
Member

For testing formatter changes we previously skipped code bases that were not already properly formatted because they would report formatter errors unrelated to the particular changes being tested.

But we actually don't need the codebase to be pre-formatted. We can run a baseline formatter first and then see if there are any changes against that.

This patch introduces a second Crystal compiler that serves as the baseline (defined by $CRYSTAL_BASE).
If a baseline compiler is available, we first run $CRYSTAL_BASE tool format and then only consider changes that $CRYSTAL tool format produces against that baseline.

CI installs the baseline compiler in /tmp/crystal-baseline/.

This allows us to run formatter tests against any code base, regardless of whether it's already well-formatted or not. This increases the potential surface area a lot.
At this point we can probably extract the individual formatter test definitions from the bats files and just iterate a long list of repository URLs.

@straight-shoota straight-shoota self-assigned this Mar 18, 2026
@straight-shoota straight-shoota added the enhancement New feature or request label Mar 18, 2026
@straight-shoota straight-shoota force-pushed the infra/test-formatter-base branch from 87d3063 to c492d82 Compare March 18, 2026 10:51
@ysbaddaden
Copy link
Copy Markdown

It seems to be broken?

@straight-shoota
Copy link
Copy Markdown
Member Author

No idea what's going on with athena. Looks like a bug in shards? 🤔

The failure in admiral.cr seems to be a formatter bug in the baseline which produces invalid code: crystal-lang/crystal#16755

@ysbaddaden
Copy link
Copy Markdown

Yeah, Shards is trying to fetch athena-framework/spec.git twice:

I: Fetching https://github.com/athena-framework/spec.git
...
I: Fetching https://github.com/athena-framework/spec.git
E: Failed to clone https://github.com/athena-framework/spec.git: Failed git clone -c core.askPass=true -c init.templateDir= --mirror --quiet -- https://github.com/athena-framework/spec.git /home/runner/.cache/shards/github.com/athena-framework/spec.git.
fatal: destination path '/home/runner/.cache/shards/github.com/athena-framework/spec.git' already exists and is not an empty directory.

Skip test due to ameba installation issue.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants