Is there a concise description of the set of grammars that are accepted by the ANTLR4 tool ?

[chetmurthy]

I'm in the process of porting the (Python) ANTLR4 runtime to OCaml; I'm porting it in a manner that should ensure bug-for-bug compatibility with the Python runtime, as I'm running all the existing tests in the runtime-testsuite. I notice that there is no similar testsuite for the tool ? I'd like in the end to clone the tool in OCaml, so such a test suite (or even a textual description of what grammars are accepted) would be useful.

Does anybody know if such a thing exists? The tests in the tool-testsuite don't appear to be the thing I describe above. What would be nice, is a collection of tests somewhat like the runtime-testsuite, where each is a grammar plus whatever indications for special invocation, and then either it passes or (more important case) the error-message that is produced by the tool when it rejects.

I've some visualization of the ATNs produced by the tool, and they're pretty simple, so perhaps the answer is "any grammar that is syntactically valid, and for which an ATN can be generated, is accepted" ??

Any advice appreciated.

ETA: by "concise", I think I mean "more concise than the source code of the tool." Of course, in the end if there's nothing, I'm perfectly OK with just diving into the tool source code and figuring it out.

[kaby76]

The best test suite for end-to-end testing of an Antlr4 grammar is the grammars-v4 repo, which contains 359 grammars and in;puts. Of the grammars with actions/predicates, almost all are in target-agnostic format, which allows the grammar to be used for all targets. Most tests are for valid input, but there are also some invalid-input tests.

[chetmurthy]

Hello, I looked at the grammars-v4 collection back when I started; my first step was to run the antlr4 tool on every grammar in the collection, as well as on all the grammars in runtime-testsuite, and ..... they all passed (at least, that's my memory). By that I mean that the tool produced a ".interp" file containing the ATN. That seems to imply that every grammar in grammars-v4 is meant to be a -correct- (or "acceptable") grammar for ANTLR4.

What I was hoping for, is some collection of grammars that -fail- to be acceptable, for whatever reason (which the tool would hopefully report as an error-message) ?

For example, in runtime-testsuite there is "LexerErrors", containing grammars/inputs that elicit errors during lexing, and I needed to ensure that my ported runtime produced those same errors.

One might describe what I'm looking for as "negative tests" for the tool. Perhaps this simply doesn't exist?

ETA: Of course, one of the ways a grammar could fail to be acceptable, is if the ANTLR4 parser failed on it; I don't mean that, but rather that a grammar failed -semantic- checks after being parsed.

[ericvergnaud]

They exist in tool-testsuite
But that would be irrelevant for testing a runtime since such errors prevent generation of the parser...
The runtime test suite contains many error test cases. By adding your target in there, you would see them succeed or fail.

[chetmurthy]

Ah, thank you for pointing this out (I was remiss in not looking more closely, my bad). Indeed, I see that some of the unit-tests in tool-testsuite do create grammars with errors and then invoke the tool. I can replicate those tests!

Also, felt I should note that

(a) I'm already re-running the entire runtime-testsuite in Python, and have my ported runtime "passing" (same results) on all tests that only involve the lexer level ("lexer grammar"). Of course I'm moving on to the parser level next, and will work to get the entire runtime-testsuite passing in OCaml as it does in Python.

(b) But I also plan on writing a work-like for the antlr4 tool, so that I can then integrate antlr4 grammars into the OCaml language (via OCaml's macro-preprocessing facility). So it's necessary that I figure out how to test the tool, and what acceptable (and unacceptable) grammars are, for that tool.

Again, thank you for your pointer, and again, I was remiss in not digging further into the tool-testsuite to see those tests myself. I didn't see anything that looked like "test descriptor" files, and I mistakenly assumed that there must not be a large number of grammars being tested against the tool.

[ericvergnaud]

As an additional guidance, grammars-v4 is not in any way a test suite. Rather it's a set of publicly available grammars, supposedly correct, hopefully useful, community provided, not ANTLR supported.
@kaby76 has invested tremendous efforts in creating automation for testing them, but that does not directly affect the tool or the runtimes.

[kaby76]

As an additional guidance, grammars-v4 is not in any way a test suite. Rather it's a set of publicly available grammars, supposedly correct, hopefully useful, community provided, not ANTLR supported.

You view of grammars-v4 as just a collection of grammars of unknown quality, but I think that is a myopic view. CI and weekly testing for grammars-v4 validates parse trees, error messages, performance, ambiguity, and various coding styles, against 4.13.2 (or 1), the "dev" branch of antlr4 (it's built weekly with artifacts cached). the antlr-ng generator, and across most targets (and soon, a complete replacement for Antlr in Trash). We don't just accept any change to any grammar in the repo: it has to "work" against the parse input. Some of the grammars are excellent and have huge test inputs. the java/java/ grammar is tested against the JDK version 24. The ada grammars are tested against the official ADATS. Performance is now posted for every grammar (https://htmlpreview.github.io/?https://raw.githubusercontent.com/kaby76/grammars-v4/refs/heads/g4-4933/performance.html).