Add Simula

[RobertFlexx]

adds simula support to linguist, including syntax highlighting, .sim detection, samples, and a language color.

description

this adds simula as a language in linguist.

the grammar covers simula 67 and newer simula-style syntax. free simula is covered as a dialect of simula rather than being added as a separate language.

.sim is used by quite a few unrelated formats, so it is now registered as a generic extension and identified using content heuristics instead of being mapped directly to simula.

while testing the heuristics against real .sim files i found two actual false-positive cases.

mcstas/mccode result files can contain simulation and a later begin on separate lines, which an earlier \s+ pattern treated like simula's Simulation Begin. that signature now requires them to be on the same line.

pascal files can also use the same basic PROGRAM name; BEGIN shape. that rule now requires additional simula-specific syntax before it can classify the file as simula.

i checked the final rules against 677 .sim files from github. 439 were simula sources and 425 were detected. none of the 238 non-simula files in the set were classified as simula.

the 14 missed simula files are old classless programs without enough distinctive syntax to separate them safely from unrelated .sim files. matching those would mean bringing back much broader begin + declaration patterns, so i left them unmatched rather than increasing false positives.

the full linguist test suite passes:

• 2024 runs
• 40476 assertions
• 0 failures
• 0 errors

the simula heuristic test also passes with 78 assertions.

checklist

• i am adding a new language.

  • the extension is used in real simula projects on github.

    • usage search:

      • https://github.com/search?q=NOT+is%3Afork+path%3A%2A.sim+%22ref%28%22+%22begin%22&type=code

  • i have included real-world samples and linked them to their exact sources.

    • GNU CIM simset.sim

      • source: https://ftp.gnu.org/gnu/cim/cim-5.1.tar.gz
      • license: GPL-2.0
      • the original license notice is retained

    • free simula sources

      • source: https://github.com/RobertFlexx/Free-Simula
      • license: MIT

    • add the exact source URLs for the remaining retained real-world samples here after trimming the sample set

  • i have included a syntax highlighting grammar:

    • https://github.com/RobertFlexx/simula-tmlanguage

  • i have added a color:

    • #4f78a8

• i am adding new or changing current functionality

  • .sim is included in the generic extensions and differentiated using heuristics
  • positive and negative heuristic fixtures are included
  • the full test suite passes

[lildude]

We do not need so many samples, especially those that are not real world examples of the language. Please cut down the samples to only real examples and update the PR template to point to the exact sources.

Additionally, your search results indicate usage doesn't meet our requirements yet, but the inverse does… ie there are significantly more .sim files that do not match your search than do. This means they'll all be misidentified. To prevent this, you will also need to identity the most common user of that extension and add support as part of the PR with heuristics to differentiate the two languages.

[RobertFlexx]

i have finally addressed this now. i cut the sample set down to real-world examples and updated the template with the exact sources.

i also moved .sim into the generic extensions and added heuristics instead of treating every .sim file as simula. testing turned up two real collision cases, mcstas/mccode output and pascal PROGRAM name; BEGIN files, so i tightened those rules and added regression fixtures for both.

i checked the final rules against 677 live .sim files: 425/439 simula files were detected, and none of the 238 non-simula files were classified as simula. the full test suite is passing as well.

[RobertFlexx]

finished changes

[lildude]

If you're going to go the generic route, we only need samples for the language you're adding. I'm not sure we need such a complex heuristic either. Several of your regexes are also vulnerable to ReDoS attack.

Several of your Simula samples still look like tutorial type illustrations of features of the language and not real-world code.

[RobertFlexx]

Thanks, I've pared this down quite a bit. The generic fixture set now only contains two sourced Simula programs. I removed the unrelated .sim fixtures and the tutorial-style samples, and replaced the heuristic with two simple linear patterns. The full test suite passes. Could you take another look when you have a chance?

[RobertFlexx]

finished (hopefully)

[lildude]

This is over-engineered. Use the same approach as other generic language tests, like test_sol_by_heuristics.