Make API docs programmatically parsable

[mark-wiemer]

Currently, the best API docs are in lua_api.md. However, those docs don't strictly follow any schema aside from "valid Markdown." They are easy to read as people, but hard to transform into other docs like a TypeScript API or something for a language server to provide IntelliSense.

To make the docs parsable, we need two things:

• An agreement on a consistent format to present the API (how to present function names, arg names, arg types, descriptions, tables, etc.)
• Content that aligns to that agreement

For the consistent format, there are multiple approaches:

• defining a stricter Markdown schema for lua_api.md and enforcing it via CI
• adding comment headers to the actual engine code and parsing that
• other approaches?

From there, we'd need to update the content no matter what choice we have.

Once the API docs (or, at least the core namespace reference) are programmatically parsable, we can focus on providing a Language Server plugin (e.g. a LuaLS addon) so that modders can get insights while editing in an IDE instead of waiting to try to run their mod. However, this hasn't been discussed so I won't create a new issue for it.

Related issues:

• Provide officials Lua API headers for LuaLS luanti#13926
• Consider autogenerating bindings (and/or serde) luanti#16685

Related efforts:

• https://github.com/corpserot/luanti_lsp_definitions/
• https://git.minetest.land/archie/luanti-api/ (gives 404 to unauth'd users)
• mark-wiemer/hello-hello@main/packages/luanti/doc-schema

Considered approaches (basic thoughts noted, Markdown accepted, see comments and #315):

• EmmyLua: still Lua, decent underlying parser, development recently picked up. Not expressive enough.
• JSON: tons of tooling support. No clear schema, hard to handwrite.
• LuaCATS: still Lua, has decent underlying parser. Not expressive enough, underlying parser development stalling
• Markdown: very expressive, tons of tooling support. No clear schema yet defined, no parser built.
• Teal: still Lua, more type support. Underlying development stalling
• TOML: not explored
• TypeScript: extremely complex types supported, tons of tooling support. Not Lua, not used elsewhere in Luanti repo
• XML/XSD: not explored, would be very verbose to handwrite. XSD = XML Schema Definition
• YAML: tons of tooling support. No clear schema, can be easy to make mistakes

[mark-wiemer]

@appgurueu asked in #297 about why we didn't go with "adding comment headers to the actual engine code and parsing that", and I'll explain that:

• we'd be starting from a smaller base (existing Lua comments generally written for engine devs vs existing lua_api.md written for modders)
• we'd need core devs buy-in
  • we've had resistance to the idea of verbose method header comments before
• we'd need to parse and organize content from a wide variety of files instead of one central file

As such I'm pursuing a Markdown schema instead. Happy to look at Lua comments if these concerns can be addressed

[mark-wiemer]

@appgurueu has mentioned parsing a Lua-based DSL in #297. @appgurueu would that mean rewriting lua_api.md?

You also say Markdown is primarily human-readable, but not machine readable, and I disagree. Markdown turns into HTML, after all. A strict Markdown schema could enforce additional machine-level requirements like "all args need types." I don't think it's a backwards approach to create a strict Markdown schema--Markdown itself is already a strict TXT-to-HTML schema, why not make a stricter TXT-to-Luanti-docs schema in the same manner?

[mark-wiemer]

Again, one of the things I want to avoid here is rewriting lua_api.md. That is a huge effort on its own, I'd rather transform the existing Markdown into something better than start over with something nobody on the team has used. Who would do this rewrite? How long would it take? What would we gain over a Markdown schema and improving the existing docs?

[mark-wiemer]

Current problem: When writing code for a Luanti mod or game, I don't have info about the Luanti API in my IDE

Current workaround: Use Luanti Tools for VS Code, which hasn't been updated since Luanti 5.11 (Feb 2025. We are in 5.14 as of November 2025)

• Provides several small features beyond core namespace info
  • new project boilerplate
  • luacheck integration
  • supports docs outside of core namespace reference
• Maintained by GreenXenith, who has complained about the high cost of maintenance
• Current control flow for Luanti Tools is lua_api.md -> highly custom transformation code with regex -> smartsnippets.json -> VS Code extension
  • Does not solve core problem: Still missing any info not present in lua_api.md
  • Does not use industry standard LSP: Instead uses custom code and VS Code API
  • Maintenance of highly custom code is expensive, esp. compared to using an existing AST parser like remark
  • Extension has minimal unit tests or automated validations

Considerations for a solution:

• For doc writers to create and maintain it:
  • Similar to existing lua_api.md for ease of adoption
  • Very lightweight for adoption and long-term maintenance
    • Including comments :)
  • Familiar to those writing Lua code
  • Easy to validate whether updated docs are correct or not
  • Do not need to rewrite lua_api.md by hand
  • Should not need to manually do backlinking or other easily-automated tasks
• For end-users to use it:
  • Available in multiple IDEs (at least: VS Code, Zed, emacs, vim)
  • Covers entire core namespace reference
  • Includes sample code with sample output (including hover text and squiggly underlines) to demonstrate most core features
  • Works without any additional setup on most machines
    • Linux Mint or similar "simple" Linux distro, Windows, macOS
    • Shouldn't need to install "luacheck" files or other "random" files to work

Proposed solution: Enforce strict "Markdown schema" and integrate with LuaLS

1. Define a Markdown schema that allows doc writers to easily include additional info in a consistent format
   • Additional info fields
     • Function arg types
     • Function return values
     • Sample usage
     • ...more, probably
   • Consistent format
     • Should be lightweight (closer to TOML than XML, minimal "markup" characters)
     • Should match or nearly match existing Lua ecosystem formats (EmmyLua, LuaLS)
2. Publish a Markdown parser to validate and enforce the consistent format
   • Doc writers can run e.g. node parse-core-ref.js while in the folder of a draft version of lua_api.md to get results
   • Parser should ignore anything outside the "core namespace reference" section
   • Include informative error messages
     • Start and end position of problematic characters
     • Expected token type (e.g. arg type, return type)
   • Early POCs are using remark, a JS package. See Luanti docs chat for details.
3. Update lua_api.md#core-namespace-reference to include additional info in a consistent format
   • If additional info isn't known, it can be left blank or explicitly marked as "unknown" in some way
4. Add a status check to PRs in the luanti repo to enforce new Markdown schema
   • GitHub Actions workflow that uses the Markdown parser and a comment action in case of errors
5. Transform lua_api.md#core-namespace-reference into a LuaLS definition file
6. Publish the add-on under LLS-Addons
7. Validate that everything works
8. Share with the world :)

[mark-wiemer]

• What about the rest of lua_api.md?
• What about the docs site?

[corpserot]

from the conversation in IRC: https://irc.luanti.org/luanti-docs/2025-12-01#i_6301136

these are my wishes for the API spec expressed as luanti specific lua annotations. the actually important part isn't really the syntax or that i used lua annotations at all, but rather the kind of information i would like to obtain from the proposed API spec.

https://gist.github.com/corpserot/6f23ae3caca48f31b6b9b460507f23dc

[mark-wiemer]

Started to experiment here and LuaCATS + LuaLS is looking very promising :)

Setup

1. Install LuaLS in VS Code or your favorite IDE

2. Write a LuaCATS definition file (this would be the schema for all our docs):

   ---@meta
   
   ---@class core
   ---The core namespace
   core = {}
   
   ---@deprecated
   ---Legacy alias for the core namespace
   minetest = nil ---@type core
   
   ---When loading a mod, returns the name of that mod
   ---@return string
   function core.get_current_modname() end

3. Add that file's relative path to your Lua.workspace.library setting

   

4. Restart extensions

User experience

IntelliSense, hover text, and deprecation rendering:

---

For all updates, ref https://github.com/mark-wiemer/hello-hello/tree/main/packages/luanti/doc-schema :)

[mark-wiemer]

Proposed solution

Manually rewrite lua_api.md as luanti_api.lua in LuaCATS and integrate with LuaLS:

1. Update lua_api.md to luanti_api.lua with additional info in LuaCATS
   • If additional info isn't known, it can be left blank or explicitly marked as "unknown" in some way
   • We can test as we go using LuaLS :)
   • Mistakes will happen but should be easy to see, reproduce, and fix.
   • Advanced automated tests are out of scope for now. They are possible but have low ROI.
2. Create a LuaLS addon with the new luanti_api.lua under LLS-Addons
3. Share with the world :)