Define Markdown schema (DSL) for API reference docs

[mark-wiemer]

Right now, the docs are all in lua_api.md, where the only requirement is "valid Markdown" (really, it just needs to pass human review). This has resulted in various issues, namely that some function args have unclear types. A stricter Markdown schema would ensure that all necessary information is documented for each function, and a check can be automated. This issue focuses simply on defining the schema, not yet implementing the checks.

This schema can also be referred to as a DSL, or domain-specific language, because it needs to be unambiguous so that machines can consistently interpret it. As such, it'll be a DSL for Luanti documentation!

Considerations

• Ease of contribution
  • Clarity of schema for new contributors without automated tooling assistance (e.g. editing in Notepad or online editor)
  • Simplicity of contributing (e.g. copy-pasting HTML boilerplate vs handwriting Markdown)
• Ease of scanning plaintext

Criteria

• Return types documented
• Argument types documented
• All information about the function should be nested within the same section (see below)
• more to be discussed...

---

All information about the function should be nested within the same section

Details

For example, in the snippet below, the table description is not syntactically nested under the list, and would be hard to parse:

- `core.get_game_info()`: returns a table...
{
   id = string,
   ...
}
- `core.get_world_path()`: ...

When rendered, the table description ends the list, then the next bullet starts a new list:

• core.get_game_info(): returns a table...

{
   id = string,
   ...
}

• core.get_world_path(): ...

---

A more correct snippet would indent the table definition:

- `core.get_game_info()`: returns a table...
   {
      id = string,
      ...
   }
- `core.get_world_path()`: ...

Now the table description is clearly nested under the get_game_info list item:

• core.get_game_info(): returns a table...

  {
     id = string,
     ...
  }

• core.get_world_path(): ...

[mark-wiemer]

Ref https://github.com/mark-wiemer/hello-hello/blob/485714fbd0245d433617117654ef27bf407a42e4/packages/remark/lua_api_transformed.md

[SmallJoker]

I know this misses the point, but as for API reference I prefer a compact notation such that it's easy to scroll through in plaintext format.
EDIT: This also includes preserving the indent level to better see the scope of the code block (or listing point).

[appgurueu]

I think viewing this as a "Markdown schema" issue is putting the horse before the carriage.

We need some suitable, terse notation for types and documentation from which (interlinked) Markdown docs can then be generated along with typedefs (for whichever tool, be it a "Lua language server", a typed Lua dialect, or something like TypeScript).

Ideally, this would even let us generate the bindings (see also luanti-org/luanti#16685).

[mark-wiemer]

@appgurueu if you're suggesting parsing the Lua or C++ code comments directly, I'm not sure that will be very useful. We'll have to ensure every file already documented in lua_api.md has the same level of documentation in the code itself. There has been resistance there about adding verbose method header comments. I can see both sides of the argument as reasonable. I opted to move forward with what we had now. Much less rewriting this way. As always, happy to discuss.

[appgurueu]

if you're suggesting parsing the Lua or C++ code comments directly

That would be one option, but it's not really what I'm suggesting.

The point I wanted to make is that we should start with a suitable machine-readable format, not try to turn what is intended to be a human-readable format into a machine-readable format.

I have previously referenced how LÖVR does it, which is using a custom Lua format, but really any kind of general purpose "object notation" can be used, be it JSON, YAML, TOML, or whatnot, down to S-expressions. However, some may be less suitable than others; e.g. JSON will end up rather verbose and has no comments, YAML is a mess.
I would prefer Lua because I know that it can be made sufficiently concise, does not require anyone to learn a new notation (it's safe to assume that someone working with Luanti needs to learn Lua), and is extremely customizable.

Ultimately we will have to try it and see. I'll see if I can get a PoC for a Lua-DSL-based approach up in a couple days.

[mark-wiemer]

Let's move this discussion into the high-level parent issue

[corpserot]

2 Proposed markdown formats i've written initially in riseup, with some ideas WIP.

• RETRACTED: Contracts/Constraints idea. It is non-trivial to investigate and should be ignored. So for now, that means I propose that the constraints are written in descriptions and it can be added later in a "v1.1" big update to the spec (if even desired by tooling makers).

https://gist.github.com/corpserot/7a8caaf9695d99d01dae9575096b753f

markdown-custom sample:

### Function `core.after(time, func, args...)`
Calls the function `func` after `time` seconds, returning a job handle.

Jobs set for earlier times are executed earlier. If multiple jobs expire at exactly the same time, then they are executed in registration order.

`time` is a lower bound. The job is executed in the first server-step that started at least `time` seconds after the last time a server-step started, measured with globalstep dtime.

- Args:
    - `time` : `number`, unit `seconds` : If `0`, the job is executed in the next step.
    - `fn` : `core.after.F`
    - `args` : `core.after.F.@args`
- Envs:
    - `server-main`
    - `server-async`
    - `server-mapgen`
    - `client`

#### Function `F(args...)`
<!-- TODO `args` should be `(T...)` if we want to be correct. I have not define generics yet. I also think we don't need to explore generics just yet. it is a pretty ugly can of worms -->
- Args:
    - `args` : `(any...)`
<!-- again, no Returns ~= Returns `nil` -->

markdown-xml sample:

## Globalstep
<!-- TODO friendly human description about the globalstep -->

```xml
<Function name="core.after(time, func, args...)">
  Calls the function `func` after `time` seconds, returning a job handle.

  Jobs set for earlier times are executed earlier. If multiple jobs expire at exactly the same time, then they are executed in registration order.

  `time` is a lower bound. The job is executed in the first server-step that started at least `time` seconds after the last time a server-step started, measured with globalstep dtime.

  <Param name="time" type="number" unit="seconds">
    If `0`, the job is executed in the next step.
  </Param>
  <Param name="fn" type="core.after::F" />
  <Param name="args" type="core.after::F::@args" />

  <Function name="core.after::F(args...)">
    <Param name="args" type="(any...)" />
  </Function>

  <Environment value="server-main" />
</Function>
```

[wsor4035]

my proposal as html is valid markdown and would work within our existing tooling, for more logic, read through chat channel, a suggestion

<block>
  <function>core.function_that_does_thing(something, something2)</function>
  <param type="number">something</param>
  <param type="vector">something2</param>
  <description>
    *luanti* function that ~~does~~ things
  </description>
</block>

note this is more so proposing html for the schema than the actual elements, just to given an idea of what it could be.
some notes:

• custom html elments are a thing in the spec, but they should include - in them to not conflict with the browser provided ones
• before someone runs of screaming about requiring client side javascript for this, no, it wont custom elements are supported by browsers and rendered as spans (iirc), so just some custom styling is needed

happy to discuss actual element types/names if this is selected

[mark-wiemer]

I'm pretty against a tag-based system, I think it goes against the lightweight nature of Markdown and will discourage contributors more than it will help provide structure. Happy to be proven wrong, of course. But ultimately I imagine folks will start writing their own "headcanon" lightweight schema that they then translate into our tag-based canonical docs, that we then translate again. Would be great to create something that is easy for others to contribute to. I know tags add structure, but they also add a lot of typing overhead that is genuinely discouraging to new contributors.