Best way to embed LaTeX lexer in Markdown lexer

[dcst-it]

Hi everyone. For academic reasons, I need to use LaTeX math syntax in Markdown documents, which are then processed with Pandoc. Almost no text editor provides syntax highlighting for LaTeX math inside .md files, but some time ago I managed to add initial support for this feature in Textadept by copying the markdown.lua lexer in my local configuration folder and adding the following code at the end of the file:

-- Embedded LaTeX math.
local latex = lexer.load('latex')
local lmath = latex:get_rule('math')
local latex_start_rule	= #(P('\\begin') + P('\\(') + P('\\[') + P('$$') + P('$')) * lmath
local latex_end_rule	= #(P('\\end') + P('\\)') + P('\\]') + P('$$') + P('$')) * lmath
lex:embed(latex, latex_start_rule, latex_end_rule)

This works fine in most cases, with Textadept correctly highlighting both inline and display math... But, as I recently found, does not work completely when nesting multiple LaTeX display math environments: for example, when nesting multiple align blocks inside a gather block, the result looks like this:

As you can see, the \\ commands after the \end{align*} ones and the entire formula right before the \end{gather*} command are not highlighted. It should have been easy to see this limit, but when I first implemented this solution I was on a rush and I didn't think much on it. Is there a way to improve my solution and to support such cases?

Thank you in advance. By the way, nowadays various people use LaTeX math inside Markdown documents; it would be nice to support for this particular syntax in the default Markdown lexer of Textadept.

[orbitalquark]

No, sorry, I cannot think of a way to support nested embeddings like that :( The end rule would be dependent on data the start rule captures, and accurately keeping track of that information may not be possible since Textadept invokes the lexer in chunks of code, not the entire document at once. You might get correct highlighting when opening the document for the first time, but as you edit it, things would probably get out of sync.

[dcst-it]

Thank you for the answer, and sorry for the delayed response, I had a busy weekend...

This is not a great problem. Nested math environments are not so common, and I will simply avoid to use them - and, by the way, with those 5 lines of code of mine added to the Markdown lexer Textadept in this aspect is still better than 99% of the text editors in existence.

I will still use this thread to suggest a couple of ideas for future official Textadept releases:

• Despite this (secondary) limitation, could that snippet of mine be added to the default Markdown lexer? As I previously said, nowadays quite a few people use LaTeX math syntax in Markdown documents - and various technologies of primary relevance support that syntax as an extension, such as Pandoc and at least another Markdown processor, and also GitHub in readme files if I'm not wrong.
• The math environments alignat flalign could be added to the default LaTeX lexer, which currently does not support them. I copied the default lexer in my personal configuration folder and added them, but to have them by default would be nice for many LaTeX users, I think.

[orbitalquark]

I would like to keep the Markdown lexer restricted to the original specification, so I won't add LaTeX by default. However, as you pointed out, it's easy for others who want to to add it themselves.

I've added those keywords to the default LaTeX lexer: orbitalquark/scintillua@19ff62f.