Issues with LaTeX3 syntax highlighting

[Skillmon]

Description

I found two problems with the syntax highlighting inside the texE3Zone:

1. texE3Cmd matches things that can't be part of macro names with normal category codes, namely digits. The definition should be changed to

   syntax match texE3Cmd "\\\h\+"
       \ contained containedin=@texClusterE3
       \ nextgroup=texE3Opt,texE3Arg skipwhite skipnl

2. I don't know why but for some reason the texE3Function matching is overruled by the texSpecialChar matching it seems for L3 functions starting with underscores like \__my_internal_function:n. (This one I really don't know how to fix, hence no PR for this, sorry.)

Steps to reproduce

1. nvim -u minimal.vim minimal.tex
2. Use the following TeX code:

   \ExplSyntaxOn
   \@@_internal_command:nn % works, brilliant
   \__my_internal_command:nn % doesn't work, not brilliant

Expected behavior

Nice function-syntax-highlighting for \__my_internal_command

Actual behavior

The \_ part of the name gets syntax highlighting, the rest doesn't.

Do you use a latexmkrc file?

no

VimtexInfo

System info:
  OS: Arch Linux
  LaTeX version: pdfTeX 3.141592653-2.6-1.40.28 (TeX Live 2025)
  Vim version: NVIM v0.11.5
  Has clientserver: true
  Servername: /run/user/1000/nvim.8910.0
  $PATH:
    - /usr/bin
    - /usr/local/bin
    - /usr/local/bin/texbin
    - /usr/local/sbin

VimTeX project: bugreport
  base: bugreport.tex
  root: /home/jps/tmp/tex
  tex: /home/jps/tmp/tex/bugreport.tex
  main parser: fallback current file
  document class: 
  document class options: 
  compiler: latexmk
    engine: -pdf
    options:
      -verbose
      -file-line-error
      -synctex=1
      -interaction=nonstopmode
    callback: 1
    continuous: 1
    executable: latexmk
  viewer: Zathura
    xwin id: 0
  qf method: LaTeX logfile

[cfr42]

@Skillmon do @s confuse it for you? If I have internal 2e syntax as well, it seems not to cope. (But I do not know if this is to be expected.)

[Skillmon]

@cfr42 interestingly enough, inside E3-scope they don't and \@@_...: is correctly detected. I remember that I did get weird syntax highlighting once in the past for an internal command starting with \@, but I can't reproduce that now.

Do you have an example of that incorrect highlighting?

[cfr42]

I think this is the same problem @Skillmon described above:

but also:

Where expl3 syntax is inside \@ifpackageloaded{}{}{}, all of the underscores are marked invalid. But notice that as soon as the scope of that conditional ends, it is fine again. Here it doesn't matter if there are @ or double underscores. It all gets highlighted as erroneous.

[Rmano]

I have another example:

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{hyperref}
\ExplSyntaxOn
\NewDocumentCommand \autorefs { s O{and} m }
  { \__rmano_autorefs:nnn {#1} {#2} {#3} }

% #1 = star flag (boolean-likeI Oxford comma), #2 = conjunction, #3 = clist
\cs_new_protected:Npn \__rmano_autorefs:nnn #1 #2 #3
  {
    \clist_set:Nn \l_tmpa_clist { #3 }
    \int_set:Nn \l_tmpa_int { \clist_count:N \l_tmpa_clist }

    \int_case:nnF { \l_tmpa_int }
      {
        { 0 } { }
        { 1 } { \autoref{ \clist_item:Nn \l_tmpa_clist { 1 } } }
        { 2 }
          {
            \autoref{ \clist_item:Nn \l_tmpa_clist { 1 } }
            {}~#2~
            \autoref{ \clist_item:Nn \l_tmpa_clist { 2 } }
          }
      }
      {
        % 3 or more
        \int_step_inline:nn { \int_eval:n { \l_tmpa_int - 1 } }
          {
            \int_compare:nNnTF {##1} = {1}
              { \autoref{ \clist_item:Nn \l_tmpa_clist {##1} } }
              { ,~\autoref{ \clist_item:Nn \l_tmpa_clist {##1} } }
          }
        \IfBooleanTF{#1}
          { ~#2~ }      % starred: no Oxford comma
          { ,~#2~ }     % unstarred: Oxford comma
        \autoref{ \clist_item:Nn \l_tmpa_clist { \l_tmpa_int } }
      }
  }
\ExplSyntaxOff

\begin{document}
\section{one}\label{a}\label{b}\label{c}\label{d}

\autorefs[y]{a,b,c,d}

\end{document}

is highlighted like this:

nvim called with:

nvim --clean -u nvim-mini.vim highligh3.tex

and nvim.mini contains:

set nocompatible
set runtimepath^=~/.config/nvim/plugged/vimtex
set runtimepath+=~/.config/nvim/vimtex/plugged/after
set runtimepath^=~/tmp
filetype plugin indent on
syntax enable

[lervag]

First, I'm so sorry for the late reply! I don't have any good reasons, except perhaps too many things going on.

1. `texE3Cmd` matches things that can't be part of macro names …

If I understand correctly, you are saying I should change \w to \h?

2. I don't know why but for some reason the `texE3Function` matching is overruled by the `texSpecialChar` matching it seems …

That's strange. On my side, it looks to be working as expected.

@cfr42

I think this is the same problem @Skillmon described above: …

In the code from your screenshot, I don't see that the code is limited by \ExplSyntaxOn ... ...Off? So I'm not sure if this is the same problem...

[lervag]

I have another example: …

Thanks! I can now reproduce the errors. For some reason, it seems my own config does not have the problem with \__..., and I can't easily figure out why.

Now, I'm curious: ony way to "fix" the problem with the \_ is to specifically disallow the texSpecialChar inside the expl3 syntax. Would that work for you?

Now, you also have another issue that is quite interesting:

...
          {
            \autoref{ \clist_item:Nn \l_tmpa_clist { 1 } }
            {}~#2~
            \autoref{ \clist_item:Nn \l_tmpa_clist { 2 } }
          }
      }

Here the \autoref command gives the texRefArg group which does not allow to include subgroups. That is, the texRefArg assumes there is not nested { ... } inside the argument. If you remove the { 1 } and { 2 } you should see that it works as expected. However, that is probably not possibly for you here, so we would need to think about other options.

And I'm not quite sure what would be reasonable here.

[Skillmon]

There is no need to feel sorry for taking your time! I'm glad for many features of vimtex I get and I totally respect that you are working on it in your spare time!

If I understand correctly, you are saying I should change \w to \h?

Exactly.

Regarding the other issue: I'll see whether I can provide a better example that might reproduce it for you.

[Rmano]

Hi @lervag - please don't feel pressed - take your time!

I am not sure about your comment about \autoref; I'm fine on how the arguments are rendered, my problem are the closing braces marked as errors... Or are they related?

[Skillmon]

@Rmano as I understood it the issue with the incorrectly highlighted closing braces is because \autoref triggers another syntax region that doesn't accept nested braces, so stops with the first closing brace and not with the first matching brace. So the reason for the highlighting in your example is precisely the fact that you use it in the argument of a function named \autoref.

[cfr42]

First, I'm so sorry for the late reply! I don't have any good reasons, except perhaps too many things going on.

As the others said, there is no need to be sorry. 'too many things going on' sounds like a pretty good reason to me ;).

The examples I gave are (I think) from a .dtx which uses \ProvidesExplPackage.. One problem I have is the the highlighter frequently forgets it is in an expl3 zone. E.g. if I use :%s/<something>/<something>/, everything will suddenly be highlighted as erroneous. Reloading the .dtx or .tex file does the same. Proximity to \ExplSyntaxOn definitely helps, so it is possible to add additional \ExplSyntaxOns into the source, but this is probably not best practice in coding terms :-).

[lervag]

There is no need to feel sorry for taking your time! I'm glad for many features of vimtex I get and I totally respect that you are working on it in your spare time!

Thanks for understanding; both to you and the others!

If I understand correctly, you are saying I should change \w to \h?

Exactly.

Great, I'll make the update and push now.

I am not sure about your comment about \autoref; I'm fine on how the arguments are rendered, my problem are the closing braces marked as errors... Or are they related?

Yes, they're related; @Skillmon is spot on in his explanation.

The examples I gave are (I think) from a .dtx which uses \ProvidesExplPackage.. One problem I have is the the highlighter frequently forgets it is in an expl3 zone. E.g. if I use :%s/<something>/<something>/, everything will suddenly be highlighted as erroneous. Reloading the .dtx or .tex file does the same. Proximity to \ExplSyntaxOn definitely helps, so it is possible to add additional \ExplSyntaxOns into the source, but this is probably not best practice in coding terms :-).

That's interesting. If I understand correctly, this problem is related to the syntax sync options. You could try to manually check this with :syntax sync fromstart, which should always ensure that the syntax is parsed from the start of the file. It should keep things accurate, but may be (quite a bit) slower.

---

Do I understand correctly that the \w to \h change fixes the present issue?

Regarding the other problem that @Rmano has: I'm really not sure how to fix that in a good way. One possibility could be to simply limit the content of the E3 zones instead of allowing everything. That is, right now, the E3 zones are more or less equivalent to a general latex file and will allow special syntax highlighting for things like the reference commands, sub packages, and everything. However, one possibility could be to restrict the syntax inside E3 zones to just a small subset

[cfr42]

The examples I gave are (I think) from a .dtx which uses \ProvidesExplPackage.. One problem I have is the the highlighter frequently forgets it is in an expl3 zone. E.g. if I use :%s/<something>/<something>/, everything will suddenly be highlighted as erroneous. Reloading the .dtx or .tex file does the same. Proximity to \ExplSyntaxOn definitely helps, so it is possible to add additional \ExplSyntaxOns into the source, but this is probably not best practice in coding terms :-).

That's interesting. If I understand correctly, this problem is related to the syntax sync options. You could try to manually check this with :syntax sync fromstart, which should always ensure that the syntax is parsed from the start of the file. It should keep things accurate, but may be (quite a bit) slower.

Thanks. I will give this a try. With this, I probably need to try any suggestion out for a bit, because I'm not altogether sure of the exact conditions. But I will try if this works rather than moving around the file. (We will see how long it takes!)

Regarding the other problem that @Rmano has: I'm really not sure how to fix that in a good way. One possibility could be to simply limit the content of the E3 zones instead of allowing everything. That is, right now, the E3 zones are more or less equivalent to a general latex file and will allow special syntax highlighting for things like the reference commands, sub packages, and everything. However, one possibility could be to restrict the syntax inside E3 zones to just a small subset

I can't reproduce the issue with \autoref, but I'm not sure why:

[Rmano]

I updated now, with the same minimal conf of #3245 (comment) I have the same error highlights...