Built-in Regex-Based Syntax Highlighting for Code Blocks

[Samyssmile]

Context

notectl aims to provide IDE-like capabilities. The CodeBlockPlugin already ships a complete code editing experience (keyboard handling, indentation, copy-to-clipboard, language attribute, input rules). It also defines a pluggable SyntaxHighlighter interface with a built-in regex-based tokenizer. Users get syntax coloring out of the box for supported languages.

This discussion captures research findings and tracks what has been delivered and what remains for full syntax highlighting support.

Current Status

Component	Status
CodeBlockPlugin with full editing support	✅ Done
Pluggable SyntaxHighlighter interface	✅ Done
Decorations system (notectl-token--{type} classes)	✅ Done
Code block colors in Theme system (bg, fg, header)	✅ Done
Language attribute per code block	✅ Done
Input rules (```java + Space → code block with language)	✅ Done
Service API (setLanguage(), getSupportedLanguages())	✅ Done
Regex tokenizer (RegexTokenizer.ts) — Java, JSON, XML	✅ Done
Syntax token colors in Theme system (ThemeSyntax, 16 token types, light + dark)	✅ Done
Additional languages: Python, Go, JavaScript, TypeScript	❌ Missing
Language picker UI in header	❌ Missing

What Was Implemented

Regex Tokenizer (RegexTokenizer.ts)

• Implements the SyntaxHighlighter interface with regex-based tokenization
• Currently supports: Java (Java 21), JSON, XML
• Supports language registration and aliases
• Lives inside plugins/code-block/highlighter/

Syntax Token Colors in Theme System (SyntaxTokenTypes.ts)

• ThemeSyntax mapped type with 16 canonical token types: keyword, string, comment, number, function, operator, punctuation, boolean, null, property, type, annotation, tag, attribute, constant, regex
• Each token supports color, optional fontStyle and fontWeight
• Full color schemes in both LIGHT_THEME and DARK_THEME
• Token colors update automatically when the theme switches — no consumer CSS needed

The SyntaxHighlighter Interface

interface SyntaxHighlighter {
  tokenize(code: string, language: string): readonly SyntaxToken[];
  getSupportedLanguages(): readonly string[];
}

interface SyntaxToken {
  readonly from: number;
  readonly to: number;
  readonly type: string; // e.g. keyword, string, comment, number
}

The RegexTokenizer implements this interface. The existing Decorations system handles the rest — mapping tokens to inline decorations with CSS classes.

User Workflow

Power user: Markdown shortcut (works today)

1. User types: ```java [Space]
2. → Input rule fires, code block appears with language="java"
3. → Header shows "Java"
4. → User writes code with live syntax highlighting

Regular user: Toolbar / keyboard shortcut (UX gap)

1. User presses Ctrl+Shift+M or clicks toolbar button
2. → Code block appears with header showing "plain"
3. → ⚠️ No UI to change the language — only possible via markdown shortcut
4. → Need: clickable language label in header → dropdown with supported languages

Remaining Scope

Additional target languages

Python, Go, JavaScript, TypeScript

Language picker UI

Make the language label in the code block header clickable, show dropdown of supported languages.

Open Questions

• Should the language picker be a simple dropdown or a searchable combobox (relevant if we add many languages later)?