Reworked how comments are handled in JSONC.

Author: uhop

AGENTS.md

@@ -109,7 +109,7 @@ stream-json/
 - **Keep `.js` and `.d.ts` files in sync** for all modules under `src/`.
 - **Token-based architecture.** The parser produces a stream of `{name, value}` tokens. All filters, streamers, and utilities operate on this token protocol.
 - **Backpressure must be handled correctly.** All stream components rely on Node.js stream infrastructure via `stream-chain`.
-- **Intended input is data the user owns or trusts** (dumps, exports, logs). The library is not designed for hostile input; docs say so, and code changes are not hardened against adversarial JSON beyond `JSON.parse` parity (`__proto__` becomes an own property) and the filters' `maxDepth` guard.
+- **Intended input is data the user owns or trusts** (dumps, exports, logs). The library is not designed for hostile input; docs say so, and code changes are not hardened against adversarial JSON or JSONC beyond `JSON.parse` parity (`__proto__` becomes an own property) and the filters' `maxDepth` guard.
 
 ## Architecture
 
@@ -136,7 +136,7 @@ stream-json/
   - `withParser(fn, options)` creates a `gen(parser(options), fn(options))` pipeline — the most common pattern.
   - Most components export `.withParser(options)` and `.withParserAsStream(options)` static methods.
 - **JSONL** (**deprecated — slated for removal in a future major**): `jsonl/parser.js` and `jsonl/stringer.js` are thin re-exports of stream-chain's JSONL (the parser API was absorbed into stream-chain). The Node/Web wrappers delegate `.asStream`/`.asWebStream` to stream-chain's bundled `stream-chain/node/jsonl/*` and `stream-chain/web/jsonl/*` factories. Use stream-chain's JSONL directly. Rationale: stream-json is a JSON _token_ library; JSONL yields whole objects per line and belongs in stream-chain with the other substrate components.
-- **JSONC**: `jsonc/parser.js`, `jsonc/stringer.js`, and `jsonc/verifier.js` for JSON with Comments. Extend the standard parser/stringer/verifier (same `charCodeAt` tokenizer/validator) with `whitespace`/`comment`/`comma` tokens, trailing comma support, and `streamWhitespace`/`streamComments`/`streamCommas` (parser) plus `useCommas` (stringer) options. `streamCommas` + `useCommas` give byte-faithful comma round-trips (incl. trailing commas) for streaming edits; both default off. Raw inner exports: `jsoncParser`, `jsoncVerifier`.
+- **JSONC**: `jsonc/parser.js`, `jsonc/stringer.js`, and `jsonc/verifier.js` for JSON with Comments. Extend the standard parser/stringer/verifier (same `charCodeAt` tokenizer/validator) with `whitespace` / comment / `comma` tokens, trailing comma support, and `streamWhitespace`/`streamComments`/`packComments`/`streamCommas` (parser) plus `useCommentValues`/`useCommas` (stringer) options. Comments mirror strings (`startComment` / `commentChunk` / `endComment`, packed `commentValue`) and their scan resumes across chunks — never rescan an accumulated comment from its start. `streamCommas` + `useCommas` give byte-faithful comma round-trips (incl. trailing commas) for streaming edits; both default off. Raw inner exports: `jsoncParser`, `jsoncVerifier`.
 - **File I/O (Node-only)** (`src/file/`, since 3.3.0): `parseFile()` is an input-edge stage that turns a file path into a token stream (`gen(asyncBlockReader, jsonParser)`); `stringerToFile(path)` is the symmetric output-edge sink (`gen(stringer, asyncBlockWriter)`); `verifyFile(path)` is a standalone async validator returning `Promise<void>`. JSONC variants under `src/file/jsonc/`. NOT mirrored in `core/` or `web/` because they use `node:fs/promises`. Compose with `pipe(...)` (one-shot driver with auto-flush — `gen(...)` alone doesn't flush, so `stringerToFile` wouldn't close the file) and `drain(asyncGen)` (returns the last yielded value or `undefined`). Both helpers live in `core/utils/` (web-safe, no Node deps).
 - **Substrate split**: `src/core/` holds the pure substrate-agnostic factories (no Node-stream imports — checked by `tests/node/test-browser-safe.js` which scans `.d.ts` for `node:*` imports and `extends DuplexOptions`). `src/` (Node entry) attaches `.asStream` (Node Duplex) and `.asWebStream` (Web `{readable, writable}` pair). `src/web/` attaches only `.asWebStream` and `.withParserAsWebStream`, with no Node-stream imports — safe for browser bundles. The `chain` from `stream-chain` (Node) or `stream-chain/web` (Web) auto-wraps the pure flushables on both substrates, so user-facing pipeline code is identical.
 

ARCHITECTURE.md

@@ -194,8 +194,8 @@ The item shape is the exported `KeyedValue<K, T>` type (`core/streamers/stream-b
 
 ### JSONC support
 
-- `jsonc/parser.js` — the `charCodeAt` tokenizer of `parser.js` extended with `//` and `/* */` comments, trailing commas, and optional `whitespace`/`comment`/`comma` tokens (raw inner export `jsoncParser`). Options: `streamWhitespace` (default: true), `streamComments` (default: true), `streamCommas` (default: false — emit a valueless `comma` token at every comma's position; the comma byte is already buffered, so emission needs no lookahead and is fully resumable). All standard parser options are supported.
-- `jsonc/stringer.js` — fork of `stringer.js` that passes `whitespace` and `comment` tokens through verbatim. Option `useCommas` (default: false) renders streamed `comma` tokens as `,` (a separator is still auto-inserted before a value when no `comma` token preceded it, so output stays valid even if commas were dropped upstream) — `streamCommas` + `useCommas` give byte-faithful comma round-trips, incl. trailing commas. All standard stringer options are supported.
+- `jsonc/parser.js` — the `charCodeAt` tokenizer of `parser.js` extended with `//` and `/* */` comments, trailing commas, and optional `whitespace` / comment / `comma` tokens (raw inner export `jsoncParser`). Comments mirror strings — streamed `startComment` / `commentChunk` / `endComment` and packed `commentValue`; the comment scan resumes across input chunks, so a comment of any length costs linear time (GHSA-hqr4-qq8f-hg3x). Options: `streamWhitespace` (default: true), `streamComments` (default: true), `packComments` (default: true), `streamCommas` (default: false — emit a valueless `comma` token at every comma's position; the comma byte is already buffered, so emission needs no lookahead and is fully resumable). All standard parser options are supported.
+- `jsonc/stringer.js` — fork of `stringer.js` that passes `whitespace` and comment tokens through verbatim (`useCommentValues` selects the packed form). Option `useCommas` (default: false) renders streamed `comma` tokens as `,` (a separator is still auto-inserted before a value when no `comma` token preceded it, so output stays valid even if commas were dropped upstream) — `streamCommas` + `useCommas` give byte-faithful comma round-trips, incl. trailing commas. All standard stringer options are supported.
 - `jsonc/verifier.js` — the `charCodeAt` validator of `utils/verifier.js` extended to accept comments and trailing commas (raw inner export `jsoncVerifier`). Reports error offset, line, and position for invalid JSONC.
 - Downstream compatibility: all existing filters, streamers, and utilities ignore unknown token types, so they work with JSONC parser output unmodified.
 

README.md

@@ -16,7 +16,7 @@ Why it might be for you:
 
 ## Intended input
 
-`stream-json` is built for data you own or trust — database dumps, exports, logs, and files produced by your own systems. It is not designed for hostile input: do not feed it JSON from the open internet or from untrusted users. Untrusted JSON needs validation of its own before it reaches a pipeline.
+`stream-json` is built for data you own or trust — database dumps, exports, logs, and files produced by your own systems. It is not designed for hostile input: do not feed it JSON or JSONC from the open internet or from untrusted users. Untrusted JSON needs validation of its own before it reaches a pipeline.
 
 ## Example
 

bench/parser-jsonc-comments.js

@@ -0,0 +1,32 @@
+// Scaling meter for JSONC comment scanning (GHSA-hqr4-qq8f-hg3x): one block
+// comment fed in 16 KB chunks; time must grow linearly with size — a quadratic
+// curve (4x per doubling) means the scan restarted from the comment's start.
+// Run via `npx nano-bench-io bench/parser-jsonc-comments.js` and read the ratios.
+import jsoncParser from '../src/jsonc/parser.js';
+
+const feed = doc =>
+  new Promise((resolve, reject) => {
+    const stream = jsoncParser.asStream();
+    stream.on('data', () => {});
+    stream.on('error', reject);
+    stream.on('end', resolve);
+    for (let i = 0; i < doc.length; i += 16384) stream.write(doc.slice(i, i + 16384));
+    stream.end();
+  });
+
+const K = 1024;
+const docs = {
+  'comment 256K': '/*' + 'a'.repeat(256 * K) + '*/1',
+  'comment 512K': '/*' + 'a'.repeat(512 * K) + '*/1',
+  'comment 1M': '/*' + 'a'.repeat(1024 * K) + '*/1',
+  'string 1M': '"' + 'a'.repeat(1024 * K) + '"'
+};
+
+export default Object.fromEntries(
+  Object.entries(docs).map(([name, doc]) => [
+    name,
+    async n => {
+      for (let i = 0; i < n; ++i) await feed(doc);
+    }
+  ])
+);

llms-full.txt

@@ -14,7 +14,7 @@
 - Proper backpressure handling via Node.js stream infrastructure
 - Works with `stream-chain` for pipeline composition
 
-**Intended input:** data you own or trust — database dumps, exports, logs, files produced by your own systems. `stream-json` is not designed for hostile input: do not feed it JSON from the open internet or from untrusted users; untrusted JSON needs validation of its own before it reaches a pipeline.
+**Intended input:** data you own or trust — database dumps, exports, logs, files produced by your own systems. `stream-json` is not designed for hostile input: do not feed it JSON or JSONC from the open internet or from untrusted users; untrusted JSON needs validation of its own before it reaches a pipeline.
 
 ## Quick start
 
@@ -810,7 +810,7 @@ for await (const chunk of ts.readable) console.log(chunk);
 
 ### jsonc/Parser
 
-Streaming JSONC (JSON with Comments) parser. Uses the same `charCodeAt` tokenizer as the standard parser, extended with `//` and `/* */` comments, trailing commas, and optional `whitespace`/`comment`/`comma` tokens. Has `asStream` (Node Duplex) and `asWebStream` (Web pair). Browser-safe Web-only entry: `stream-json/web/jsonc/parser.js`.
+Streaming JSONC (JSON with Comments) parser. Uses the same `charCodeAt` tokenizer as the standard parser, extended with `//` and `/* */` comments, trailing commas, and optional `whitespace` / comment / `comma` tokens. Comments mirror strings: streamed as `startComment` / `commentChunk` / `endComment` (resumable across input chunks — linear time, constant memory without packing) and packed as `commentValue`. Has `asStream` (Node Duplex) and `asWebStream` (Web pair). Browser-safe Web-only entry: `stream-json/web/jsonc/parser.js`.
 
 Static methods:
 - `jsoncParser(options)` — factory function returning a composable function for `chain()`.
@@ -820,13 +820,15 @@ Static methods:
 
 Options (in addition to all standard parser options):
 - `streamWhitespace` (boolean, default: true) — emit `whitespace` tokens.
-- `streamComments` (boolean, default: true) — emit `comment` tokens.
+- `streamComments` (boolean, default: true) — emit `startComment` / `commentChunk` / `endComment` tokens.
+- `packComments` (boolean, default: true) — emit `commentValue` tokens holding the whole comment. Both off: comments are consumed silently.
 - `streamCommas` (boolean, default: false) — emit a valueless `comma` token at the position of every comma (separator or trailing). For faithful round-trip editing: pair with the stringer's `useCommas` to reproduce comma placement (incl. trailing commas) exactly. No lookahead — the comma is already buffered when seen, so emission is fully resumable.
 
 Additional tokens:
 - `{name: 'whitespace', value: '  \n'}` — contiguous whitespace between tokens.
-- `{name: 'comment', value: '// ...\n'}` — single-line comment (includes EOL).
-- `{name: 'comment', value: '/* ... */'}` — multi-line comment (includes delimiters).
+- `{name: 'startComment'}`, `{name: 'commentChunk', value: '...'}`, `{name: 'endComment'}` — a comment in chunks (delimiters included; a chunk may end anywhere).
+- `{name: 'commentValue', value: '// ...\n'}` — single-line comment (includes EOL).
+- `{name: 'commentValue', value: '/* ... */'}` — multi-line comment (includes delimiters).
 - `{name: 'comma'}` — a `,` (separator or trailing), valueless; only with `streamCommas`.
 
 ```js
@@ -846,15 +848,15 @@ chain([
 // Suppress whitespace/comment tokens
 chain([
   fs.createReadStream('settings.jsonc'),
-  jsoncParser({streamWhitespace: false, streamComments: false}),
+  jsoncParser({streamWhitespace: false, streamComments: false, packComments: false}),
   streamArray(),
   ({value}) => console.log(value)
 ]);
 ```
 
 ### jsonc/Stringer
 
-JSONC stringer that passes `whitespace` and `comment` tokens through verbatim. All other tokens are handled identically to the standard stringer. Has `asStream` (Node Duplex) and `asWebStream` (Web pair). Browser-safe Web-only entry: `stream-json/web/jsonc/stringer.js`.
+JSONC stringer that passes `whitespace` and comment tokens through verbatim — comments from `commentChunk`s by default, or from `commentValue` with `useCommentValues` (default: false). All other tokens are handled identically to the standard stringer. Has `asStream` (Node Duplex) and `asWebStream` (Web pair). Browser-safe Web-only entry: `stream-json/web/jsonc/stringer.js`.
 
 Static methods:
 - `jsoncStringer(options)` — factory function returning a flushable function for `chain()`.

llms.txt

@@ -2,7 +2,7 @@
 
 > Micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. Parse JSON files far exceeding available memory using a SAX-inspired streaming token API. One dependency: `stream-chain`.
 
-**Intended input:** data you own or trust (database dumps, exports, logs, your own systems' files). Not designed for hostile input — do not feed it JSON from the open internet or from untrusted users; validate untrusted JSON before it reaches a pipeline.
+**Intended input:** data you own or trust (database dumps, exports, logs, your own systems' files). Not designed for hostile input — do not feed it JSON or JSONC from the open internet or from untrusted users; validate untrusted JSON before it reaches a pipeline.
 
 ## Install
 
@@ -190,10 +190,10 @@ chain([fs.createReadStream('data.jsonl'), parser(), ({value}) => transform(value
 
 ## JSONC support
 
-- **`jsonc/parser(options)`** — JSONC parser (JSON with Comments). Same `charCodeAt` tokenizer as the standard parser, extended with `//` and `/* */` comments, trailing commas, and `whitespace` / `comment` / `comma` tokens.
-  - Extra options: `streamWhitespace` (default: true), `streamComments` (default: true), `streamCommas` (default: false — emit a valueless `comma` token at every comma, separator or trailing, for faithful round-trip editing).
+- **`jsonc/parser(options)`** — JSONC parser (JSON with Comments). Same `charCodeAt` tokenizer as the standard parser, extended with `//` and `/* */` comments, trailing commas, and `whitespace` / comment / `comma` tokens. Comments mirror strings: streamed as `startComment` / `commentChunk` / `endComment` (resumable across input chunks, linear time) and packed as `commentValue`.
+  - Extra options: `streamWhitespace` (default: true), `streamComments` (default: true — the streamed form), `packComments` (default: true — `commentValue`; both off consumes comments silently), `streamCommas` (default: false — emit a valueless `comma` token at every comma, separator or trailing, for faithful round-trip editing).
   - All standard parser options are supported.
-- **`jsonc/stringer(options)`** — JSONC stringer. Passes `whitespace` and `comment` tokens through verbatim. Extra option: `useCommas` (default: false — render streamed `comma` tokens as `,`, auto-inserting a separator only when no comma token arrived, so output stays valid even if commas were dropped upstream).
+- **`jsonc/stringer(options)`** — JSONC stringer. Passes `whitespace` and comment tokens through verbatim (comments from `commentChunk`s, or from `commentValue` with `useCommentValues`). Extra options: `useCommentValues` (default: false), `useCommas` (default: false — render streamed `comma` tokens as `,`, auto-inserting a separator only when no comma token arrived, so output stays valid even if commas were dropped upstream).
 - **`jsonc/verifier(options)`** — JSONC validator. Same `charCodeAt` validator as `Verifier`, accepting comments and trailing commas. Reports exact error position.
 
 ```js

src/core/jsonc/parser.d.ts

@@ -21,11 +21,14 @@ declare function parser(options?: parser.JsoncParserOptions): Flushable<string,
 declare namespace parser {
   /**
    * A single token emitted by the JSONC parser. Extends the base JSON `Token`
-   * with `comment` — single-line (`//`) and block (`/* ... *​/`) comments
-   * surfaced when `streamComments` is set — and `comma`, a valueless marker
-   * emitted at every comma's position when `streamCommas` is set.
+   * with comments — single-line (`//`) and block (`/* ... *​/`), delimiters
+   * included — in the same two forms as strings: streamed as
+   * `startComment` / `commentChunk` / `endComment` when `streamComments` is set,
+   * packed as `commentValue` when `packComments` is set — and `comma`, a
+   * valueless marker emitted at every comma's position when `streamCommas` is set.
    */
-  export type Token = BaseToken | {name: 'comment'; value: string} | {name: 'comma'};
+  export type Token =
+    BaseToken | {name: 'startComment'} | {name: 'commentChunk'; value: string} | {name: 'endComment'} | {name: 'commentValue'; value: string} | {name: 'comma'};
   /** Alias of `Token` — disambiguates when both JSON and JSONC tokens are imported. */
   export type JsoncToken = Token;
 
@@ -56,8 +59,19 @@ declare namespace parser {
     jsonStreaming?: boolean;
     /** Emit `whitespace` tokens. Default: `true`. */
     streamWhitespace?: boolean;
-    /** Emit `comment` tokens. Default: `true`. */
+    /**
+     * Emit `startComment`/`endComment`/`commentChunk` tokens. Default: `true`.
+     * A comment may arrive in several chunks; the scan resumes across input
+     * chunks, so a long comment costs linear time and, without `packComments`,
+     * constant memory.
+     */
     streamComments?: boolean;
+    /**
+     * Pack comments into `commentValue` tokens (the whole comment, delimiters
+     * included). Default: `true`. With both `streamComments` and `packComments`
+     * off, comments are consumed silently.
+     */
+    packComments?: boolean;
     /**
      * Emit a valueless `comma` token at the position of every comma (separator
      * or trailing), so a parse → stringify round-trip can reproduce comma