Skip to content

AGENTS.md

Latest commit

 

History

History
212 lines (185 loc) · 17.8 KB

AGENTS.md

File metadata and controls

212 lines (185 loc) · 17.8 KB
Raw

AGENTS.md — stream-json

stream-json is a micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. It can parse JSON files far exceeding available memory streaming individual primitives using a SAX-inspired API. It depends on stream-chain for pipeline composition.

For project structure, module dependencies, and the architecture overview see ARCHITECTURE.md. For detailed usage docs and API references see the wiki.

Setup

This project uses a git submodule for the wiki:

git clone --recursive https://github.com/uhop/stream-json.git
cd stream-json
npm install

Commands

  • Install: npm install
  • Test: npm test (runs tape6 --flags FO)
  • Test (Bun): npm run test:bun
  • Test (Deno): npm run test:deno
  • Test (sequential): npm run test:seq
  • Test (process-per-file): npm run test:proc
  • Test (single file): node tests/test-<name>.js
  • TypeScript check: npm run ts-check
  • JS check (checkJs): npm run js-check
  • Bench: npm run bench -- bench/<name>.js
  • Lint: npm run lint (Prettier check)
  • Lint fix: npm run lint:fix (Prettier write)

Project structure

stream-json/
├── package.json          # Package config; "tape6" section configures test discovery
├── src/                  # Source code
│   ├── index.js          # Main entry point: creates Parser + emit()
│   ├── index.d.ts        # TypeScript definitions for the main module
│   ├── parser.js         # Streaming SAX-like JSON parser (token stream)
│   ├── parser.d.ts       # TypeScript definitions for parser
│   ├── assembler.js      # Token stream → JavaScript objects (plain class, `onDone` callback)
│   ├── assembler.d.ts    # TypeScript definitions for assembler
│   ├── disassembler.js   # JavaScript objects → token stream
│   ├── disassembler.d.ts # TypeScript definitions for disassembler
│   ├── stringer.js       # Token stream → JSON text (flushable function + asStream)
│   ├── stringer.d.ts     # TypeScript definitions for stringer
│   ├── emitter.js        # Token stream → EventEmitter events (Writable); .asWebStream → EventTarget
│   ├── emitter.d.ts      # TypeScript definitions for emitter
│   ├── filters/          # Token stream editors
│   │   ├── filter-base.js    # Base for all filters (filterBase + makeStackDiffer)
│   │   ├── pick.js           # Pick subobjects by path
│   │   ├── replace.js        # Replace subobjects with a value
│   │   ├── ignore.js         # Remove subobjects (Replace variant)
│   │   └── filter.js         # Filter tokens preserving shape
│   ├── streamers/        # Token stream → object stream
│   │   ├── stream-base.js    # Base for all streamers (uses Assembler)
│   │   ├── stream-values.js  # Stream successive JSON values
│   │   ├── stream-array.js   # Stream array elements
│   │   └── stream-object.js  # Stream object properties
│   ├── utils/            # Utilities
│   │   ├── emit.js           # Decorate a Node Readable with token events (Web variant at src/web/utils/emit.js → EventTarget)
│   │   ├── with-parser.js    # Create parser + component pipelines
│   │   ├── batch.js          # Batch items into arrays (wraps stream-chain batch)
│   │   ├── verifier.js       # Validate JSON text (gen pipeline + asStream)
│   │   └── flex-assembler.js # Assembler with custom containers (Map, Set, etc.)
│   ├── jsonl/            # JSONL (line-separated JSON) support
│   │   ├── parser.js         # JSONL parser → {key, value} objects
│   │   └── stringer.js       # Objects → JSONL text (Transform stream; .asWebStream → Web TransformStream)
│   ├── jsonc/            # JSONC (JSON with Comments) support
│   │   ├── parser.js         # JSONC parser → token stream (extends parser.js; raw export jsoncParser)
│   │   ├── stringer.js       # JSONC token stream → text (extends stringer.js)
│   │   └── verifier.js       # JSONC validator with error locations (extends verifier.js; raw export jsoncVerifier)
│   ├── file/             # Node-only file I/O (uses node:fs/promises; NOT mirrored in core/ or web/)
│   │   ├── index.js          # Barrel: parseFile, verifyFile, stringerToFile, pipe, drain
│   │   ├── parser.js         # parseFile() — file path → token stream (input-edge stage)
│   │   ├── verifier.js       # verifyFile() — standalone async validator (Promise<void>)
│   │   ├── stringer.js       # stringerToFile() — token stream → file (output-edge sink)
│   │   ├── jsonc/{index,parser,verifier,stringer}.js  # JSONC variants
│   │   └── internal/{block-reader,block-writer}.js     # Shared async-fs primitives
│   ├── core/             # Pure, substrate-agnostic factories (no Node-stream imports)
│   │   ├── utils/{drain,pipe}.js  # New generic helpers — last-value drain + one-shot flush driver
│   │   └── …                 # Mirrors src/ layout: each component's runtime + .d.ts
│   └── web/              # Web Streams substrate entries (browser-safe)
│       └── …                 # Mirrors src/ layout: each component's factory + asWebStream + .d.ts
├── tests/                # Test files (test-*.js, using tape-six)
├── bench/                # Micro-benchmarks (nano-benchmark)
├── wiki/                 # GitHub wiki documentation (git submodule)
└── .github/              # CI workflows, Dependabot config

Code style

  • ESM throughout ("type": "module" in package.json). Runs on currently-supported Node.js.
  • No transpilation — code runs directly.
  • Prettier for formatting (see .prettierrc): 160 char width, single quotes, no bracket spacing, no trailing commas, arrow parens "avoid".
  • 2-space indentation.
  • Semicolons are enforced by Prettier (default semi: true).
  • Imports use import syntax with explicit .js extensions on all relative paths.
  • Each module exports a default + a named mirror per the fleet's default-export with named mirror convention — the generic name (parser/verifier/stringer). Parser/verifier modules additionally export the raw inner factory under a format-named export (jsonParser, jsoncParser, jsonlParser, jsonVerifier, jsoncVerifier): the bare tokenizer/validator with no fixUtf8Stream front, so the default/parser is gen(fixUtf8Stream(), jsonParser()). Stringers (no UTF-8 front) export stringer plus a format-named alias.
  • The package is stream-json. It depends on stream-chain 4.x for pipeline composition.
  • Comments are why-markers only — a non-trivial decision or constraint, an algorithm reference, or explicitly requested JSDoc; never narrate what the code does (fleet convention no-narrating-comments).

Critical rules

  • One runtime dependency: stream-chain. Do not add other packages to dependencies. Only devDependencies are allowed.
  • Do not modify or delete test expectations without understanding why they changed.
  • Keep .js and .d.ts files in sync for all modules under src/.
  • Token-based architecture. The parser produces a stream of {name, value} tokens. All filters, streamers, and utilities operate on this token protocol.
  • Backpressure must be handled correctly. All stream components rely on Node.js stream infrastructure via stream-chain.
  • Intended input is data the user owns or trusts (dumps, exports, logs). The library is not designed for hostile input; docs say so, and code changes are not hardened against adversarial JSON or JSONC beyond JSON.parse parity (__proto__ becomes an own property) and the filters' maxDepth guard.

Architecture

  • Parser (src/parser.js) is the core. It consumes text and produces a SAX-like token stream: {name: 'startObject'}, {name: 'keyValue', value: 'key'}, {name: 'stringValue', value: '...'}, etc.
    • Uses stream-chain's gen(), flushable(), many(), none, fixUtf8Stream, and asStream. The default/parser is gen(fixUtf8Stream(), jsonParser()); the named jsonParser is the raw inner tokenizer (charCodeAt classification + whole-lexeme fast paths, falling back to an incremental regex machine).
    • Options: packKeys, packStrings, packNumbers, streamKeys, streamStrings, streamNumbers, jsonStreaming.
  • Assembler (src/assembler.js, implementation in src/core/assembler.js) interprets the token stream and reconstructs JavaScript objects. Plain class — no EventEmitter inheritance in 3.x.
    • Used internally by all streamers via streamBase.
    • Reads only packed tokens (keyValue, stringValue, numberValue); streamed chunks are ignored.
    • Materializes like JSON.parse: a __proto__ key becomes an own property via Object.defineProperty, never the prototype (plain assignment would hit the inherited setter). Same in FlexAssembler.
    • Assembler.connectTo(stream, {onDone: asm => …}) is substrate-aware: accepts either a Node Readable (attaches 'data' listener) or a Web ReadableStream (pumps via getReader()). Detection via typeof stream.getReader === 'function'. asm.onDone(fn) can set/clear the callback after construction.
    • For hot paths, prefer a manual for await (const tok of readable) asm.consume(tok) loop over connectTo — no async-closure overhead, errors propagate directly. FlexAssembler has the same shape.
    • asm.tapChain is a function for use in chain().
  • Disassembler (src/disassembler.js) does the inverse: JS objects → token stream.
  • Stringer (src/stringer.js) converts a token stream back to JSON text. Functional: flushable + asStream().
  • Emitter (src/emitter.js) factory returning a Writable that re-emits tokens as named EventEmitter events (subscribe with .on(name, fn)). Web counterpart (src/web/emitter.js) returns an EventTarget with a .writable WritableStream attached; each token dispatches as a CustomEvent(name, {detail: value}) (subscribe with .addEventListener(name, ev => ev.detail)). EventTarget + CustomEvent are universal across modern Node, Bun, Deno, and browsers. For hot paths, prefer the for await form over the emitter — same model, no per-token CustomEvent allocation, no listener-registry indirection.
  • Filters (src/filters/) edit the token stream: pick, replace, ignore, filter. All built on filterBase.
    • filterBase provides a state machine that tracks JSON path stack and applies accept/reject actions.
    • makeStackDiffer generates structural tokens to reconstruct the surrounding JSON envelope.
  • Streamers (src/streamers/) assemble complete JS objects from the token stream: streamValues, streamArray, streamObject. All built on streamBase.
    • streamBase uses Assembler internally and supports objectFilter for early rejection.
  • Utilities: emit(), withParser(), batch, verifier, FlexAssembler.
    • FlexAssembler (src/utils/flex-assembler.js) is a standalone clone of Assembler that supports custom containers via path-matching rules. Same API surface as Assembler.
    • withParser(fn, options) creates a gen(parser(options), fn(options)) pipeline — the most common pattern.
    • Most components export .withParser(options) and .withParserAsStream(options) static methods.
  • JSONL (deprecated — slated for removal in a future major): jsonl/parser.js and jsonl/stringer.js are thin re-exports of stream-chain's JSONL (the parser API was absorbed into stream-chain). The Node/Web wrappers delegate .asStream/.asWebStream to stream-chain's bundled stream-chain/node/jsonl/* and stream-chain/web/jsonl/* factories. Use stream-chain's JSONL directly. Rationale: stream-json is a JSON token library; JSONL yields whole objects per line and belongs in stream-chain with the other substrate components.
  • JSONC: jsonc/parser.js, jsonc/stringer.js, and jsonc/verifier.js for JSON with Comments. Extend the standard parser/stringer/verifier (same charCodeAt tokenizer/validator) with whitespace / comment / comma tokens, trailing comma support, and streamWhitespace/streamComments/packComments/streamCommas (parser) plus useCommentValues/useCommas (stringer) options. Comments mirror strings (startComment / commentChunk / endComment, packed commentValue) and their scan resumes across chunks — never rescan an accumulated comment from its start. streamCommas + useCommas give byte-faithful comma round-trips (incl. trailing commas) for streaming edits; both default off. Raw inner exports: jsoncParser, jsoncVerifier.
  • File I/O (Node-only) (src/file/, since 3.3.0): parseFile() is an input-edge stage that turns a file path into a token stream (gen(asyncBlockReader, jsonParser)); stringerToFile(path) is the symmetric output-edge sink (gen(stringer, asyncBlockWriter)); verifyFile(path) is a standalone async validator returning Promise<void>. JSONC variants under src/file/jsonc/. NOT mirrored in core/ or web/ because they use node:fs/promises. Compose with pipe(...) (one-shot driver with auto-flush — gen(...) alone doesn't flush, so stringerToFile wouldn't close the file) and drain(asyncGen) (returns the last yielded value or undefined). Both helpers live in core/utils/ (web-safe, no Node deps).
  • Substrate split: src/core/ holds the pure substrate-agnostic factories (no Node-stream imports — checked by tests/node/test-browser-safe.js which scans .d.ts for node:* imports and extends DuplexOptions). src/ (Node entry) attaches .asStream (Node Duplex) and .asWebStream (Web {readable, writable} pair). src/web/ attaches only .asWebStream and .withParserAsWebStream, with no Node-stream imports — safe for browser bundles. The chain from stream-chain (Node) or stream-chain/web (Web) auto-wraps the pure flushables on both substrates, so user-facing pipeline code is identical.

Writing tests

import test from 'tape-six';
import {parser} from '../src/index.js';
import {streamArray} from '../src/streamers/stream-array.js';
import chain from 'stream-chain';
import {Readable} from 'node:stream';

test('example', async t => {
  const output = [];
  const pipeline = chain([Readable.from(['[1, 2, 3]']), parser(), streamArray()]);
  pipeline.on('data', item => output.push(item));
  await new Promise(resolve => pipeline.on('end', resolve));
  t.deepEqual(
    output.map(o => o.value),
    [1, 2, 3]
  );
});
  • Test files use tape-six: .js for runtime tests, .ts for typed tests.
  • Test file naming convention: test-*.js (or test-types-*.ts) in tests/.
  • Tests are configured in package.json under the "tape6" section.
  • Property-based tests live in tests/node/test-property-parser.js (tape-six-fast-check, t.prop() over fast-check arbitraries): chunk-boundary invariance for the JSON and JSONC parsers and the two round-trips. A new streaming invariant belongs there, not in an example-based file.
  • Test files should be directly executable: node tests/test-foo.js.

Token protocol

The parser emits these token types:

Token name Value Meaning
startObject — { encountered
endObject — } encountered
startArray — [ encountered
endArray — ] encountered
startKey — Start of object key string
endKey — End of object key string
keyValue string Packed key value
startString — Start of string value
endString — End of string value
stringChunk string Piece of a string
stringValue string Packed string value
startNumber — Start of number
endNumber — End of number
numberChunk string Piece of a number
numberValue string Packed number (as string)
nullValue null null literal
trueValue true true literal
falseValue false false literal

Token is a discriminated union over name; TokenName is the closed name set. Stage shapes are named types on the parser entries: TokenSource (text → tokens), TokenTransform (tokens → tokens), TokenConsumer<Item> (tokens → items), TokenStringer (tokens → text). Streamer items are KeyedValue<K, T> (streamers/stream-base.js) — K is string for streamObject, number for streamArray/streamValues.

Key conventions

  • The only runtime dependency is stream-chain. Do not add others.
  • All public API is in src/. Keep .js and .d.ts files in sync.
  • Wiki documentation lives in the wiki/ submodule.
  • Most components follow the factory pattern: import {pick} from 'stream-json/filters/pick.js'.
  • Components that work with a parser typically export .withParser() and .withParserAsStream().
  • The Assembler.tapChain property returns a function suitable for use in chain().

When reading the codebase

  • Start with ARCHITECTURE.md for the module map and dependency graph.
  • src/parser.js is the core — read it first to understand the token protocol.
  • src/filters/filter-base.js is the foundation for all filters — read it to understand path matching.
  • src/streamers/stream-base.js is the foundation for all streamers — read it to understand object assembly.
  • Wiki markdown files in wiki/ contain detailed usage docs.