Skip to content

Commit f01384f

Browse files
committed
Updated AI docs.
1 parent d74229b commit f01384f

4 files changed

Lines changed: 31 additions & 20 deletions

File tree

AGENTS.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -185,6 +185,8 @@ The parser emits these token types:
185185
| `trueValue` | true | `true` literal |
186186
| `falseValue` | false | `false` literal |
187187

188+
`Token` is a discriminated union over `name`; `TokenName` is the closed name set. Stage shapes are named types on the parser entries: `TokenSource` (`text``tokens`), `TokenTransform` (`tokens``tokens`), `TokenConsumer<Item>` (`tokens` → items), `TokenStringer` (`tokens``text`). Streamer items are `KeyedValue<K, T>` (`streamers/stream-base.js`) — `K` is `string` for `streamObject`, `number` for `streamArray`/`streamValues`.
189+
188190
## Key conventions
189191

190192
- The only runtime dependency is `stream-chain`. Do not add others.

ARCHITECTURE.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,8 @@ The parser produces a stream of `{name, value}` tokens — a SAX-inspired protoc
108108

109109
All downstream components (filters, streamers, stringer, emitter) consume and/or produce tokens in this format. This is the universal interchange protocol of the library.
110110

111+
The typings name the stage shapes as exported aliases on the parser entries (`core/parser.d.ts`, re-exported by the Node and Web wrappers): `TokenSource` (`text``tokens`), `TokenTransform` (`tokens``tokens`), `TokenConsumer<Item>` (`tokens` → items), `TokenStringer` (`tokens``text`). Component factories declare their returns in these terms.
112+
111113
### How the Parser works
112114

113115
1. `parser(options)` returns a `gen(fixUtf8Stream(), jsonParser(options))` pipeline — a function for use in `chain()`.
@@ -172,6 +174,8 @@ All streamers are built on `streamBase` (`src/streamers/stream-base.js`):
172174
| `streamArray` | 1 | `{key: index, value: ...}` | Single top-level array |
173175
| `streamObject` | 1 | `{key: string, value: ...}` | Single top-level object |
174176

177+
The item shape is the exported `KeyedValue<K, T>` type (`core/streamers/stream-base.d.ts`): `K` is `string` for `streamObject`, `number` for `streamArray` and `streamValues`; the per-streamer `StreamXxxItem<T>` types are aliases of it.
178+
175179
### Utilities
176180

177181
- **`emit(stream)`** — attaches a `'data'` listener that re-emits each token as a named event on the stream.

llms-full.txt

Lines changed: 22 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,8 @@ By default, the parser emits both streamed tokens (`startString`/`stringChunk`/`
128128

129129
The token-type names form a closed set, exported as the `TokenName` type. `Token` is a discriminated union over `name` — narrowing on `token.name` (e.g. in a `switch`) tightens `token.value` per arm. Both are exported from `stream-json/parser.js` and `stream-json/core/parser.js`.
130130

131+
Four more exported aliases name the stage shapes: `TokenSource` (`text` → `tokens` — the parser), `TokenTransform` (`tokens` → `tokens` — the filters), `TokenConsumer<Item>` (`tokens` → items — the streamers), and `TokenStringer` (`tokens` → `text` — the stringer). All are exported from `stream-json/parser.js`, `stream-json/core/parser.js`, and `stream-json/web/parser.js`, and are available on the `parser` namespace (e.g. `parser.TokenTransform`). Streamer items have a common named shape too: `KeyedValue<K, T>` from `stream-json/streamers/stream-base.js` — see § Streamers.
132+
131133
## Main module
132134

133135
The default export is `parserStream` — an alias for `parser.asStream()`:
@@ -433,9 +435,10 @@ Extra option:
433435

434436
```js
435437
import {replace} from 'stream-json/filters/replace.js';
438+
import {stringer} from 'stream-json/stringer.js';
436439

437440
// Replace 'extra' with null
438-
chain([parser(), replace({filter: /^\d+\.extra\b/, replacement: [{name: 'nullValue', value: null}]}), Stringer.make()]);
441+
chain([parser(), replace({filter: /^\d+\.extra\b/, replacement: [{name: 'nullValue', value: null}]}), stringer()]);
439442

440443
// Replace with custom function
441444
chain([parser(), replace({
@@ -450,9 +453,10 @@ Removes matching subobjects completely. A variant of Replace with `replacement =
450453

451454
```js
452455
import {ignore} from 'stream-json/filters/ignore.js';
456+
import {stringer} from 'stream-json/stringer.js';
453457

454458
// Remove 'extra' properties
455-
chain([parser(), ignore({filter: /^\d+\.extra\b/}), Stringer.make()]);
459+
chain([parser(), ignore({filter: /^\d+\.extra\b/}), stringer()]);
456460
```
457461

458462
### filter(options)
@@ -464,9 +468,10 @@ Extra option:
464468

465469
```js
466470
import {filter} from 'stream-json/filters/filter.js';
471+
import {stringer} from 'stream-json/stringer.js';
467472

468473
// Keep only 'data', preserving outer structure: {"data": [...]}
469-
chain([parser(), filter({filter: /^data\b/}), Stringer.make()]);
474+
chain([parser(), filter({filter: /^data\b/}), stringer()]);
470475
```
471476

472477
### filterBase(config)
@@ -505,7 +510,7 @@ The differ honors `streamKeys`, `streamValues`, `packKeys`, and `pathSeparator`
505510

506511
## Streamers
507512

508-
All streamers are built on `streamBase` and produce `{key, value}` objects. Each is generic in the assembled value type — `streamArray<T>()`, `streamValues<T>()`, `streamObject<T>()` (and their `.withParser<T>()`) carry `T` through to the item's `value` field; the default is `unknown`.
513+
All streamers are built on `streamBase` and produce `{key, value}` objects. Each is generic in the assembled value type — `streamArray<T>()`, `streamValues<T>()`, `streamObject<T>()` (and their `.withParser<T>()`) carry `T` through to the item's `value` field; the default is `unknown`. The item shape is the exported `KeyedValue<K, T>` type (`stream-json/streamers/stream-base.js`): `K` is `string` for `streamObject` (the property name) and `number` for `streamArray` (the array index) and `streamValues` (a sequential counter); `StreamArrayItem<T>` / `StreamObjectItem<T>` / `StreamValuesItem<T>` are aliases of it.
509514

510515
Common option:
511516
- `objectFilter` (function) `(asm) => boolean|null` — called during assembly. Return `true` to accept, `false` to reject (abandon assembly), `null`/`undefined` for undecided.
@@ -751,20 +756,20 @@ Options:
751756
- **any value** — lines that fail to parse produce this value instead, or are skipped if `undefined`.
752757

753758
```js
754-
import JsonlParser from 'stream-json/jsonl/parser.js';
759+
import jsonlParser from 'stream-json/jsonl/parser.js';
755760
import chain from 'stream-chain';
756761
import fs from 'node:fs';
757762

758763
chain([
759764
fs.createReadStream('data.jsonl'),
760-
JsonlParser.make(),
765+
jsonlParser(),
761766
({key, value}) => console.log(key, value)
762767
]);
763768

764769
// Silently skip bad lines
765770
chain([
766771
fs.createReadStream('data.jsonl'),
767-
JsonlParser.make({errorIndicator: undefined}),
772+
jsonlParser({errorIndicator: undefined}),
768773
({key, value}) => processItem(value)
769774
]);
770775
```
@@ -1040,13 +1045,13 @@ chain([
10401045

10411046
```js
10421047
import {ignore} from 'stream-json/filters/ignore.js';
1043-
import Stringer from 'stream-json/stringer.js';
1048+
import stringer from 'stream-json/stringer.js';
10441049

10451050
chain([
10461051
fs.createReadStream('input.json'),
10471052
parser(),
10481053
ignore({filter: /\bsecret\b/}),
1049-
Stringer.make(),
1054+
stringer(),
10501055
fs.createWriteStream('output.json')
10511056
]);
10521057
```
@@ -1089,14 +1094,14 @@ chain([
10891094
### JSONL roundtrip
10901095

10911096
```js
1092-
import JsonlParser from 'stream-json/jsonl/parser.js';
1093-
import JsonlStringer from 'stream-json/jsonl/stringer.js';
1097+
import jsonlParser from 'stream-json/jsonl/parser.js';
1098+
import jsonlStringer from 'stream-json/jsonl/stringer.js';
10941099

10951100
chain([
10961101
fs.createReadStream('input.jsonl'),
1097-
JsonlParser.make(),
1102+
jsonlParser(),
10981103
({value}) => transform(value),
1099-
JsonlStringer.make(),
1104+
jsonlStringer(),
11001105
fs.createWriteStream('output.jsonl')
11011106
]);
11021107
```
@@ -1145,13 +1150,13 @@ chain([
11451150
### Batch processing
11461151

11471152
```js
1148-
import Batch from 'stream-json/utils/batch.js';
1153+
import batch from 'stream-json/utils/batch.js';
11491154

11501155
chain([
11511156
fs.createReadStream('data.json'),
11521157
parser(),
11531158
streamArray(),
1154-
Batch.make({batchSize: 100}),
1159+
batch({batchSize: 100}),
11551160
async batch => {
11561161
await db.insertMany(batch.map(({value}) => value));
11571162
return chain.none;
@@ -1162,9 +1167,9 @@ chain([
11621167
### JSON validation
11631168

11641169
```js
1165-
import Verifier from 'stream-json/utils/verifier.js';
1170+
import verifier from 'stream-json/utils/verifier.js';
11661171

1167-
const v = Verifier.make();
1172+
const v = verifier.asStream();
11681173
v.on('error', err => {
11691174
console.error(`Invalid at offset ${err.offset}, line ${err.line}, pos ${err.pos}: ${err.message}`);
11701175
});

llms.txt

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -143,7 +143,7 @@ chain([
143143

144144
## Streamers
145145

146-
Assemble complete JS objects from a token stream. All produce `{key, value}` objects, generic in the assembled value type (`streamArray<T>()`, `streamValues<T>()`, `streamObject<T>()`; `value` defaults to `unknown`).
146+
Assemble complete JS objects from a token stream. All produce `{key, value}` objects, generic in the assembled value type (`streamArray<T>()`, `streamValues<T>()`, `streamObject<T>()`; `value` defaults to `unknown`). The item shape is the exported `KeyedValue<K, T>` type — `key` is a `string` for `streamObject`, a `number` for the others.
147147

148148
- **`streamValues(options)`** — streams successive JSON values. Use with `jsonStreaming` or after `pick`.
149149
- **`streamArray(options)`** — streams elements of a single top-level array.
@@ -265,7 +265,7 @@ chain([
265265
fs.createReadStream('input.json'),
266266
parser(),
267267
ignore({filter: /\bsecret\b/}),
268-
Stringer.make(),
268+
stringer(),
269269
fs.createWriteStream('output.json')
270270
]);
271271
```
@@ -274,7 +274,7 @@ chain([
274274

275275
The parser emits `{name, value}` tokens: `startObject`, `endObject`, `startArray`, `endArray`, `startKey`, `endKey`, `keyValue`, `startString`, `endString`, `stringChunk`, `stringValue`, `startNumber`, `endNumber`, `numberChunk`, `numberValue`, `nullValue`, `trueValue`, `falseValue`.
276276

277-
These names are the closed `TokenName` type; `Token` is a discriminated union over `name` (narrowing on `token.name` tightens `token.value`). Both are exported from `stream-json/parser.js`.
277+
These names are the closed `TokenName` type; `Token` is a discriminated union over `name` (narrowing on `token.name` tightens `token.value`). Both are exported from `stream-json/parser.js`. Stage shapes are named types too: `TokenSource` (`text` → `tokens`), `TokenTransform` (`tokens` → `tokens`), `TokenConsumer<Item>` (`tokens` → items), `TokenStringer` (`tokens` → `text`) — same exports; streamer items are `KeyedValue<K, T>` from `stream-json/streamers/stream-base.js`.
278278

279279
## Links
280280

0 commit comments

Comments
 (0)