You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: AGENTS.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -185,6 +185,8 @@ The parser emits these token types:
185
185
|`trueValue`| true |`true` literal |
186
186
|`falseValue`| false |`false` literal |
187
187
188
+
`Token` is a discriminated union over `name`; `TokenName` is the closed name set. Stage shapes are named types on the parser entries: `TokenSource` (`text` → `tokens`), `TokenTransform` (`tokens` → `tokens`), `TokenConsumer<Item>` (`tokens` → items), `TokenStringer` (`tokens` → `text`). Streamer items are `KeyedValue<K, T>` (`streamers/stream-base.js`) — `K` is `string` for `streamObject`, `number` for `streamArray`/`streamValues`.
189
+
188
190
## Key conventions
189
191
190
192
- The only runtime dependency is `stream-chain`. Do not add others.
Copy file name to clipboardExpand all lines: ARCHITECTURE.md
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -108,6 +108,8 @@ The parser produces a stream of `{name, value}` tokens — a SAX-inspired protoc
108
108
109
109
All downstream components (filters, streamers, stringer, emitter) consume and/or produce tokens in this format. This is the universal interchange protocol of the library.
110
110
111
+
The typings name the stage shapes as exported aliases on the parser entries (`core/parser.d.ts`, re-exported by the Node and Web wrappers): `TokenSource` (`text` → `tokens`), `TokenTransform` (`tokens` → `tokens`), `TokenConsumer<Item>` (`tokens` → items), `TokenStringer` (`tokens` → `text`). Component factories declare their returns in these terms.
112
+
111
113
### How the Parser works
112
114
113
115
1.`parser(options)` returns a `gen(fixUtf8Stream(), jsonParser(options))` pipeline — a function for use in `chain()`.
@@ -172,6 +174,8 @@ All streamers are built on `streamBase` (`src/streamers/stream-base.js`):
172
174
|`streamArray`| 1 |`{key: index, value: ...}`| Single top-level array |
173
175
|`streamObject`| 1 |`{key: string, value: ...}`| Single top-level object |
174
176
177
+
The item shape is the exported `KeyedValue<K, T>` type (`core/streamers/stream-base.d.ts`): `K` is `string` for `streamObject`, `number` for `streamArray` and `streamValues`; the per-streamer `StreamXxxItem<T>` types are aliases of it.
178
+
175
179
### Utilities
176
180
177
181
-**`emit(stream)`** — attaches a `'data'` listener that re-emits each token as a named event on the stream.
Copy file name to clipboardExpand all lines: llms-full.txt
+22-17Lines changed: 22 additions & 17 deletions
Original file line number
Diff line number
Diff line change
@@ -128,6 +128,8 @@ By default, the parser emits both streamed tokens (`startString`/`stringChunk`/`
128
128
129
129
The token-type names form a closed set, exported as the `TokenName` type. `Token` is a discriminated union over `name` — narrowing on `token.name` (e.g. in a `switch`) tightens `token.value` per arm. Both are exported from `stream-json/parser.js` and `stream-json/core/parser.js`.
130
130
131
+
Four more exported aliases name the stage shapes: `TokenSource` (`text` → `tokens` — the parser), `TokenTransform` (`tokens` → `tokens` — the filters), `TokenConsumer<Item>` (`tokens` → items — the streamers), and `TokenStringer` (`tokens` → `text` — the stringer). All are exported from `stream-json/parser.js`, `stream-json/core/parser.js`, and `stream-json/web/parser.js`, and are available on the `parser` namespace (e.g. `parser.TokenTransform`). Streamer items have a common named shape too: `KeyedValue<K, T>` from `stream-json/streamers/stream-base.js` — see § Streamers.
132
+
131
133
## Main module
132
134
133
135
The default export is `parserStream` — an alias for `parser.asStream()`:
@@ -433,9 +435,10 @@ Extra option:
433
435
434
436
```js
435
437
import {replace} from 'stream-json/filters/replace.js';
@@ -505,7 +510,7 @@ The differ honors `streamKeys`, `streamValues`, `packKeys`, and `pathSeparator`
505
510
506
511
## Streamers
507
512
508
-
All streamers are built on `streamBase` and produce `{key, value}` objects. Each is generic in the assembled value type — `streamArray<T>()`, `streamValues<T>()`, `streamObject<T>()` (and their `.withParser<T>()`) carry `T` through to the item's `value` field; the default is `unknown`.
513
+
All streamers are built on `streamBase` and produce `{key, value}` objects. Each is generic in the assembled value type — `streamArray<T>()`, `streamValues<T>()`, `streamObject<T>()` (and their `.withParser<T>()`) carry `T` through to the item's `value` field; the default is `unknown`. The item shape is the exported `KeyedValue<K, T>` type (`stream-json/streamers/stream-base.js`): `K` is `string` for `streamObject` (the property name) and `number` for `streamArray` (the array index) and `streamValues` (a sequential counter); `StreamArrayItem<T>` / `StreamObjectItem<T>` / `StreamValuesItem<T>` are aliases of it.
509
514
510
515
Common option:
511
516
- `objectFilter` (function) `(asm) => boolean|null` — called during assembly. Return `true` to accept, `false` to reject (abandon assembly), `null`/`undefined` for undecided.
@@ -751,20 +756,20 @@ Options:
751
756
- **any value** — lines that fail to parse produce this value instead, or are skipped if `undefined`.
752
757
753
758
```js
754
-
import JsonlParser from 'stream-json/jsonl/parser.js';
759
+
import jsonlParser from 'stream-json/jsonl/parser.js';
755
760
import chain from 'stream-chain';
756
761
import fs from 'node:fs';
757
762
758
763
chain([
759
764
fs.createReadStream('data.jsonl'),
760
-
JsonlParser.make(),
765
+
jsonlParser(),
761
766
({key, value}) => console.log(key, value)
762
767
]);
763
768
764
769
// Silently skip bad lines
765
770
chain([
766
771
fs.createReadStream('data.jsonl'),
767
-
JsonlParser.make({errorIndicator: undefined}),
772
+
jsonlParser({errorIndicator: undefined}),
768
773
({key, value}) => processItem(value)
769
774
]);
770
775
```
@@ -1040,13 +1045,13 @@ chain([
1040
1045
1041
1046
```js
1042
1047
import {ignore} from 'stream-json/filters/ignore.js';
1043
-
import Stringer from 'stream-json/stringer.js';
1048
+
import stringer from 'stream-json/stringer.js';
1044
1049
1045
1050
chain([
1046
1051
fs.createReadStream('input.json'),
1047
1052
parser(),
1048
1053
ignore({filter: /\bsecret\b/}),
1049
-
Stringer.make(),
1054
+
stringer(),
1050
1055
fs.createWriteStream('output.json')
1051
1056
]);
1052
1057
```
@@ -1089,14 +1094,14 @@ chain([
1089
1094
### JSONL roundtrip
1090
1095
1091
1096
```js
1092
-
import JsonlParser from 'stream-json/jsonl/parser.js';
1093
-
import JsonlStringer from 'stream-json/jsonl/stringer.js';
1097
+
import jsonlParser from 'stream-json/jsonl/parser.js';
1098
+
import jsonlStringer from 'stream-json/jsonl/stringer.js';
Copy file name to clipboardExpand all lines: llms.txt
+3-3Lines changed: 3 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -143,7 +143,7 @@ chain([
143
143
144
144
## Streamers
145
145
146
-
Assemble complete JS objects from a token stream. All produce `{key, value}` objects, generic in the assembled value type (`streamArray<T>()`, `streamValues<T>()`, `streamObject<T>()`; `value` defaults to `unknown`).
146
+
Assemble complete JS objects from a token stream. All produce `{key, value}` objects, generic in the assembled value type (`streamArray<T>()`, `streamValues<T>()`, `streamObject<T>()`; `value` defaults to `unknown`). The item shape is the exported `KeyedValue<K, T>` type — `key` is a `string` for `streamObject`, a `number` for the others.
147
147
148
148
- **`streamValues(options)`** — streams successive JSON values. Use with `jsonStreaming` or after `pick`.
149
149
- **`streamArray(options)`** — streams elements of a single top-level array.
These names are the closed `TokenName` type; `Token` is a discriminated union over `name` (narrowing on `token.name` tightens `token.value`). Both are exported from `stream-json/parser.js`.
277
+
These names are the closed `TokenName` type; `Token` is a discriminated union over `name` (narrowing on `token.name` tightens `token.value`). Both are exported from `stream-json/parser.js`. Stage shapes are named types too: `TokenSource` (`text` → `tokens`), `TokenTransform` (`tokens` → `tokens`), `TokenConsumer<Item>` (`tokens` → items), `TokenStringer` (`tokens` → `text`) — same exports; streamer items are `KeyedValue<K, T>` from `stream-json/streamers/stream-base.js`.
0 commit comments