|
3 | 3 | [npm-image]: https://img.shields.io/npm/v/stream-json.svg |
4 | 4 | [npm-url]: https://npmjs.org/package/stream-json |
5 | 5 |
|
6 | | -`stream-json` is a micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. It can parse JSON files far exceeding available memory. Even individual data items (keys, strings, and numbers) can be streamed piece-wise. A SAX-inspired event-based API is included. |
7 | | - |
8 | | -Components: |
9 | | - |
10 | | -- **[Parser](https://github.com/uhop/stream-json/wiki/Parser)** — streaming JSON parser producing a SAX-like token stream. |
11 | | - - Optionally packs keys, strings, and numbers (controlled separately). |
12 | | - - The [main module](https://github.com/uhop/stream-json/wiki/Main-module) creates a parser decorated with `emit()`. |
13 | | -- **Filters** edit a token stream: |
14 | | - - [Pick](https://github.com/uhop/stream-json/wiki/Pick) — selects matching subobjects, ignoring the rest. |
15 | | - - [Replace](https://github.com/uhop/stream-json/wiki/Replace) — substitutes matching subobjects with a replacement. |
16 | | - - [Ignore](https://github.com/uhop/stream-json/wiki/Ignore) — removes matching subobjects entirely. |
17 | | - - [Filter](https://github.com/uhop/stream-json/wiki/Filter) — filters subobjects while preserving the JSON shape. |
18 | | -- **Streamers** assemble tokens into JavaScript objects: |
19 | | - - [StreamValues](https://github.com/uhop/stream-json/wiki/StreamValues) — streams successive JSON values (for JSON Streaming or after `pick()`). |
20 | | - - [StreamArray](https://github.com/uhop/stream-json/wiki/StreamArray) — streams elements of a top-level array. |
21 | | - - [StreamObject](https://github.com/uhop/stream-json/wiki/StreamObject) — streams top-level properties of an object. |
22 | | -- **Essentials:** |
23 | | - - [Assembler](https://github.com/uhop/stream-json/wiki/Assembler) — reconstructs JavaScript objects from tokens (EventEmitter). |
24 | | - - [Disassembler](https://github.com/uhop/stream-json/wiki/Disassembler) — converts JavaScript objects into a token stream. |
25 | | - - [Stringer](https://github.com/uhop/stream-json/wiki/Stringer) — converts a token stream back into JSON text. |
26 | | - - [Emitter](https://github.com/uhop/stream-json/wiki/Emitter) — re-emits tokens as named events. |
27 | | -- **Utilities:** |
28 | | - - [emit()](<https://github.com/uhop/stream-json/wiki/emit()>) — attaches token events to any stream. |
29 | | - - [withParser()](<https://github.com/uhop/stream-json/wiki/withParser()>) — creates parser + component pipelines. |
30 | | - - [Batch](https://github.com/uhop/stream-json/wiki/Batch) — groups items into arrays. |
31 | | - - [Verifier](https://github.com/uhop/stream-json/wiki/Verifier) — validates JSON text, pinpoints errors. |
32 | | - - [FlexAssembler](https://github.com/uhop/stream-json/wiki/FlexAssembler) — Assembler with custom containers (Map, Set, etc.) at specific paths. |
33 | | -- **JSONL** ([JSON Lines](http://jsonlines.org/) / [NDJSON](http://ndjson.org/)) — **⚠️ deprecated; use [`stream-chain`'s JSONL](https://github.com/uhop/stream-chain/wiki/jsonl-parser) directly.** stream-json's JSONL is now a thin re-export of stream-chain's (which carries the full `reviver` / `errorIndicator` API) and is slated for removal in a future major — JSONL yields whole objects per line and belongs in stream-chain, not in this token-oriented library. |
34 | | - - [jsonl/Parser](https://github.com/uhop/stream-json/wiki/jsonl-Parser) — parses JSONL into `{key, value}` objects. Faster than `parser({jsonStreaming: true})` + `streamValues()` when items fit in memory. |
35 | | - - [jsonl/Stringer](https://github.com/uhop/stream-json/wiki/jsonl-Stringer) — serializes objects to JSONL text. Faster than `disassembler()` + `stringer()`. |
36 | | -- **JSONC** ([JSON with Comments](https://jsonc.org/)): |
37 | | - - [jsonc/Parser](https://github.com/uhop/stream-json/wiki/jsonc-Parser) — streaming JSONC parser with comment and whitespace tokens, plus optional `comma` tokens (`streamCommas`) for faithful round-trip editing. |
38 | | - - [jsonc/Stringer](https://github.com/uhop/stream-json/wiki/jsonc-Stringer) — converts JSONC token streams back to text; with `useCommas` it reproduces comma placement (incl. trailing commas) exactly. |
39 | | - - [jsonc/Verifier](https://github.com/uhop/stream-json/wiki/jsonc-Verifier) — validates JSONC text, pinpoints errors. |
40 | | - |
41 | | -All components are building blocks for custom data processing pipelines. They can be combined with each other and with custom code via [stream-chain](https://www.npmjs.com/package/stream-chain). |
42 | | - |
43 | | -Distributed under the New BSD license. |
44 | | - |
45 | | -## Introduction |
| 6 | +`stream-json` is a micro-library of components for processing JSON files and streams, with a minimal memory footprint. Point it at a document far larger than available memory and it streams straight through — you pick out only the parts you care about and handle them one at a time, instead of loading the whole thing with `JSON.parse`. Even individual keys, strings, and numbers can be streamed piece by piece, and a SAX-inspired event API is included. |
| 7 | + |
| 8 | +Each component is one stage of a pipeline: the parser turns text into a token stream, filters trim and reshape that stream on the fly, and streamers assemble the surviving tokens back into JavaScript objects. They compose with each other and with your own code through [stream-chain](https://www.npmjs.com/package/stream-chain), the zero-dependency library this one is built on; TypeScript typings are bundled. |
| 9 | + |
| 10 | +Why it might be for you: |
| 11 | + |
| 12 | +- **Surgical.** `pick`, `ignore`, `replace`, and `filter` keep just the subobjects you want out of a massive document and drop the rest — the bytes you skip are never assembled into memory. |
| 13 | +- **Composable.** Every component is an ordinary pipeline stage. Mix them with each other, with plain functions and generators, and with any Node or Web stream. |
| 14 | +- **Performance-minded.** The parser and assemblers are measured and tuned along the hot paths. Real numbers depend on your data and hardware, so [benchmark](https://github.com/uhop/stream-json/wiki/Benchmarks) on your own. |
| 15 | +- **Solid.** ESM, bundled TypeScript typings, and a broad test suite exercised across Node, Bun, Deno, and the browser. |
| 16 | + |
| 17 | +## Example |
| 18 | + |
| 19 | +Pull one array out of a JSON document larger than memory and tally it — one record at a time, in constant memory: |
46 | 20 |
|
47 | 21 | ```js |
48 | 22 | import chain from 'stream-chain'; |
49 | | - |
50 | 23 | import {parser} from 'stream-json'; |
51 | 24 | import {pick} from 'stream-json/filters/pick.js'; |
52 | | -import {ignore} from 'stream-json/filters/ignore.js'; |
53 | | -import {streamValues} from 'stream-json/streamers/stream-values.js'; |
54 | | - |
| 25 | +import {streamArray} from 'stream-json/streamers/stream-array.js'; |
55 | 26 | import fs from 'node:fs'; |
56 | | -import zlib from 'node:zlib'; |
57 | 27 |
|
| 28 | +// data.json: { "meta": {...}, "data": [ ...millions of records... ] } |
58 | 29 | const pipeline = chain([ |
59 | | - fs.createReadStream('sample.json.gz'), |
60 | | - zlib.createGunzip(), |
| 30 | + fs.createReadStream('data.json'), // a file far bigger than RAM is fine |
61 | 31 | parser(), |
62 | | - pick({filter: 'data'}), |
63 | | - ignore({filter: /\b_meta\b/i}), |
64 | | - streamValues(), |
65 | | - data => { |
66 | | - const value = data.value; |
67 | | - // keep data only for the accounting department |
68 | | - return value && value.department === 'accounting' ? data : null; |
69 | | - } |
| 32 | + pick({filter: 'data'}), // descend into "data", ignore everything else |
| 33 | + streamArray() // emit one array element at a time |
70 | 34 | ]); |
71 | 35 |
|
72 | | -let counter = 0; |
73 | | -pipeline.on('data', () => ++counter); |
74 | | -pipeline.on('end', () => console.log(`The accounting department has ${counter} employees.`)); |
| 36 | +const byDepartment = {}; |
| 37 | +pipeline.on('data', ({value}) => { |
| 38 | + byDepartment[value.department] = (byDepartment[value.department] ?? 0) + 1; |
| 39 | +}); |
| 40 | +pipeline.on('end', () => console.log(byDepartment)); |
75 | 41 | ``` |
76 | 42 |
|
77 | | -`stream-json` 3.x is ESM-only and requires Node.js 22+. The default Node-flavored entries (`stream-json/...`) attach both `.asStream` (Node `Duplex`) and `.asWebStream` (Web Streams pair) on every component, since modern Node and Bun support both stream flavors natively. For browser bundles, import from the `stream-json/web/...` subpath instead — it pulls no Node-stream code into the dep graph. Advanced consumers can also import from `stream-json/core/...` to get bare factories with no adapters attached. See [Migrating from 2.x to 3.x](https://github.com/uhop/stream-json/wiki/Migrating-from-2.x-to-3.x). |
78 | | - |
79 | | -See the full documentation in [Wiki](https://github.com/uhop/stream-json/wiki). |
80 | | - |
81 | | -Companion projects: |
82 | | - |
83 | | -- [stream-csv-as-json](https://www.npmjs.com/package/stream-csv-as-json) streams huge CSV files in a format compatible with `stream-json`: |
84 | | - rows as arrays of string values. If a header row is used, it can stream rows as objects with named fields. |
| 43 | +Each stage is a building block; `stream-chain` wires them into one stream and handles the streaming and backpressure. For input straight from a file you can drop `createReadStream` in favor of the Node-only [file edges](https://github.com/uhop/stream-json/wiki/parseFile) (`parseFile()`); to write a token stream back to disk, use [stringerToFile()](https://github.com/uhop/stream-json/wiki/stringerToFile). See [Recipes](https://github.com/uhop/stream-json/wiki/Recipes) for more. |
85 | 44 |
|
86 | 45 | ## Installation |
87 | 46 |
|
88 | 47 | ```bash |
89 | 48 | npm install --save stream-json |
90 | | -# or: yarn add stream-json |
91 | 49 | ``` |
92 | 50 |
|
93 | | -## Use |
| 51 | +## What's in the box |
| 52 | +
|
| 53 | +- **[Parser](https://github.com/uhop/stream-json/wiki/Parser)** — the streaming JSON parser producing a SAX-like token stream (optionally packing keys, strings, and numbers). The [main module](https://github.com/uhop/stream-json/wiki/Main-module) decorates it with `emit()`. |
| 54 | +- **Filters** — edit a token stream on the fly: [pick](https://github.com/uhop/stream-json/wiki/Pick), [replace](https://github.com/uhop/stream-json/wiki/Replace), [ignore](https://github.com/uhop/stream-json/wiki/Ignore), [filter](https://github.com/uhop/stream-json/wiki/Filter). |
| 55 | +- **Streamers** — assemble tokens into JavaScript objects: [streamValues](https://github.com/uhop/stream-json/wiki/StreamValues), [streamArray](https://github.com/uhop/stream-json/wiki/StreamArray), [streamObject](https://github.com/uhop/stream-json/wiki/StreamObject). |
| 56 | +- **Essentials** — [Assembler](https://github.com/uhop/stream-json/wiki/Assembler), [Disassembler](https://github.com/uhop/stream-json/wiki/Disassembler), [Stringer](https://github.com/uhop/stream-json/wiki/Stringer), [Emitter](https://github.com/uhop/stream-json/wiki/Emitter). |
| 57 | +- **Utilities** — [emit()](<https://github.com/uhop/stream-json/wiki/emit()>), [withParser()](<https://github.com/uhop/stream-json/wiki/withParser()>), [Batch](https://github.com/uhop/stream-json/wiki/Batch), [Verifier](https://github.com/uhop/stream-json/wiki/Verifier), [FlexAssembler](https://github.com/uhop/stream-json/wiki/FlexAssembler), and Node-only file edges ([parseFile](https://github.com/uhop/stream-json/wiki/parseFile), [stringerToFile](https://github.com/uhop/stream-json/wiki/stringerToFile), [verifyFile](https://github.com/uhop/stream-json/wiki/verifyFile)). |
| 58 | +- **[JSONC](https://github.com/uhop/stream-json/wiki/jsonc-Parser)** ([JSON with Comments](https://jsonc.org/)) — streaming parser, stringer, and verifier, with faithful comma round-trip (`streamCommas` / `useCommas`). |
| 59 | +- **JSONL** ([JSON Lines](https://jsonlines.org/)) — **deprecated**: now a thin re-export of [stream-chain's JSONL](https://github.com/uhop/stream-chain/wiki/jsonl), slated for removal in a future major. |
| 60 | +- **Subpaths** — default `stream-json/...` entries attach both `.asStream` (Node `Duplex`) and `.asWebStream` (Web Streams) to every component; import from `stream-json/web/...` for browser bundles (no Node-stream code in the graph), or `stream-json/core/...` for bare factories. ESM-only, Node 22+. |
| 61 | +
|
| 62 | +Full documentation is in the **[wiki](https://github.com/uhop/stream-json/wiki)** — browse the [index](https://github.com/uhop/stream-json/wiki/Home), or [search it](https://uhop.github.io/wiki-search/app/?wiki=uhop/stream-json) by name. |
| 63 | +
|
| 64 | +## Companion projects |
| 65 | +
|
| 66 | +- [stream-chain](https://www.npmjs.com/package/stream-chain) — the pipeline-composition substrate `stream-json` is built on (wire functions, generators, and streams into one chain); also home to streaming [JSONL](https://github.com/uhop/stream-chain/wiki/jsonl). |
| 67 | +- [stream-csv-as-json](https://www.npmjs.com/package/stream-csv-as-json) — streams huge CSV files in a `stream-json`-compatible token format: rows as arrays of strings, or as objects when a header row is present. |
94 | 68 |
|
95 | | -The library is organized as small composable components based on Node.js [streams](http://nodejs.org/api/stream.html) and [events](http://nodejs.org/api/events.html). The source code is compact — read it to understand how things work and to build your own components. |
| 69 | +## License |
96 | 70 |
|
97 | | -Bug reports, simplifications, and new generic components are welcome — open a ticket or pull request. |
| 71 | +BSD-3-Clause |
98 | 72 |
|
99 | 73 | ## Release History |
100 | 74 |
|
101 | 75 | - 3.3.0 _File I/O components (`parseFile`, `stringerToFile`, `verifyFile`), faithful JSONC comma round-trip (`streamCommas` / `useCommas`), JSONL delegated to `stream-chain`._ |
102 | 76 | - 3.2.0 _Improvements in TS typings, faster JSON parser._ |
103 | 77 | - 3.1.0 _Web Streams parity sweep._ |
104 | 78 | - 3.0.0 _Moved to ESM using `stream-chain` 4.x. See [Migrating from 2.x to 3.x](https://github.com/uhop/stream-json/wiki/Migrating-from-2.x-to-3.x)._ |
105 | | -- 2.1.0 _new: [jsonc/Verifier](https://github.com/uhop/stream-json/wiki/jsonc-Verifier) — validates JSONC text with exact error locations. Parser performance improvements (pre-allocated token singletons)._ |
| 79 | +- 2.1.0 _new: [jsonc/Verifier](https://github.com/uhop/stream-json/wiki/jsonc-Verifier) — validates JSONC text with exact error locations. Parser performance improvements (pre-allocated token singletons)._ |
106 | 80 | - 2.0.0 _major rewrite: functional API based on `stream-chain` 3.x, bundled TypeScript definitions. New: JSONC parser/stringer, FlexAssembler. See [Migrating from 1.x to 2.x](https://github.com/uhop/stream-json/wiki/Migrating-from-1.x-to-2.x)._ |
107 | 81 |
|
108 | 82 | The full history is in the wiki: [Release history](https://github.com/uhop/stream-json/wiki/Release-history). |
0 commit comments