Skip to content

Commit cb3066e

Browse files
committed
Worked on README.
1 parent 6363b75 commit cb3066e

1 file changed

Lines changed: 45 additions & 71 deletions

File tree

README.md

Lines changed: 45 additions & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -3,106 +3,80 @@
33
[npm-image]: https://img.shields.io/npm/v/stream-json.svg
44
[npm-url]: https://npmjs.org/package/stream-json
55

6-
`stream-json` is a micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. It can parse JSON files far exceeding available memory. Even individual data items (keys, strings, and numbers) can be streamed piece-wise. A SAX-inspired event-based API is included.
7-
8-
Components:
9-
10-
- **[Parser](https://github.com/uhop/stream-json/wiki/Parser)** — streaming JSON parser producing a SAX-like token stream.
11-
- Optionally packs keys, strings, and numbers (controlled separately).
12-
- The [main module](https://github.com/uhop/stream-json/wiki/Main-module) creates a parser decorated with `emit()`.
13-
- **Filters** edit a token stream:
14-
- [Pick](https://github.com/uhop/stream-json/wiki/Pick) — selects matching subobjects, ignoring the rest.
15-
- [Replace](https://github.com/uhop/stream-json/wiki/Replace) — substitutes matching subobjects with a replacement.
16-
- [Ignore](https://github.com/uhop/stream-json/wiki/Ignore) — removes matching subobjects entirely.
17-
- [Filter](https://github.com/uhop/stream-json/wiki/Filter) — filters subobjects while preserving the JSON shape.
18-
- **Streamers** assemble tokens into JavaScript objects:
19-
- [StreamValues](https://github.com/uhop/stream-json/wiki/StreamValues) — streams successive JSON values (for JSON Streaming or after `pick()`).
20-
- [StreamArray](https://github.com/uhop/stream-json/wiki/StreamArray) — streams elements of a top-level array.
21-
- [StreamObject](https://github.com/uhop/stream-json/wiki/StreamObject) — streams top-level properties of an object.
22-
- **Essentials:**
23-
- [Assembler](https://github.com/uhop/stream-json/wiki/Assembler) — reconstructs JavaScript objects from tokens (EventEmitter).
24-
- [Disassembler](https://github.com/uhop/stream-json/wiki/Disassembler) — converts JavaScript objects into a token stream.
25-
- [Stringer](https://github.com/uhop/stream-json/wiki/Stringer) — converts a token stream back into JSON text.
26-
- [Emitter](https://github.com/uhop/stream-json/wiki/Emitter) — re-emits tokens as named events.
27-
- **Utilities:**
28-
- [emit()](<https://github.com/uhop/stream-json/wiki/emit()>) — attaches token events to any stream.
29-
- [withParser()](<https://github.com/uhop/stream-json/wiki/withParser()>) — creates parser + component pipelines.
30-
- [Batch](https://github.com/uhop/stream-json/wiki/Batch) — groups items into arrays.
31-
- [Verifier](https://github.com/uhop/stream-json/wiki/Verifier) — validates JSON text, pinpoints errors.
32-
- [FlexAssembler](https://github.com/uhop/stream-json/wiki/FlexAssembler) — Assembler with custom containers (Map, Set, etc.) at specific paths.
33-
- **JSONL** ([JSON Lines](http://jsonlines.org/) / [NDJSON](http://ndjson.org/)) — **⚠️ deprecated; use [`stream-chain`'s JSONL](https://github.com/uhop/stream-chain/wiki/jsonl-parser) directly.** stream-json's JSONL is now a thin re-export of stream-chain's (which carries the full `reviver` / `errorIndicator` API) and is slated for removal in a future major — JSONL yields whole objects per line and belongs in stream-chain, not in this token-oriented library.
34-
- [jsonl/Parser](https://github.com/uhop/stream-json/wiki/jsonl-Parser) — parses JSONL into `{key, value}` objects. Faster than `parser({jsonStreaming: true})` + `streamValues()` when items fit in memory.
35-
- [jsonl/Stringer](https://github.com/uhop/stream-json/wiki/jsonl-Stringer) — serializes objects to JSONL text. Faster than `disassembler()` + `stringer()`.
36-
- **JSONC** ([JSON with Comments](https://jsonc.org/)):
37-
- [jsonc/Parser](https://github.com/uhop/stream-json/wiki/jsonc-Parser) — streaming JSONC parser with comment and whitespace tokens, plus optional `comma` tokens (`streamCommas`) for faithful round-trip editing.
38-
- [jsonc/Stringer](https://github.com/uhop/stream-json/wiki/jsonc-Stringer) — converts JSONC token streams back to text; with `useCommas` it reproduces comma placement (incl. trailing commas) exactly.
39-
- [jsonc/Verifier](https://github.com/uhop/stream-json/wiki/jsonc-Verifier) — validates JSONC text, pinpoints errors.
40-
41-
All components are building blocks for custom data processing pipelines. They can be combined with each other and with custom code via [stream-chain](https://www.npmjs.com/package/stream-chain).
42-
43-
Distributed under the New BSD license.
44-
45-
## Introduction
6+
`stream-json` is a micro-library of components for processing JSON files and streams, with a minimal memory footprint. Point it at a document far larger than available memory and it streams straight through &mdash; you pick out only the parts you care about and handle them one at a time, instead of loading the whole thing with `JSON.parse`. Even individual keys, strings, and numbers can be streamed piece by piece, and a SAX-inspired event API is included.
7+
8+
Each component is one stage of a pipeline: the parser turns text into a token stream, filters trim and reshape that stream on the fly, and streamers assemble the surviving tokens back into JavaScript objects. They compose with each other and with your own code through [stream-chain](https://www.npmjs.com/package/stream-chain), the zero-dependency library this one is built on; TypeScript typings are bundled.
9+
10+
Why it might be for you:
11+
12+
- **Surgical.** `pick`, `ignore`, `replace`, and `filter` keep just the subobjects you want out of a massive document and drop the rest &mdash; the bytes you skip are never assembled into memory.
13+
- **Composable.** Every component is an ordinary pipeline stage. Mix them with each other, with plain functions and generators, and with any Node or Web stream.
14+
- **Performance-minded.** The parser and assemblers are measured and tuned along the hot paths. Real numbers depend on your data and hardware, so [benchmark](https://github.com/uhop/stream-json/wiki/Benchmarks) on your own.
15+
- **Solid.** ESM, bundled TypeScript typings, and a broad test suite exercised across Node, Bun, Deno, and the browser.
16+
17+
## Example
18+
19+
Pull one array out of a JSON document larger than memory and tally it &mdash; one record at a time, in constant memory:
4620

4721
```js
4822
import chain from 'stream-chain';
49-
5023
import {parser} from 'stream-json';
5124
import {pick} from 'stream-json/filters/pick.js';
52-
import {ignore} from 'stream-json/filters/ignore.js';
53-
import {streamValues} from 'stream-json/streamers/stream-values.js';
54-
25+
import {streamArray} from 'stream-json/streamers/stream-array.js';
5526
import fs from 'node:fs';
56-
import zlib from 'node:zlib';
5727

28+
// data.json: { "meta": {...}, "data": [ ...millions of records... ] }
5829
const pipeline = chain([
59-
fs.createReadStream('sample.json.gz'),
60-
zlib.createGunzip(),
30+
fs.createReadStream('data.json'), // a file far bigger than RAM is fine
6131
parser(),
62-
pick({filter: 'data'}),
63-
ignore({filter: /\b_meta\b/i}),
64-
streamValues(),
65-
data => {
66-
const value = data.value;
67-
// keep data only for the accounting department
68-
return value && value.department === 'accounting' ? data : null;
69-
}
32+
pick({filter: 'data'}), // descend into "data", ignore everything else
33+
streamArray() // emit one array element at a time
7034
]);
7135

72-
let counter = 0;
73-
pipeline.on('data', () => ++counter);
74-
pipeline.on('end', () => console.log(`The accounting department has ${counter} employees.`));
36+
const byDepartment = {};
37+
pipeline.on('data', ({value}) => {
38+
byDepartment[value.department] = (byDepartment[value.department] ?? 0) + 1;
39+
});
40+
pipeline.on('end', () => console.log(byDepartment));
7541
```
7642
77-
`stream-json` 3.x is ESM-only and requires Node.js 22+. The default Node-flavored entries (`stream-json/...`) attach both `.asStream` (Node `Duplex`) and `.asWebStream` (Web Streams pair) on every component, since modern Node and Bun support both stream flavors natively. For browser bundles, import from the `stream-json/web/...` subpath instead — it pulls no Node-stream code into the dep graph. Advanced consumers can also import from `stream-json/core/...` to get bare factories with no adapters attached. See [Migrating from 2.x to 3.x](https://github.com/uhop/stream-json/wiki/Migrating-from-2.x-to-3.x).
78-
79-
See the full documentation in [Wiki](https://github.com/uhop/stream-json/wiki).
80-
81-
Companion projects:
82-
83-
- [stream-csv-as-json](https://www.npmjs.com/package/stream-csv-as-json) streams huge CSV files in a format compatible with `stream-json`:
84-
rows as arrays of string values. If a header row is used, it can stream rows as objects with named fields.
43+
Each stage is a building block; `stream-chain` wires them into one stream and handles the streaming and backpressure. For input straight from a file you can drop `createReadStream` in favor of the Node-only [file edges](https://github.com/uhop/stream-json/wiki/parseFile) (`parseFile()`); to write a token stream back to disk, use [stringerToFile()](https://github.com/uhop/stream-json/wiki/stringerToFile). See [Recipes](https://github.com/uhop/stream-json/wiki/Recipes) for more.
8544
8645
## Installation
8746
8847
```bash
8948
npm install --save stream-json
90-
# or: yarn add stream-json
9149
```
9250
93-
## Use
51+
## What's in the box
52+
53+
- **[Parser](https://github.com/uhop/stream-json/wiki/Parser)** &mdash; the streaming JSON parser producing a SAX-like token stream (optionally packing keys, strings, and numbers). The [main module](https://github.com/uhop/stream-json/wiki/Main-module) decorates it with `emit()`.
54+
- **Filters** &mdash; edit a token stream on the fly: [pick](https://github.com/uhop/stream-json/wiki/Pick), [replace](https://github.com/uhop/stream-json/wiki/Replace), [ignore](https://github.com/uhop/stream-json/wiki/Ignore), [filter](https://github.com/uhop/stream-json/wiki/Filter).
55+
- **Streamers** &mdash; assemble tokens into JavaScript objects: [streamValues](https://github.com/uhop/stream-json/wiki/StreamValues), [streamArray](https://github.com/uhop/stream-json/wiki/StreamArray), [streamObject](https://github.com/uhop/stream-json/wiki/StreamObject).
56+
- **Essentials** &mdash; [Assembler](https://github.com/uhop/stream-json/wiki/Assembler), [Disassembler](https://github.com/uhop/stream-json/wiki/Disassembler), [Stringer](https://github.com/uhop/stream-json/wiki/Stringer), [Emitter](https://github.com/uhop/stream-json/wiki/Emitter).
57+
- **Utilities** &mdash; [emit()](<https://github.com/uhop/stream-json/wiki/emit()>), [withParser()](<https://github.com/uhop/stream-json/wiki/withParser()>), [Batch](https://github.com/uhop/stream-json/wiki/Batch), [Verifier](https://github.com/uhop/stream-json/wiki/Verifier), [FlexAssembler](https://github.com/uhop/stream-json/wiki/FlexAssembler), and Node-only file edges ([parseFile](https://github.com/uhop/stream-json/wiki/parseFile), [stringerToFile](https://github.com/uhop/stream-json/wiki/stringerToFile), [verifyFile](https://github.com/uhop/stream-json/wiki/verifyFile)).
58+
- **[JSONC](https://github.com/uhop/stream-json/wiki/jsonc-Parser)** ([JSON with Comments](https://jsonc.org/)) &mdash; streaming parser, stringer, and verifier, with faithful comma round-trip (`streamCommas` / `useCommas`).
59+
- **JSONL** ([JSON Lines](https://jsonlines.org/)) &mdash; **deprecated**: now a thin re-export of [stream-chain's JSONL](https://github.com/uhop/stream-chain/wiki/jsonl), slated for removal in a future major.
60+
- **Subpaths** &mdash; default `stream-json/...` entries attach both `.asStream` (Node `Duplex`) and `.asWebStream` (Web Streams) to every component; import from `stream-json/web/...` for browser bundles (no Node-stream code in the graph), or `stream-json/core/...` for bare factories. ESM-only, Node 22+.
61+
62+
Full documentation is in the **[wiki](https://github.com/uhop/stream-json/wiki)** &mdash; browse the [index](https://github.com/uhop/stream-json/wiki/Home), or [search it](https://uhop.github.io/wiki-search/app/?wiki=uhop/stream-json) by name.
63+
64+
## Companion projects
65+
66+
- [stream-chain](https://www.npmjs.com/package/stream-chain) &mdash; the pipeline-composition substrate `stream-json` is built on (wire functions, generators, and streams into one chain); also home to streaming [JSONL](https://github.com/uhop/stream-chain/wiki/jsonl).
67+
- [stream-csv-as-json](https://www.npmjs.com/package/stream-csv-as-json) &mdash; streams huge CSV files in a `stream-json`-compatible token format: rows as arrays of strings, or as objects when a header row is present.
9468
95-
The library is organized as small composable components based on Node.js [streams](http://nodejs.org/api/stream.html) and [events](http://nodejs.org/api/events.html). The source code is compact — read it to understand how things work and to build your own components.
69+
## License
9670
97-
Bug reports, simplifications, and new generic components are welcome — open a ticket or pull request.
71+
BSD-3-Clause
9872
9973
## Release History
10074
10175
- 3.3.0 _File I/O components (`parseFile`, `stringerToFile`, `verifyFile`), faithful JSONC comma round-trip (`streamCommas` / `useCommas`), JSONL delegated to `stream-chain`._
10276
- 3.2.0 _Improvements in TS typings, faster JSON parser._
10377
- 3.1.0 _Web Streams parity sweep._
10478
- 3.0.0 _Moved to ESM using `stream-chain` 4.x. See [Migrating from 2.x to 3.x](https://github.com/uhop/stream-json/wiki/Migrating-from-2.x-to-3.x)._
105-
- 2.1.0 _new: [jsonc/Verifier](https://github.com/uhop/stream-json/wiki/jsonc-Verifier) validates JSONC text with exact error locations. Parser performance improvements (pre-allocated token singletons)._
79+
- 2.1.0 _new: [jsonc/Verifier](https://github.com/uhop/stream-json/wiki/jsonc-Verifier) &mdash; validates JSONC text with exact error locations. Parser performance improvements (pre-allocated token singletons)._
10680
- 2.0.0 _major rewrite: functional API based on `stream-chain` 3.x, bundled TypeScript definitions. New: JSONC parser/stringer, FlexAssembler. See [Migrating from 1.x to 2.x](https://github.com/uhop/stream-json/wiki/Migrating-from-1.x-to-2.x)._
10781
10882
The full history is in the wiki: [Release history](https://github.com/uhop/stream-json/wiki/Release-history).

0 commit comments

Comments
 (0)