Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/actions/setup-playwright/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@ runs:

- name: Install playwright browsers if they don't exist
shell: bash
run: npx playwright install --with-deps
run: pnpm --filter @supabase/pg-parser exec playwright install --with-deps
if: steps.playwright-cache.outputs.cache-hit != 'true'

- name: Install playwright dependencies if binaries exist
shell: bash
run: npx playwright install-deps
run: pnpm --filter @supabase/pg-parser exec playwright install-deps
if: steps.playwright-cache.outputs.cache-hit == 'true'
2 changes: 1 addition & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
- name: Build libs
run: pnpm run build
- name: Get playwright version
run: echo "PLAYWRIGHT_VERSION=$(pnpm list @playwright/test | grep @playwright/test | awk '{print $2}')" >> $GITHUB_ENV
run: echo "PLAYWRIGHT_VERSION=$(pnpm --filter @supabase/pg-parser exec playwright --version | tail -1 | awk '{print $NF}')" >> $GITHUB_ENV
- name: Setup playwright
uses: ./.github/actions/setup-playwright
with:
Expand Down
96 changes: 96 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# Contributing

Guide for contributors working on `@supabase/pg-parser`.

## Overview

pg-parser compiles [libpg_query](https://github.com/pganalyze/libpg_query) (PostgreSQL's SQL parser) to WebAssembly via Emscripten. It supports PG versions 15, 16, and 17 as separate WASM binaries and exposes `parse()` and `deparse()` methods in TypeScript.

## Why Emscripten (not WASI)

WASI would be preferred for its portability and smaller runtime footprint, but PostgreSQL's parser uses `setjmp`/`longjmp` for error handling (its `PG_TRY`/`PG_CATCH` mechanism). This requires stack unwinding/rewinding, which WASI has no support for. Emscripten does — it rewrites `setjmp`/`longjmp` into JavaScript exception handling at compile time, which is the main reason we depend on it.

## Architecture

Three layers: **TypeScript API** → **C bindings** (compiled to WASM) → **libpg_query** + **protobuf-JSON bridge**.

We use `pg_query_parse_protobuf` / `pg_query_deparse_protobuf` (the protobuf API, not the JSON one). A protobuf-JSON bridge converts between protobuf and the JSON AST that TypeScript consumers work with.

### Parse Flow

```
SQL string
→ [libpg_query] pg_query_parse_protobuf() → protobuf binary
→ [protobuf-c] unpack → C message structs
→ [protobuf2json] protobuf2json_string() → JSON string
→ [TypeScript] JSON.parse() → ParseResult<Version>
```

### Deparse Flow

```
ParseResult<Version>
→ [TypeScript] JSON.stringify() → JSON string
→ [protobuf2json] json2protobuf_string() → C message structs
→ [protobuf-c] pack → protobuf binary
→ [libpg_query] pg_query_deparse_protobuf() → SQL string
```

### Memory

All WASM memory is managed explicitly. The TypeScript side follows the same pattern for both flows:

1. Encode input string → `_malloc` → copy to WASM heap → call C export → `_free` the input
2. Read result from struct pointers on the heap
3. Free the result struct in a `finally` block (`_free_parse_result` / `_free_deparse_result`)

**Multi-byte string gotcha:** always use `TextEncoder.encode().length` for byte length, never `string.length` (UTF-16 code units).

### Error Handling

- `ParseError` — has `message`, `type` (`'syntax'` | `'semantic'` | `'unknown'`), and `position` (0-based offset into the SQL string)
- `DeparseError` — has `message` only (no position or type, since errors occur on the AST)

### The Protobuf-JSON Bridge

The protobuf-JSON conversion happens in C (inside WASM) rather than in JavaScript. A JS-based protobuf library (e.g. protobuf.js) would add significant bundle size, and libpg_query already vendors protobuf-c for its own serialization — so we reuse that and just add a thin JSON layer on top.

It was adapted from [protobuf2json-c](https://github.com/Sannis/protobuf2json-c) with changes for proto3 semantics (the original was proto2-only). Uses a [forked protobuf-c](https://github.com/gregnr/protobuf-c/tree/feat/json_name) that adds `json_name` descriptor support, required because libpg_query's `.proto` uses `json_name` annotations for node names.

## Development

### Prerequisites

- [Docker](https://www.docker.com/) (for the Emscripten toolchain)
- [pnpm](https://pnpm.io/) (v10+)
- Node.js 18+

### Building

```bash
pnpm install

# Build everything (WASM for all PG versions + JS bundle)
pnpm build

# Build a single PG version's WASM
pnpm --filter @supabase/pg-parser make:17 build

# Rebuild JS only (after WASM is already built)
pnpm --filter @supabase/pg-parser build:js
```

The WASM build runs inside Docker via `docker compose run --rm emsdk emmake make`. Most of the build logic lives in `packages/pg-parser/Makefile` — vendoring libpg_query and jansson, patching protobuf-c for `json_name` support, compiling the C bindings, and linking the final WASM binary. Vendor dependencies are cloned on first build.

### Testing

```bash
# All tests (Node, Vercel Edge, Chromium, WebKit, Firefox)
pnpm test

# Node only
pnpm --filter @supabase/pg-parser test:unit:node

# Browser only
pnpm --filter @supabase/pg-parser test:unit:browser
```
136 changes: 131 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ Postgres SQL parser that can run anywhere (Browser, Node.js, Deno, Bun, etc.).

## Features

- **AST:** Parses Postgres SQL queries into an abstract syntax tree (AST)
- **Parse:** Parses Postgres SQL queries into an abstract syntax tree (AST)
- **Deparse:** Converts an AST back into a SQL string
- **Accurate:** Uses real Postgres C code compiled to WASM
- **Multi-version:** Supports multiple Postgres versions at runtime (15, 16, 17)
- **Multi-runtime:** Works on any modern JavaScript runtime (Browser, Node.js, Deno, Bun, etc.)
Expand Down Expand Up @@ -32,19 +33,40 @@ pnpm add @supabase/pg-parser

## Usage

### Parse SQL to AST

```typescript
import { PgParser } from '@supabase/pg-parser';

const parser = new PgParser(); // Defaults to latest version (17)

// Parse a SQL query
const { tree } = await parser.parse('SELECT * FROM users WHERE id = 1');

console.log(tree);

// { version: 170004, stmts: [ ... ] }
```

### Deparse AST to SQL

```typescript
import { PgParser } from '@supabase/pg-parser';

const parser = new PgParser();

// Parse SQL into an AST
const { tree } = await parser.parse('SELECT * FROM users WHERE id = 1');

// { version: 170004, stmts: [ ... ] }

// Deparse the AST back into SQL
const { sql } = await parser.deparse(tree);

console.log(sql);

// SELECT * FROM users WHERE id = 1
```

## API

### `PgParser` class
Expand Down Expand Up @@ -99,6 +121,58 @@ const tree = await unwrapParseResult(parser.parse(sql)); // Throws an error if t
console.log('Parsed AST:', tree);
```

### `deparse()` method

To convert an AST back into a SQL string, use the `deparse()` method:

```typescript
const { tree } = await parser.parse('SELECT * FROM users WHERE id = 1');
const result = await parser.deparse(tree);
```

This returns a `WrappedDeparseResult` object. If the deparse was successful, `WrappedDeparseResult` will contain a `sql` property containing the reconstructed SQL string.

If the deparse failed, `WrappedDeparseResult` will contain an `error` property with the error message.

Use the `error` property to check if the deparse was successful:

```typescript
if (result.error) {
console.error('Deparse error:', result.error);
} else {
console.log('SQL:', result.sql);
}
```

TypeScript will correctly narrow the type of `result` based on whether there was an error or not.

If you prefer throwing an error instead of returning a result object, you can wrap `deparse()` in the `unwrapDeparseResult()` helper (see [Utility functions](#utility-functions)).

#### Modifying the AST

One of the most useful applications of deparse is modifying SQL programmatically. You can parse a query, modify the AST, and then deparse it back into SQL:

```typescript
import { PgParser, unwrapNode } from '@supabase/pg-parser';

const parser = new PgParser();

// Parse the original query
const { tree } = await parser.parse('SELECT 1 + 1');

// Modify the AST: add an alias to the expression
const { node: selectStmt } = unwrapNode(tree.stmts[0].stmt);
const { node: resTarget } = unwrapNode(selectStmt.targetList[0]);
resTarget.name = 'total';

// Deparse the modified AST back into SQL
const { sql } = await parser.deparse(tree);

console.log(sql);

// SELECT 1 + 1 AS total
```

### `tree` object

The `tree` AST is a JavaScript object that represents the structure of the SQL query.
Expand Down Expand Up @@ -208,7 +282,7 @@ const supportedVersions = getSupportedVersions();
console.log(supportedVersions); // [15, 16, 17]
```

### `error` object
### Parse `error` object

If the parse fails, `PgParser` will return an `error` of type `ParseError` with the following properties:

Expand All @@ -231,9 +305,42 @@ If the parse fails, `PgParser` will return an `error` of type `ParseError` with

**Note:** This is relative to the entire SQL string, not just the statement being parsed or line numbers within a statement. If you are parsing a multi-statement query, the position will be relative to the entire query string, where newlines are counted as single characters.

### Deparse `error` object

If the deparse fails, `PgParser` will return an `error` of type `DeparseError` with the following property:

- `message`: A human-readable error message describing what went wrong.

Unlike `ParseError`, deparse errors don't have a `position` or `type` since they operate on an AST rather than a SQL string. Deparse errors typically occur when the AST contains invalid structure, such as a wrong type for a field (e.g. passing a string where an array is expected).

### Utility functions

The following utility functions are available to help with parsing:
The following utility functions are available:

#### `unwrapParseResult()`

Unwraps a `WrappedParseResult` by throwing an error if the result contains an `error`, or otherwise returning the parsed `tree`. Supports both synchronous and asynchronous results.

```typescript
import { PgParser, unwrapParseResult } from '@supabase/pg-parser';
const parser = new PgParser();
const tree = await unwrapParseResult(parser.parse('SELECT 1'));
```

#### `unwrapDeparseResult()`

Unwraps a `WrappedDeparseResult` by throwing an error if the result contains an `error`, or otherwise returning the deparsed SQL string. Supports both synchronous and asynchronous results.

```typescript
import {
PgParser,
unwrapParseResult,
unwrapDeparseResult,
} from '@supabase/pg-parser';
const parser = new PgParser();
const tree = await unwrapParseResult(parser.parse('SELECT 1'));
const sql = await unwrapDeparseResult(parser.deparse(tree));
```

#### `unwrapNode()`

Expand Down Expand Up @@ -317,9 +424,28 @@ switch (type) {
}
```

## Bundle size

WASM binaries are lazy-loaded — only fetched when you construct a `PgParser`, and only for the version you request. The JS bundle itself is **~3 KB compressed**.

Each Postgres version ships as a separate `.wasm` file. Most CDNs and hosting providers serve WASM with brotli compression by default (gzip as fallback), so transfer size is what matters in practice.

| | Raw | Brotli | Gzip |
| ------------------- | ------ | ----------- | ----------- |
| JS bundle | 9 KB | **~3 KB** | **~3 KB** |
| Emscripten loader | 52 KB | **~16 KB** | **~18 KB** |
| WASM binary (PG 15) | 1.4 MB | **~231 KB** | **~303 KB** |
| WASM binary (PG 16) | 1.5 MB | **~241 KB** | **~318 KB** |
| WASM binary (PG 17) | 1.7 MB | **~254 KB** | **~341 KB** |

The WASM binary is dominated by Postgres's LALR parser tables (~583 KB raw) and protobuf descriptors for ~200 AST node types (~160 KB raw). These compress well (~80%) because they're highly repetitive integer arrays. The parser tables are a direct property of Postgres's grammar - any parser (in any language) that fully supports Postgres syntax will carry similar overhead.

For context, the compressed transfer size sits between three.js (~150 KB gzip) and sql.js/SQLite (~450 KB gzip).

## Roadmap

- [ ] Deparse SQL queries (AST -> SQL)
- [x] Parse SQL queries (SQL -> AST)
- [x] Deparse SQL queries (AST -> SQL)
- [ ] Expose Postgres scanner (lexer)
- [ ] Version compatibility checks

Expand Down
4 changes: 4 additions & 0 deletions packages/pg-parser/.clang-format
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
BasedOnStyle: Google
IndentWidth: 2
ColumnLimit: 0
AllowShortIfStatementsOnASingleLine: false
Loading