feat: end-to-end report rendering - data/layout/Page-IR pipeline with HTML/SVG/PDF/PNG backends, chart & cross-tab drawing, formula evaluator, live-DB and saved-data sources, byte-faithful .rpt writer, and a render-parity test corpus

Author: MrSrsen

.dockerignore

@@ -6,8 +6,6 @@ target/
 # Sensitive / local-only — must never enter the build context or image.
 samples/
 research/
-dump/
-caches/
 .claude/
 CLAUDE.md
 

.github/workflows/ci.yml

@@ -23,8 +23,11 @@ jobs:
         run: cargo fmt --all --check
       - name: Clippy
         run: cargo clippy --all-targets --all-features -- -D warnings
-      - name: Build (all features)
-        run: cargo build --all-features
+      # --all-targets compiles every binary (rpt, rpt-render), example
+      # (record_bytes), and test/bench target, so a broken build in any of them
+      # fails CI here rather than only when that target is exercised.
+      - name: Build (all features, all targets)
+        run: cargo build --all-features --all-targets
 
   test:
     runs-on: ubuntu-latest
@@ -36,6 +39,34 @@ jobs:
       - name: Test
         run: cargo test --all-features
 
+  # Render-parity corpus: seed the committed SQL fixtures into a real PostgreSQL (the single DB
+  # technology for render testing) and diff the rendered HTML against the committed baselines. The
+  # test skips without RPT_DB_URL, so it only runs where a server is provided — here, and locally via
+  # `docker compose up -d --wait` (see docker-compose.yml / Makefile).
+  render-fixtures:
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:16-alpine
+        env:
+          POSTGRES_USER: rpt
+          POSTGRES_PASSWORD: rpt
+          POSTGRES_DB: rptfixtures
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U rpt -d rptfixtures"
+          --health-interval 2s
+          --health-timeout 3s
+          --health-retries 15
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+      - name: Render fixtures against PostgreSQL
+        env:
+          RPT_DB_URL: postgres://rpt:rpt@localhost:5432/rptfixtures
+        run: cargo test -p rpt-render --test postgres_fixtures
+
   msrv:
     runs-on: ubuntu-latest
     steps:
@@ -73,7 +104,7 @@ jobs:
       - name: XML baseline regression
         env:
           RPT_REQUIRE_SANDBOX: "1"
-        run: cargo test -p rpt-to-xml --test baseline
+        run: cargo test -p rpt-cli --test baseline
 
   # Create the GitHub Release once, before the matrix upload jobs. The upload
   # action only uploads assets to an existing release; if the per-target jobs
@@ -131,45 +162,9 @@ jobs:
       - name: Build and upload binaries
         uses: taiki-e/upload-rust-binary-action@v1
         with:
-          bin: rpt,rpt-to-xml
+          bin: rpt,rpt-render
           target: ${{ matrix.target }}
           archive: rpt-rs-$tag-$target
           include: README.md,LICENSE
           checksum: sha256
           token: ${{ secrets.GITHUB_TOKEN }}
-
-  # Publish the Docker image to the GitHub Container Registry, tagged with the
-  # release version and `latest`. Same gating as the release stage.
-  docker:
-    needs: [check, test, msrv, feature-matrix, baseline]
-    if: startsWith(github.ref, 'refs/tags/v')
-    runs-on: ubuntu-latest
-    permissions:
-      contents: read
-      packages: write
-    steps:
-      - uses: actions/checkout@v4
-      - uses: docker/setup-buildx-action@v3
-      - name: Log in to GHCR
-        uses: docker/login-action@v3
-        with:
-          registry: ghcr.io
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
-      - name: Image tags and labels
-        id: meta
-        uses: docker/metadata-action@v5
-        with:
-          images: ghcr.io/${{ github.repository }}
-          tags: |
-            type=semver,pattern={{version}}
-            type=raw,value=latest
-      - name: Build and push
-        uses: docker/build-push-action@v6
-        with:
-          context: .
-          push: true
-          tags: ${{ steps.meta.outputs.tags }}
-          labels: ${{ steps.meta.outputs.labels }}
-          cache-from: type=gha
-          cache-to: type=gha,mode=max

.gitignore

@@ -1,4 +1,5 @@
-.idea
+**/.idea/
+**/.vs/
 
 # Sensitive or work-in-progress staff that should not be committed
 /samples/
@@ -12,3 +13,15 @@
 /CLAUDE.md
 /.claude/
 
+# Beads will be ignored until project is not stabilized
+.beads/
+render-out/
+
+# Example programs are dev/scratch tools — kept on disk for `cargo run --example`, not tracked.
+**/examples/
+
+# Private-report SQL fixtures carry real production schema/table/column names — never commit.
+/tests/fixtures/sql/private/
+
+# Private-report HTML baselines (real schema-derived render) — never commit.
+/tests/fixtures/baselines/html/private/

CHANGELOG.md

@@ -6,6 +6,120 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 ## [Unreleased]
 
+This release turns `rpt-rs` from a reader/exporter into a full reporting engine: a complete render pipeline
+(data → layout → Page IR → HTML/SVG/PDF/PNG), a formula evaluator, live-database support, saved-data decoding,
+a byte-faithful writer, and the test corpus to validate it all against the native Crystal engine.
+
+### Added
+
+- **End-to-end report rendering.** A new render & data pipeline built purely on the decoded model:
+  - **`rpt-data`** — the record pipeline: `RowSource` → record selection → sort → grouping → summaries → running
+    totals, with the formula evaluation context (`Global`/`Shared` variables, per-record cache, evaluation-time
+    scheduling). Two-field summaries (`WeightedAverage`, `Correlation`, `Covariance`) resolve to an
+    empty/unavailable value rather than a plausible-but-wrong one until the second field is decoded.
+  - **`rpt-layout`** — the layout & pagination engine: places every object at its twip position, paginates
+    band-by-band, and honours the section-break controls (New Page Before/After, Keep Group Together, Print at
+    Bottom of Page, Reset Page Number After, Underlay Following Sections). Resolves each field's display format
+    from the locale + its stored format spec.
+  - **`rpt-pages`** — the backend-agnostic Page IR: `Rect` / `Ellipse` / `Line` / `Text` / `Polygon` / `Image`
+    draw-ops in twips, solid/gradient/hatch fills, rotated text runs, image assets, checkpoints, and fidelity
+    diagnostics. `serde`-serializable — the frozen contract between layout and backends.
+  - **Four output backends** — `rpt-render-html` (self-contained XHTML, images inlined), `rpt-render-svg` (one file
+    per page), `rpt-render-pdf` (krilla with real font-subset embedding, plus a dependency-free fallback writer),
+    and `rpt-render-raster` (tiny-skia → PNG per page).
+  - **`rpt-text`** — the real text stack (cosmic-text): font metrics, Unicode/CJK line breaking, bidi, and font
+    fallback behind a swappable `TextLayout` trait, with bundled Liberation fonts and a dependency-free
+    `ApproxLayout` for deterministic output.
+  - **`rpt-render`** — the orchestration facade: `ReportDocument` (load → inspect → `to_pdf`/`to_html`/…) and an
+    options-driven `render_with(report, RenderOptions)` threading the datasource (`Saved`/`Rows`/`Dataset`),
+    parameters, locale, and subreport scope, with typed `RenderError`s.
+  - **The `rpt-render` CLI** — renders a report to HTML / PDF / SVG / PNG from its saved data or a live database
+    (`--db`), with `--param`, `--locale`, format inference from the output extension, and stdout piping.
+  - The pipeline up to the Page IR plus the HTML and SVG backends compile to **WebAssembly**; a CI job guards the
+    boundary.
+
+- **Chart rendering — 16 chart types as native vector draw-ops** (no rasterization): bar (clustered / stacked /
+  percent, multi-series), line, area, pie, doughnut, 3-D riser, 3-D surface, 3-D area, scatter, bubble, stock
+  (hi-lo / OHLC), histogram, radar, gauge, funnel, and numeric-axis, plus a bar fallback (with a diagnostic) for
+  types without a dedicated renderer. Matches the native engine's defaults: the full 20-colour palette, axis
+  titles, tick density, compact temporal category labels with period bucketing (weekly/monthly/quarterly/…), label
+  thinning on dense axes, family-dependent legend rules, and a perspective 3-D scene (corner room, floor grid,
+  painter-sorted risers). Per-axis gridline modes are decoded from the chart styling record.
+
+- **Cross-tab rendering.** A cross-tab pivots the dataset by row × column dimensions with an aggregate measure per
+  cell and draws a native grid (cell rects, grid lines, headers, grand totals). Current cut: one row dimension ×
+  one column dimension × the first measure.
+
+- **Live database support.** `rpt-query` generates the joined `SELECT` from the report's table/link graph —
+  projecting only the tables and columns the report actually uses (matching the native engine's SQL and avoiding
+  accidental cartesian joins) and pushing the translatable record-selection subset into `WHERE`. `rpt-db-postgres`
+  and `rpt-db-sqlite` implement `RowSource` (native-only, isolated behind the trait so the portable core links no
+  driver). Connection URLs are read only from the environment (`RPT_DB_URL`, `DATABASE_URL`,
+  `RPT_DB_URL_<SERVER>`), never from flags.
+
+- **Formula evaluation.** The `crystal-formula` engine gained a full evaluator: a bytecode VM (with a tree-walking
+  reference evaluator), the builtin library across every family — string, math, conversion, date/time (incl.
+  `DatePart` week-numbering modes), financial (`Pmt`/`FV`/`PV`/`NPV`/`IRR`/`Rate`/`DDB`/`SLN`/`SYD`), statistical
+  (sample + population), and numeral (`ToWords`, `Roman`) — plus loop `Exit`, textual `#Month d, yyyy#` date
+  literals, and a **semantic validation pass** (`validate`/`validate_str`): unknown/misspelled builtins with
+  suggestions, arity and operator-type errors, and unknown field/parameter/formula references, as spanned,
+  severity-tagged diagnostics.
+
+- **Saved-data decoding.** A report saved with data now yields its schema (column names + types) and cached rows
+  (`Report::saved_data`, the `rpt saved` subcommand) — both the external-memo and inline packed batch layouts
+  decode, including the per-batch encryption and the memo heap.
+
+- **`.rpt` writer (byte-faithful).** The decode pipeline is invertible at the record-substrate level:
+  `Rpt::reencode` re-serializes, deflates, encrypts, and rewrites a valid `.rpt` that re-opens byte-identically at
+  the logical level, and `Rpt::patch_record_leaf` overwrites a same-size region of a decoded record's leaf. Exposed
+  as the `rpt reencode` / `rpt patch` subcommands. There is no model→records lowering yet.
+
+- **New decoders in the reader:** charts (bindings, analytic layout, data-value labels, gridline modes),
+  cross-tabs (dimensions, grid formats), object hyperlinks, hierarchical grouping, formula variables
+  (name/type/scope), typed field sub-formats (number / currency / date / time / boolean / string masks), subreport
+  re-import metadata, save metadata, designer state (rulers, guidelines), and the field-manager census. Every
+  record type observed in the corpus is now named in the registry. `rpt` decode errors carry structured context
+  (stream, byte offset, record type) via dedicated error types instead of message strings.
+
+- **CLI additions.** `rpt inspect` shows each chart's data binding ("show value(s)" / "on change of");
+  `rpt tree` prints the colorized decoded record DOM; `rpt streams` reports per-stream decode coverage;
+  `rpt dump` is a byte-layout workbench (annotated hex, string scan, scalar probe, minimal-pair diff);
+  `rpt saved` prints the decoded saved rows; `rpt sql` lists every SQL a report can run against its database
+  (the generated join query + stored SQL Commands + SQL Expression fields, recursively through subreports, with
+  connection/table provenance and a `--dialect` selector); `Report::objects()` / `objects_mut()` iterate all
+  report objects.
+
+- **Render-parity test corpus and infrastructure.** A committed corpus of 36 reports over one synthetic
+  "parking" database, each with an XML decode baseline and an HTML render baseline, validated out-of-band against
+  the native Crystal engine; golden-file tests for the Page IR and every backend; `docker-compose.yml` and a
+  `Makefile` for the fixture database; DB-gated CI regression tests.
+
+- **Documentation.** New guides: rendering (`docs/11-rendering.md`), a compile-verified render cookbook
+  (`docs/12-render-examples.md`), and the five-part formula-engine set (architecture/VM, language reference,
+  builtins, validation). GitHub-native Mermaid diagrams throughout, a rewritten README (status section, quick
+  start, example render), and a docs↔code audit bringing the block catalog, support matrix, and saved-data docs
+  in line with the code.
+
+### Changed
+
+- **Workspace restructured into a 20-crate, two-layer workspace** with compiler-enforced boundaries:
+  - The semantic model moved out of `rpt` into the standalone, pure-data **`rpt-model`** crate (no I/O, WASM-safe,
+    optional `serde`); `rpt` re-exports it as `rpt::model`. The whole render/data pipeline depends on `rpt-model`,
+    not the decoder, so the render stack links no CFB/inflate. Byte-level provenance notes live in the
+    documentation-only `rpt::provenance` module.
+  - The formula language moved into the standalone **`crystal-formula`** crate (depends only on
+    `rpt-format-value`), reusable without the binary reader (LSP, WASM sandbox, validator).
+  - The `rpt-engine` crate was dissolved (its derived analytics now live in `rpt-cli`'s private
+    `export::analysis`), and the `rpt-to-xml` binary was folded into `rpt-cli` as the **`rpt xml-dump`**
+    subcommand — one `rpt` binary for all inspection and export. XML output is byte-identical.
+- **Naming:** standalone "SAP" was dropped in favor of "Crystal Reports" across the README, docs, CLI help, and
+  crate metadata.
+
+### Removed
+
+- The standalone **`rpt-to-xml`** binary (now `rpt xml-dump`) and the **`rpt-engine`** crate (dissolved into
+  `rpt-cli` / `crystal-formula`), per the restructuring above.
+
 ## [0.1.0]
 
 ### Added

CONTRIBUTING.md

@@ -17,17 +17,16 @@ The workspace forbids `unsafe` code (`unsafe_code = "forbid"`); keep contributio
 
 ## Workspace layout
 
-rpt-rs is a four-crate workspace, and the boundary between the crates is intentional:
+The reader crates, and the boundary between them, are intentional:
 
 | Crate        | Role                                                                                          |
 | ------------ | --------------------------------------------------------------------------------------------- |
 | `rpt`        | Pure I/O. Decodes the **stored** facts from the bytes: container → records → typed model.     |
-| `rpt-engine` | **Derived** analytics computed on top of the model (e.g. field use counts, formula analysis). |
-| `rpt-to-xml` | Exports a report to a structured XML document.                                                |
-| `rpt-cli`    | The `rpt` inspection CLI.                                                                     |
+| `crystal-formula` | The standalone Crystal/Basic formula-language engine (no `.rpt` coupling).                 |
+| `rpt-cli`    | The `rpt` inspection CLI; its `xml-dump` subcommand exports a report to a structured XML document and owns the **derived** analytics (`export::analysis`: field use counts, summaries, formula analysis). |
 
 **The boundary is load-bearing.** If a value is read directly from the file, decode it in `rpt`. If a value is _computed
-or inferred_ (not present in the bytes), it belongs in `rpt-engine` — not as a stored field on an `rpt` model struct.
+or inferred_ (not present in the bytes), it belongs in the derive layer (`rpt-cli`'s `export::analysis`, the XML exporter) — not as a stored field on an `rpt` model struct.
 Keeping I/O separate from derivation is the core design rule.
 
 ## The `.rpt` format
@@ -43,21 +42,21 @@ Keeping I/O separate from derivation is the core design rule.
 
 ## Tests and XML baselines
 
-The regression suite in `crates/rpt-to-xml/tests/baseline.rs` exports a set of public sample reports (under
+The regression suite in `crates/rpt-cli/tests/baseline.rs` exports a set of public sample reports (under
 `tests/fixtures/`) to XML and compares the output against committed baselines. To keep results deterministic across
 machines, the exporter runs inside a [Bubblewrap](https://github.com/containers/bubblewrap) sandbox with the report
 bind-mounted at a fixed path, so path-derived attributes are identical everywhere.
 
 Run it (requires `bwrap` on Linux):
 
 ```sh
-cargo test -p rpt-to-xml --test baseline
+cargo test -p rpt-cli --test baseline
 ```
 
 When a change _intentionally_ alters the XML output, regenerate the baselines and review the diff:
 
 ```sh
-RPT_BLESS=1 cargo test -p rpt-to-xml --test baseline
+RPT_BLESS=1 cargo test -p rpt-cli --test baseline
 ```
 
 Only commit baseline changes you can explain. A mismatch prints a git-style unified diff showing exactly which lines