feat(server): support array-of-arrays parameter shape for multi-row VALUES (#117)

* feat(server): support array-of-arrays parameter shape for multi-row VALUES

Previously `substitute_sql_params` rendered a flat scalar array as the
pgvector / VECTOR text literal `[v1, v2, …]` and rejected anything more
deeply nested. That meant a pipeline could only insert one row per call
unless the YAML hardcoded a fixed number of named parameters — a
non-starter for batched WAL replay where the consumer chooses batch
size dynamically.

This patch teaches the renderer that an array whose every element is
itself an array is a multi-row tuple list:

    {"rows": [["a", [0.1, 0.2]], ["b", [0.3, 0.4]]]}

becomes `('a', [0.1, 0.2]), ('b', [0.3, 0.4])`, so a pipeline like

    INSERT INTO docs (id, embedding) VALUES {rows}

handles arbitrary batch sizes without changing its YAML. Inner arrays
inside a row tuple (the embedding cell here) render as the bracketed
scalar form so `VECTOR` and pgvector columns accept them as text
literals — same shape used by single-row inserts. Mixed-shape arrays
(some scalar, some array elements) are explicitly rejected as
Unsupported rather than silently emitting malformed SQL.

Pre-existing flat-array behaviour is preserved; the new path only
fires when *every* element is an array.

Adds 6 unit tests covering scalar-only batches, batches with a vector
cell, single-row-batch edge case, quote escaping, the
mixed-shape rejection path, and a regression test for the legacy flat
vector-literal shape.

Adds a "Parameter shapes" section to docs/pipelines.md documenting the
JSON value → SQL literal mapping including the new tuple-list form,
plus three runnable example pipelines:

  - docs/postgres/pipelines/batch_insert_users.yaml
  - docs/seekdb/pipelines/batch_insert_users.yaml
  - docs/seekdb/pipelines/batch_insert_docs_with_embeddings.yaml

Verified: cargo build -p skardi-server clean; cargo test -p
skardi-server --lib passes 109/109.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: cover Bool / Null / fallback scalar_to_sql arms

Codecov flagged 3 uncovered lines in the new helper — the Bool, Null,
and `other` fallback arms in scalar_to_sql. The existing tuple-list
tests only exercise Number, String, and nested Array cells.

Adds one focused test that puts a Bool, a Null, and a Value::Object
side-by-side in a row tuple so all three branches fire through
row_cell_to_sql's `_ => scalar_to_sql(v)` path. The Object-fallback
assertion locks in the pre-existing JSON-form behaviour as the
documented contract for unexpected shapes — a future change has to
be deliberate.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix check

* fix(sql_validator): allow `VALUES {rows}` placeholder shape at load time

The runtime renderer in `substitute_sql_params` expands an array-of-arrays
parameter into a multi-row tuple list (`(c1, c2), (c1, c2)`) so a pipeline
like `INSERT INTO docs (...) VALUES {rows}` can batch-insert without
hardcoding row count in YAML. But `preprocess_parameters` in the load-time
SQL validator always substituted `{name}` with the quoted scalar literal
`'__PARAM__'`, so the same template parsed as `VALUES '__PARAM__'` and
sqlparser rejected it with `Expected: (, found: '__PARAM__'`. Result:
skardi-server crashed during config load, the container restarted in a
loop, and any pipeline using the new tuple-list shape was unloadable.

Switch the placeholder to `(NULL)`, which parses both as a scalar
expression (`WHERE x = (NULL)`) and as a single-row VALUES tuple
(`VALUES (NULL)`), so the validator accepts every shape the runtime
renderer can emit. The validator only checks DDL/access-mode
restrictions, not types or arity, so the choice of literal is purely
about parseability.

Adds a regression test covering `VALUES {rows}` (with and without an
`ON CONFLICT` tail) and confirming access-mode enforcement still fires
on the tuple-list shape against a read-only table.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fmt

* add tests for all data sources

* add more tests

* no ci change

* ci: restore pgvector image and docs table seed for postgres tests

Reinstates the `pgvector/pgvector:pg16` image and `docs` table seed
that test_insert_multi_row_values_with_vector_cell relies on for
introspecting an `embedding vector(4)` column.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ci: free more disk space to avoid linker SIGBUS on smaller runner VMs

GitHub-hosted runners sometimes assign a 72 GB-rootfs VM (vs the 145 GB
pool), leaving only ~25 GB free after the toolchains. The
coverage-instrumented `--all-features` link step exhausts that and LLD
crashes with `ld terminated with signal 7 [Bus error]`. Remove
additional preinstalled toolchains (Android SDK, CodeQL, PowerShell,
Chromium, node_modules, vendor SDKs) and unused docker images so the
link has headroom on either VM type.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: reluctanthero104

.github/workflows/ci.yml

@@ -52,7 +52,12 @@ jobs:
           --health-retries=5
 
       postgres:
-        image: postgres:16
+        # `pgvector/pgvector:pg16` is the upstream Postgres 16 image with
+        # the pgvector extension pre-built in. Used so the `docs` table
+        # can declare an `embedding vector(N)` column and the integration
+        # tests can round-trip vector cells through the multi-row VALUES
+        # renderer without a separate extension build step.
+        image: pgvector/pgvector:pg16
         env:
           POSTGRES_DB: mydb
           POSTGRES_USER: skardi_user
@@ -126,11 +131,24 @@ jobs:
       - name: install nextest
         uses: taiki-e/install-action@nextest
       - name: Free up disk space
+        # GitHub-hosted runners ship with ~25 GB free on the smaller VM
+        # pool — not enough for the coverage-instrumented `--all-features`
+        # link step (LLD crashes with SIGBUS when the build write hits a
+        # full filesystem). Remove the bulky preinstalled toolchains and
+        # caches to give the linker headroom.
         run: |
           sudo rm -rf /usr/share/dotnet
           sudo rm -rf /opt/ghc
           sudo rm -rf /usr/local/share/boost
+          sudo rm -rf /usr/local/lib/android
+          sudo rm -rf /opt/hostedtoolcache/CodeQL
+          sudo rm -rf /usr/local/share/powershell
+          sudo rm -rf /usr/local/share/chromium
+          sudo rm -rf /usr/local/lib/node_modules
+          sudo rm -rf /opt/microsoft
+          sudo rm -rf /opt/google
           sudo rm -rf "$AGENT_TOOLSDIRECTORY"
+          sudo docker image prune -af || true
           df -h
       - uses: Swatinem/rust-cache@v2
       - name: Check code format
@@ -184,6 +202,7 @@ jobs:
       - name: Seed PostgreSQL data
         run: |
           PGPASSWORD=skardi_pass psql -h 127.0.0.1 -U skardi_user -d mydb <<'EOF'
+          CREATE EXTENSION IF NOT EXISTS vector;
           CREATE TABLE users (
               id SERIAL PRIMARY KEY,
               name VARCHAR(100) NOT NULL,
@@ -223,6 +242,16 @@ jobs:
               ('Database Query Optimization', 'database query optimization indexing performance tuning relational algebra', 'database'),
               ('Deep Learning Advances', 'machine learning classification supervised training model convolutional neural network', 'research'),
               ('Neural Network Architectures', 'deep learning neural network convolutional image recognition transformer attention', 'ai');
+          -- pgvector-backed `docs` table — exercises the multi-row VALUES
+          -- renderer's nested-array cell shape (`{"rows": [["a", "b", [v1,
+          -- v2, v3, v4]], ...]}`) end-to-end. Mirrors the SeekDB `docs`
+          -- table schema so the same parameter shape works on both.
+          CREATE TABLE docs (
+              id        TEXT PRIMARY KEY,
+              title     TEXT NOT NULL,
+              category  TEXT NOT NULL,
+              embedding vector(4) NOT NULL
+          );
           EOF
 
       - name: Seed MongoDB data