feat(gooddata-eval): generate eval datasets from a workspace's insights

Hand-authoring eval questions means writing a question and then guessing the
metric, dimension and filter it should produce, which is how a dataset ends up
full of questions the data model cannot answer.

`gd-eval generate` inverts that. It reads the charts a customer already built
via the declarative analytics model, translates each visible insight's buckets,
sorts and filters into an `expected_output.visualization` spec, and only then
asks an LLM to write the analyst question that chart answers. The expected
output is copied out of a live object rather than invented, so every question
is answerable in the real LDM by construction and the LLM only writes English.

Anything inexpressible is skipped with a printed reason rather than
approximated, and a question that contradicts its own spec is a hard error:
ranking words require a real sort, filter words a real filter, a breakdown
clause a non-empty view_by. One rewrite is attempted, then the item is dropped.

`--enrich-ranked N` derives ranked items, because analysts sort in Analytical
Designer and save without persisting the sort, leaving that coverage near zero
on real models. Adding a limit to a definition that already executes cannot
make it unanswerable. The budget goes to the best-grounded first: insights
whose own title promised a ranking their definition never implemented, then
ranking filters added to a plain breakdown. Derived items carry `derived_from`
and `derived_basis` so a pass rate over them stays separable.

Only a ranking filter is derived, never a sort: on the same base a ranking is
the stronger item, since "the top 3 X by Y" has one correct spec while "X
sorted by Y" leaves the direction to the reader.

The sort a question asks for is now graded, which it previously was not.
`sort_by` was written into every fixture and read by nobody -- the evaluator
loads `expected_output.visualization` into `CreatedVisualization`, which
declared no such field and is configured `extra="ignore"`, so pydantic
discarded it and `strict_pass` covered cross-references, metrics, dimensions,
filters and chart type only. A question saying "sorted by Order id ascending"
-- seven of forty on one real workspace -- asked for something no check saw.

`AacQuery` gains `sort_by`, `check_sorts` compares it, and `strict_pass` counts
it. Entries stay raw dicts for the same reason `filter_by` does: the agent adds
keys this does not read, and a typed model would reject a chart that is
otherwise correct. The comparison uses the shape the agent emits, taken from
recorded runs: `{type: metric_sort, direction, metrics: [alias]}` and `{type:
attribute_sort, direction, by: alias}`, with one build sending both `by` and
`metrics` on a metric sort -- so the entry's own `type` decides which key names
the fields, never whichever key is present. Aliases resolve to uris and date
granularities fold to one spelling, as filters already do, and order is
significant: sorted by region then revenue is not sorted by revenue then
region.

The check is deliberately not symmetric with the filter ones. An empty
`sort_by` records that the fixture has no sort, not that the chart must be
unsorted -- a generated item inherits that emptiness from an insight whose
author sorted in Analytical Designer and saved without the sort sticking. A
spurious filter changes which rows a reader sees and is always wrong, while a
volunteered sort changes only their order, and ascending on a time axis is what
any renderer picks unprompted. So a required sort is enforced and a volunteered
one is free; the cost is that a wrong sort over an unsorted fixture goes
ungraded, the lesser error while `[]` cannot distinguish "unsorted" from
"unrecorded".

A tiebreak the agent appends after the recorded sorts is free as well: "state
descending" is satisfied by "state descending, then city", so the recorded
sorts must lead and match in order, and anything after them is not compared.

Grading a previously ungraded dimension means items that passed while omitting
a sort their fixture records now fail. Re-baseline before comparing a run
against an older one.

The declarative-to-AAC mapping is not ours. `convert()` calls the platform's
own `declarative_visualization_to_aac()` (gooddata-code-convertors, via
gooddata-sdk): both definitions the evaluator compares are platform output, so
the platform's conversion is the right owner, and on one production workspace
it converts every insight where the hand-written mapping had covered 44 of 62.
What stays ours is deciding what the evaluator cannot yet score -- derived
measures, measure-level filters, a repeater's label among its metrics -- each
skipped with a printed reason, and stripping the no-op filters AD saves for an
"All" selection, which would otherwise let a question claim a filter its chart
lacks. A map's `location` bucket is skipped on purpose: it holds a rendering
label, and a question built from it reads as "broken down by City pushpin
latitude". Chart type names are the convertor's, which are also the agent's.
One granularity is patched: the convertor maps `GDC.time.week_us` to `WEEK_US`,
which is not a platform enum; the SDK's own table says `WEEK`.

Two classes of question are unwinnable however well the agent behaves, and both
are reported: a name the model carries more than once (one workspace has six
labels titled "Product Title"), droppable with `--skip-ambiguous`; and a date
granularity's cyclical twin, since MONTH walks consecutive months while
MONTH_OF_YEAR stacks every January. Granularities move to `core/granularity.py`,
shared with scoring, which also folds `attribute/x.month` and `label/x.month`
to one uri -- a date dataset exposes each granularity as an attribute whose
only label carries the same id, and comparing the raw strings failed a chart
that was correct.

Alias resolution and the uri-to-title fallback come from `core/scoring.py`:
one copy, which is the one the evaluator scores against.

The package's `AGENTS.md` gains a section running the whole pipeline --
generate, run, report, models -- with the connection precedence, the snapshot
loop that makes generation iterable offline, and the environment variables each
subcommand reads. `generate` and `report` were not mentioned there at all.

`openai` joins the package's `dev` dependency group, so a plain `uv run` in a
fresh clone has the phrasing step and the LLM judge without naming the extra.
It stays under `optional-dependencies` for anyone installing from PyPI, and
every import site is still guarded or deferred.

Three fixes from the first live run over a customer workspace. A ranking filter
with no `attribute` ranks the full dimension tuple, so the "top N <dimension>"
shorthand in the writer's brief is only true with one dimension; with two it
told the writer a within-group scope the filter lacks, and the agent built what
the question said. `CreatedVisualization.id` is optional: the agent sometimes
omits it, nothing scores on it, and a required field turned a scorable chart
into an errored item.

jira: AIS-48
risk: high
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hbj7SaGm6ucimov4NeMwqt

Author: xvalovic

.gitignore

@@ -44,3 +44,6 @@ packages/gooddata-sdk/tests/export/exports/default/
 /packages/gooddata-eval/datasets/
 # MCP tool logs, written to a relative path by whatever is started from the repo root
 /logs/
+
+# editor swap files
+*.swp

packages/gooddata-eval/AGENTS.md

@@ -9,7 +9,7 @@ experiment. The newest and most actively developed package in the repo.
 
 ## Owns
 
-- The `gd-eval` CLI (`gd-eval run`, `gd-eval models`)
+- The `gd-eval` CLI (`generate`, `run`, `report`, `models`)
 - Dataset loading and the evaluation run loop
 - Per-capability evaluators and their scoring
 - Result reporting, and pushing experiments, scores and trace links to Langfuse
@@ -27,7 +27,7 @@ experiment. The newest and most actively developed package in the repo.
 | `core/agentic/` | multi-turn agentic evaluation per capability, **plus** all Langfuse trace polling and linking (`_langfuse.py`, `_trace_linker.py`) |
 | `core/chat/` | SSE client for the agent's streaming chat endpoint |
 | `core/summary/` | HTTP client for the dedicated dashboard-summary endpoint — a single-shot chat backend, not reporting |
-| `core/dataset/` | dataset format and loading |
+| `core/dataset/` | dataset format, loading, and `from_insights.py` — dataset generation from a workspace's real insights |
 | `core/evaluators/` | single-shot evaluators and their registry |
 | `core/langfuse/` | the whole Langfuse v4 client: `_env` (base URL + credentials), `otlp` (OTLP/JSON encoding), `experiment` (root-span construction, score targets), `observations` (trace reads), `client` (httpx calls), `sink` (single-shot results as experiments) |
 | `core/reporting/` | console and JSON output rendering |
@@ -61,6 +61,107 @@ its own shape. `test_kind` on the item is what labels the result, not the evalua
 which is why `knowledge_question` can reuse `GeneralQuestionEvaluator` verbatim.
 `dashboard_summary` items additionally need `summary_input`.
 
+## Running the pipeline
+
+Four subcommands, in the order you use them. Everything runs through `uv`; never a bare
+`python`. There is no build step -- `uv run` syncs the environment from `uv.lock` on first
+use, so a fresh clone needs nothing but:
+
+```bash
+uv run --package gooddata-eval gd-eval <subcommand> --help
+```
+
+`openai` is an optional extra (`llm-judge`) so the published package stays installable
+without it, but the `dev` dependency group pulls it in, which is why a plain `uv run` here
+has the phrasing step and the LLM judge. Installing `gooddata-eval` from PyPI does not --
+there the extra is explicit, and every `openai` import site is guarded or deferred.
+
+Connection is the same for every subcommand that talks to the platform: `--host` +
+`--token`, or `GOODDATA_TOKEN` in the environment, or `--profile <name>` reading
+`~/.gooddata/profiles.yaml`. Precedence is flags > env > profile.
+
+### 1. `generate` — build a dataset from a workspace
+
+Reverse-engineers `visualization` items out of the charts a workspace already has, so the
+expected output is copied from a live object rather than invented. Needs `OPENAI_API_KEY`
+for the phrasing step, or `--no-phrase` to emit mechanical `Show <title>` questions.
+
+```bash
+uv run --package gooddata-eval gd-eval generate \
+  --host "$GOODDATA_HOST" --workspace "$WORKSPACE_ID" \
+  --dataset-name ecommerce --out ./datasets/ecommerce \
+  --snapshot-out /tmp/ws.json \
+  --phrase-model gpt-4o --enrich-ranked 5 --skip-ambiguous
+```
+
+Iterate offline instead of re-fetching: `--snapshot-out` writes everything the generator
+read as one JSON file, and `--snapshot-in` replays it with no host, token or network. Add
+`--dry-run` to print the shape counts and each spec's brief without writing anything —
+the fastest way to see what a workspace yields.
+
+Quality gates fail the command (exit 1) below `--min-questions` (15), `--min-shapes` (3)
+or `--min-filtered` (1). Lower them for a smoke test; do not lower them to ship a dataset.
+`--langfuse-out` additionally writes a Langfuse-importable file, and `--id-prefix` rewrites
+ids on that export only, because Langfuse item ids are unique per project.
+
+Insights the AAC spec cannot express without guessing are skipped with a printed reason
+(`SKIP <id>: derived measure (previousPeriodMeasure)`). Read those — they are the
+generator telling you what it refused to invent, not noise.
+
+### 2. `run` — evaluate
+
+```bash
+uv run --package gooddata-eval gd-eval run \
+  --host "$GOODDATA_HOST" --workspace "$WORKSPACE_ID" \
+  --dataset ./datasets/ecommerce --kind visualization \
+  --model gpt-5.2 --model ProviderName/gpt-4o \
+  --runs 3 --gate power --concurrency 4 \
+  --json ./results/run.json --html ./results/run.html
+```
+
+`--dataset` reads a local folder; `--langfuse-dataset` pulls one by name instead. `--kind`
+only supplies a default for items that do not carry their own `test_kind`. Repeat
+`--model` to compare models in one run. `--runs` with `--gate power` measures stability
+(every run must pass) rather than pass@K. `--langfuse` pushes the run as a scored
+experiment, needing `LANGFUSE_HOST`, `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY`.
+
+`--concurrency` is capped for you where it matters: kinds that create workspace objects
+run one at a time regardless, see the parallel-safety gotcha below.
+
+### 3. `report` — compare runs
+
+```bash
+uv run --package gooddata-eval gd-eval report \
+  ./results/*.json -o ./results/comparison.html --title "luna vs 4o" --redact
+```
+
+Several JSON reports become side-by-side columns keyed by file name. `--redact` is the
+customer-safe form: conversation ids, response ids and raw reasoning dropped, model names
+replaced with "Model A", "Model B".
+
+### 4. `models` — what the org has configured
+
+```bash
+uv run --package gooddata-eval gd-eval models --host "$GOODDATA_HOST"
+```
+
+Run this before guessing a `--model` string.
+
+### Environment
+
+| Variable | Used by |
+|---|---|
+| `GOODDATA_TOKEN` | every platform-facing subcommand |
+| `OPENAI_API_KEY` | `generate` phrasing, and the LLM-as-judge evaluators |
+| `GD_EVAL_JUDGE_MODEL` | judge model, same as `--judge-model` |
+| `GD_EVAL_AGENT_ID` | which agent to drive, same as `--agent-id` |
+| `LANGFUSE_HOST`, `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` | `--langfuse`, `--langfuse-dataset` |
+| `GOODDATA_EVAL_CHAT_*` | SSE retry, backoff and timeout knobs |
+| `GD_EVAL_TIMERS` | same as `--timers` |
+
+A gitignored `.env` at the repo root is the normal place for these; load it with
+`set -a && . ./.env && set +a` before the command.
+
 ## Gotchas
 
 **Adding an evaluator is a registry change, not a naming convention.** Single-shot kinds go
@@ -89,6 +190,23 @@ ingestion has no pass/fail signal and inflates or misattributes per-item latency
 (`run_trace_link_inline` is the synchronous alternative). Do not "fix" a slow item by
 making trace scoring synchronous again.
 
+**A generated item's `expected_output` is copied, never invented — keep it that way.**
+`core/dataset/from_insights.py` converts each insight with the platform's own
+`declarative_visualization_to_aac()` (from `gooddata-code-convertors`, via `gooddata-sdk`),
+so the mapping is not ours to get wrong. What is ours is deciding what the evaluator cannot
+yet score — derived measures and measure-level filters convert fine and then compare wrong,
+so they are skipped with a printed reason — and stripping the no-op filters AD saves for an
+"All" selection, which would otherwise let a question claim a filter its chart lacks. Teach
+the comparator about a construct and the matching skip can go; do not make one convert by
+hand. Chart type names are the convertor's, which are also the agent's — do not rename them. One
+granularity is patched in `CONVERTOR_GRANULARITY_FIXES`: `week_us` → `WEEK_US` is a convertor
+bug, the platform enum is `WEEK`.
+
+**The snapshot is a plain-JSON contract.** `--snapshot-in`/`--snapshot-out` is what makes
+the generator testable offline and iterable without re-fetching, and it is why the
+generator reads the declarative analytics model rather than `sdk.visualizations`. Anything
+that changes the fetch shape invalidates every saved snapshot.
+
 **Scoring weights do not sum to 1.** `quality_score` is the fraction of boolean-valued keys
 in `best_detail` that are true, falling back to `pass_at_k` when there are none (text
 evaluators). `value_score` is `0.6 * quality + 0.2 * speed` — the 0.8 total is what the

packages/gooddata-eval/pyproject.toml

@@ -41,6 +41,9 @@ Source = "https://github.com/gooddata/gooddata-python-sdk"
 [dependency-groups]
 dev = [
     "pytest>=8.3.5",
+    # The extra itself, so a plain `uv run` has the phrasing step and the LLM judge.
+    # It stays optional for anyone installing the published package.
+    "gooddata-eval[llm-judge]",
 ]
 test = [
     "pytest~=9.1.1",

packages/gooddata-eval/src/gooddata_eval/cli/main.py

@@ -26,6 +26,7 @@
     normalize_gate,
 )
 from gooddata_eval.core.connection import ConnectionError_, resolve_connection
+from gooddata_eval.core.dataset.from_insights import generate as generate_from_insights
 from gooddata_eval.core.dataset.local import load_local_dataset
 from gooddata_eval.core.evaluators import supported_test_kinds
 from gooddata_eval.core.langfuse.sink import LangfuseSink
@@ -208,6 +209,80 @@ def _build_parser() -> argparse.ArgumentParser:
         "model names with 'Model A', 'Model B', ...",
     )
 
+    gen = sub.add_parser(
+        "generate",
+        help="Generate a visualization dataset by reverse-engineering a workspace's insights.",
+    )
+    gen.add_argument("--host", help="GoodData host URL.")
+    gen.add_argument("--token", help="API token (or set GOODDATA_TOKEN).")
+    gen.add_argument("--profile", help="Profile name in ~/.gooddata/profiles.yaml.")
+    gen.add_argument("--workspace", help="Workspace id to read insights from.")
+    gen.add_argument(
+        "--dataset-name", dest="dataset_name", required=True, help="`dataset_name` written into every item."
+    )
+    gen.add_argument("--out", help="Output folder for the dataset JSON files (default: ./<dataset-name>).")
+    gen.add_argument(
+        "--dashboard",
+        action="append",
+        default=[],
+        help="Restrict to insights placed on this dashboard (repeatable). Default: the whole workspace.",
+    )
+    gen.add_argument(
+        "--snapshot-in", dest="snapshot_in", help="Replay a saved model snapshot instead of calling the API."
+    )
+    gen.add_argument("--snapshot-out", dest="snapshot_out", help="Save the fetched model snapshot for later replay.")
+    gen.add_argument("--langfuse-out", dest="langfuse_out", help="Also write a Langfuse-importable dataset JSON here.")
+    gen.add_argument(
+        "--id-prefix",
+        dest="id_prefix",
+        default="",
+        help="Prefix every exported Langfuse item id. Langfuse ids are unique per PROJECT, so "
+        "carrying an item into a second dataset under its original id is a 409.",
+    )
+    gen.add_argument(
+        "--no-phrase", dest="no_phrase", action="store_true", help="Skip the LLM step; emit mechanical questions."
+    )
+    gen.add_argument(
+        "--phrase-model", dest="phrase_model", default="gpt-4o", help="OpenAI model for the phrasing step."
+    )
+    gen.add_argument(
+        "--no-viz-type", dest="no_viz_type", action="store_true", help="Always blank the expected chart type."
+    )
+    gen.add_argument(
+        "--min-questions", dest="min_questions", type=int, default=15, help="Fail below this many questions."
+    )
+    gen.add_argument(
+        "--min-shapes", dest="min_shapes", type=int, default=3, help="Fail below this many distinct question shapes."
+    )
+    gen.add_argument(
+        "--min-filtered",
+        dest="min_filtered",
+        type=int,
+        default=1,
+        help="Fail below this many questions carrying a filter.",
+    )
+    gen.add_argument(
+        "--enrich-ranked",
+        dest="enrich_ranked",
+        type=int,
+        default=0,
+        metavar="N",
+        help="Additionally derive up to N ranked questions. Best-grounded first: insights whose "
+        "own title promised a ranking their definition never implemented ('Top Returned Reasons' "
+        "saved with no sort) are implemented as the title asks, then ranking filters this "
+        "generator adds to a plain breakdown, then sort-only variants. Use when the workspace has "
+        "no ranked insights of its own. Derived items carry `derived_from` and `derived_basis`. "
+        "Default: 0 (off).",
+    )
+    gen.add_argument(
+        "--skip-ambiguous",
+        dest="skip_ambiguous",
+        action="store_true",
+        help="Drop items whose metric or dimension name matches more than one object in the model "
+        "(loop has six labels titled 'Product Title'). Such a question cannot say which object it "
+        "means, so a defensible answer still scores zero. Reported either way.",
+    )
+    gen.add_argument("--dry-run", dest="dry_run", action="store_true", help="Report only; write nothing.")
     models_cmd = sub.add_parser("models", help="List LLM providers and models configured in the org.")
     models_cmd.add_argument("--host", help="GoodData host URL.")
     models_cmd.add_argument("--token", help="API token (or set GOODDATA_TOKEN).")
@@ -612,6 +687,23 @@ def _report(args: argparse.Namespace) -> int:
     return _EXIT_OK
 
 
+def _generate(args: argparse.Namespace) -> int:
+    """`gd-eval generate` -- reverse-engineer a dataset from a workspace's insights."""
+    if not args.snapshot_in and not args.workspace:
+        print("error: generate needs --workspace, or --snapshot-in to replay a saved model.", file=sys.stderr)
+        return _EXIT_OPERATIONAL_ERROR
+    if args.out is None:
+        args.out = args.dataset_name
+
+    def sdk_factory():
+        from gooddata_sdk import GoodDataSdk  # noqa: PLC0415
+
+        host, token = resolve_connection(host=args.host, token=args.token, profile=args.profile)
+        return GoodDataSdk.create(host, token)
+
+    return generate_from_insights(args, sdk_factory)
+
+
 def main(argv: list[str] | None = None) -> int:
     args = parse_args(argv if argv is not None else sys.argv[1:])
     _apply_timer_flag(getattr(args, "timers", False))
@@ -625,6 +717,9 @@ def main(argv: list[str] | None = None) -> int:
         if args.command == "report":
             return _report(args)
 
+        if args.command == "generate":
+            return _generate(args)
+
         host, token = resolve_connection(host=args.host, token=args.token, profile=args.profile)
         if args.command == "models":
             return _list_models(host, token, getattr(args, "workspace", None))