Multi-Source Candidate Data Transformer

A deterministic pipeline that ingests candidate data from a structured recruiter CSV and unstructured recruiter notes, merges each candidate into one canonical JSON profile with field-level provenance and confidence, and reshapes the output at runtime via a JSON config — same engine, no code changes between configs.

Wrong-but-confident is worse than honestly-empty. Unknown values become null (or are omitted/error per config) — never guessed.

Pipeline: detect -> extract -> normalize -> merge -> confidence -> project -> validate Fully deterministic — no LLM, no randomness, no network in the core.

---

Setup

Requires Node.js 18+ (developed on v23). Uses tsx to run TypeScript directly, no build step.

npm install

Run (CLI)

# Default config
npx tsx src/cli.ts --csv data/candidates.csv --notes data/recruiter_notes.txt --config configs/default.json

# Custom config (subset + rename + normalize + toggles)
npx tsx src/cli.ts --csv data/candidates.csv --notes data/recruiter_notes.txt --config configs/custom.json

# Graceful-degradation check (malformed input)
npx tsx src/cli.ts --csv data/malformed.csv --config configs/default.json

# Optional: write to a file instead of stdout
npx tsx src/cli.ts --csv data/candidates.csv --notes data/recruiter_notes.txt --config configs/default.json --out out.json

JSON goes to stdout; warnings/errors go to stderr, so stdout stays pipeable.

Flags: --csv/--notes repeatable, --source <file> auto-detects by extension, --config defaults to configs/default.json, --out writes to a file.

Run (web UI)

npx tsx web/server.ts

Then open http://localhost:5173. Lets you edit CSV/notes input, pick a config, and see a side-by-side default vs. custom output comparison. This is a secondary/optional surface — the CLI is the primary one.

Tests

npm test

Four focused unit tests on in-memory fixtures: conflict resolution (CSV wins, notes preserved in conflicts[]), new-candidate creation (no force-merge), confidence agreement boost, and projection genericity across both configs.

---

Config format

{
  "fields": [
    { "path": "full_name",     "type": "string",   "required": true },
    { "path": "primary_email", "from": "emails[0]", "type": "string", "required": true },
    { "path": "phone",         "from": "phones[0]", "type": "string", "normalize": "E164" },
    { "path": "skills",        "from": "skills[].name", "type": "string[]", "normalize": "canonical" },
    { "path": "country",       "from": "location.country", "type": "string", "normalize": "iso-country", "on_missing": "omit" }
  ],
  "include_provenance": false,
  "include_confidence": true,
  "on_missing": "null"
}

from supports key, nested, index, and array-map paths (skills[].name), defaulting to path when omitted. normalize is applied via a registry lookup — generic, no per-field code. on_missing (per-field overrides global) controls null / omit / error behavior.

---

Assumptions

• Phone: numbers without a country code assumed US; unparseable → null.
• Dates: normalized to YYYY-MM; year-only and Present/Current → null (never invent a month or an end date).
• Matching: email is the primary match key; name+company is a fallback only when no email is present; email takes precedence. Union-find grouping, so input order doesn't affect the result.
• Conflict resolution: fixed priority CSV > notes — CSV is the verified system-of-record, notes carry no reliable timestamp to confirm currency. Same priority decides candidate_id. Losing values are kept in conflicts[], never silently dropped.
• Confidence: 0.6 single-source, +0.2 per additional agreeing source (cap 1.0), 0.5 on conflict; overall_confidence = mean of field confidences.
• Notes extraction is high-precision: only anchored signals (Re: name, email regex, labeled Phone:/Skills:, (now|currently) a <title> at <company>, Based in <loc>) — no inference from free prose.
• Provenance/confidence keys reference canonical paths, not renamed projected output paths, to keep the projection engine free of per-field path-mapping.

---

Repository layout

src/
  types.ts          RawRecord, CanonicalProfile, Config
  sources/           SourceParser interface, csvSource, notesSource
  normalize/          phone, date, skills, location + rule registry
  merge/               matchKey (grouping), merge (conflict + confidence)
  projection/           pathResolver, project (the config engine)
  validation/            zod schema (dynamic, from config)
  pipeline.ts        orchestrator
  cli.ts             CLI entrypoint
configs/             default.json, custom.json
data/                candidates.csv, recruiter_notes.txt, malformed.csv
tests/               transformer.test.ts
web/                 minimal dummy frontend