Skip to content

CHANGELOG.md

Latest commit

 

History

History
357 lines (290 loc) · 23.8 KB

CHANGELOG.md

File metadata and controls

357 lines (290 loc) · 23.8 KB
Raw

Changelog

All notable changes to kibble, a mass-knowledge & data platform — consume anything, extract and structure it, and turn it into clean datasets and knowledge bases at scale (discover → acquire → extract → understand → curate → retrieve → tune). Entries are grouped by milestone; the project is pre-1.0 and unversioned.

Auto-taxonomy ([classify], Spec B)

  • Opt-in emergent document topics. With [classify].enabled = true, build tags every catalog document with a hierarchical auto_topic (e.g. Systems › Linux › Memory) + topic_confidence, and writes data/catalog/topics.json. It reuses the train answer-cluster space (rebalance's in-memory centroids when available — matching clusters.json — else clustered fresh), LLM-names each cluster hierarchically, and assigns each document the majority topic of its rows (ties → lowest cluster index). Default off → build stays offline and documents.jsonl byte-identical. Fail-soft (missing/ erroring embed backend, no [ask] LLM, or empty centroids → auto-topics skipped, rest of catalog intact); reporting-only (not read by index/ask); LLM naming is non-deterministic. No new dependency. (Spec A shipped the complete inventory; Spec B makes it self-organizing.)

Catalog completeness (Spec A)

  • build catalogs every source, not just data/raw: data/catalog/documents.jsonl and summary.json now contain one entry per document for all source types — Files/Web/Blog (the Document each already builds) and Dataset/Codebase (one synthesized entry per doc_id, classified on the rows' non-system text). Codebase entries default to role: "code" unless source_defaults or a per-doc override sets one. Reporting-only — the catalog is not read by index/ask (retrieval runs off [index].sources), so this changes the inventory, not what gets retrieved. No new dependency. (Spec B will add emergent auto-topics.)

kibble train — run the trainer against the tune package (Tune B)

  • kibble train runs the user-configured [train].command (an argv list, no shell) against the kibble tune package: it validates the package (config.yaml + at least one sprint/<phase>/ train.jsonl), substitutes placeholders ({config}/{out_dir}/{sprint_dir}/{model} → absolute paths; unknown token = error), then spawns the command with cwd defaulting to [tune].out, streaming its output and propagating its exit code. --dry-run prints the resolved command and runs nothing; -- <extra> appends substituted args. Backend-agnostic — worked examples for unsloth/Kaggle, MLX, and generic commands in docs/TRAIN.md, which also carries the Tune C runbook (upload → train → download adapter → kibble bench the tuned model). No new dependency.

Dogfood cleanups (#43)

  • build counts all sources' documents: stats.total_documents and documents_by_source now count distinct input documents across data/raw and every [[source]] (Files/Dataset/Codebase/ Web/Blog) — previously they counted only the data/raw pile, so a [[source]]-only build reported 0 docs.
  • cluster guards degenerate tiny topics: [cluster].min_cluster_size (default 2) merges any cluster smaller than the threshold into its nearest neighbor after k-means (deterministic; 0/1 = off), so a too-high k no longer leaves singleton topics (raise the threshold to also fold larger runts — e.g. 3 merges size-1/2 clusters). The post-merge topic count can be lower than k. No new dependency.

bench --stream — live research output

  • kibble bench --stream streams the model's output to stderr during method="research" benchmark runs, for live visibility. Research-only (non-research methods are single-shot and unaffected); the scored answer, score, --strict gate, and the printed results table are byte-identical with or without the flag — only live stderr output is added (the streamed turn's latency is measured locally, same as buffered; report.json's wall/latency fields vary run-to-run either way). Opt-in, default off. Reuses llm::chat_turn_stream; no new dependency (#21).

ask --json --stream — NDJSON event streaming

  • kibble ask --json --stream now streams the answer as newline-delimited JSON events — {"type":"search",…} for each agentic retrieval, {"type":"token","text":…} as the model generates, then a terminal {"type":"answer",…} (same fields as the buffered --json object) or {"type":"notfound",…}. Lets programmatic consumers stream an agentic RAG answer; unlike text streaming it does not require a TTY. --json alone is unchanged (single buffered object). No new dependency (#22).

kibble tune — generate the training curriculum from the build

  • kibble tune turns the built dataset into the Unsloth trainer's phased sprint curriculum (sprint/<phase>/train.jsonl) + config.yaml, mapping each [[tune.phase]] to a set of sources. build now emits a per-row provenance sidecar (<split>.sources.jsonl; train.jsonl stays messages-only), which tune reads to split rows by source. config.yaml is written as JSON (valid YAML for the trainer's yaml.safe_load). Closes the build → tune → pack → train loop; training orchestration + adapter-back/eval are deferred follow-ups. No new dependency. docs/TUNE.md.

LLM-named cluster labels — [cluster].llm_labels

  • [cluster].llm_labels = true (opt-in) names each topic via the [ask] LLM from its distinctive terms + a sample (0-old-css → Old vs modern CSS), replacing the term-based labels in clusters.json, the kibble cluster report, and eval's topic breakdown. Applies everywhere clusters.json is written (command, build auto-cluster, rebalance). One LLM call per run; fail-soft — any problem keeps the distinctive-term labels. Default off (zero LLM calls). Centroids/sizes/assignment stay deterministic; only label text varies when on. Reuses the [ask] endpoint; no new dependency (#26).

ask always shows its grounding

  • kibble ask now surfaces the retrieved passages when a (weak local) model answers from the corpus but emits no [n] citation markers — instead of 0 source(s), it prints a References (retrieved — the model didn't mark citations) block (top 3 by score) and a state-aware footer (0 cited · N retrieved · M search). General-knowledge answers (--chance / not-found "try anyway") carry a sharpened ⚠ Not in your corpus …, verify independently. banner, and the JSON output gains a bounded retrieved array. Found by dogfooding (#35), where a correct grounded answer showed zero sources with a 4B local model. Display-layer only — retrieval and the agentic loop are unchanged.

Rebalancing converges in one build (inline / Approach B)

  • [cluster].rebalance now clusters the curated train in-memory, caps over-represented topics against those centroids, and writes them as data/clusters.json — so the eval max_topic_share gate and the rebalance cap read the same centroids and it converges in a single build (fixes #34, where the old "cap against the prior clusters.json, then re-cluster" path oscillated/diverged). The first-build no-op is gone. With rebalance on, clusters.json reflects train-only topics; off, it clusters all splits as before. One embed + one clustering per build; no new dependency. Replaces the Approach-A mechanism from #31.

Topic-balance gate — eval reads clusters.json

  • kibble eval adds a max_topic_share metric (stage C of clustering, #25): it assigns each train answer to its nearest topic centroid from data/clusters.json and gates on the largest topic's share of the training set. Auto-runs when clusters.json + an embed backend are present; fail-soft otherwise (missing file, model mismatch, dimension mismatch, or embed error → the metric is skipped and eval still completes). Counts toward the quality score / overall and is gated by --strict with no extra flag. Reuses the data/embeddings cache (embeds raw train answers). Config: [eval.thresholds].max_topic_share (default 0.60). No new dependency. Build-side topic-aware rebalancing remains a follow-up.

Topic clustering — kibble cluster

  • kibble cluster [--k N] [--json] clusters the built dataset's rows on their answer embeddings (deterministic spherical k-means + k-means++ seeding, no new dep), labels each topic by its distinctive terms, prints a breakdown, and writes data/clusters.json (centroids + labels). Auto-runs during kibble build ([cluster] config, k=12 default; fail-soft — skipped without an embed backend). Centroids are the seam for a future eval topic-balance gate (#25); LLM-named labels are a follow-up (#26). Reuses the data/embeddings cache. docs/CLUSTER.md.
    • Labels filter a small static stopword list so grammar particles (the, and, to, …) never dominate a topic label.

ask token streaming — kibble ask streams live

  • kibble ask now streams the answer token-by-token to the terminal as the model generates it ([ask].stream, default on; --stream/--no-stream; active only on a TTY and not --json). During tool rounds it prints stderr activity notes (⋯ searching corpus …). New llm::chat_turn_stream (SSE via reqwest::Response::chunk() — no new dep) accumulates content + tool-call deltas and returns the same shape as chat_turn, so citations/loop are unchanged. Follow-ups: bench streaming (#21), --json streaming (#22).

Web-augmented ask — kibble ask --web

  • kibble ask "<q>" [--web] [--no-web] gives the model web_search + fetch_page tools (shared websearch module, SearXNG or DuckDuckGo) alongside search_corpus, so it answers from the corpus first and the live web when needed. Citations tag [local] vs [web] (--json sources gain "kind"). [web] config (default, base_url, max_results, allow_hosts, fetch caps); web off = corpus-only (unchanged). fetch_page stays SSRF-guarded
    • capped. No new crate dep; offline tests via localhost mocks. bench reuses the same websearch module.

Ask layer — agentic grounded RAG (kibble ask)

  • kibble ask "<q>" [--k N] [--json] [--chance] [--show-context] answers questions grounded in the indexed corpus: it seeds a search, lets the served LLM call a search_corpus tool up to [ask].max_rounds to dig further, then returns a cited answer ([n] → source) or, when the corpus doesn't cover it, an interactive not-found flow (provide a source & retry / add detail / --chance general-knowledge fallback, flagged). Reuses the retrieval index + a shared llm chat helper (extracted from bench). Needs a served LLM ([ask].base_url); retrieval stays fail-soft. No new crate dep; deterministic offline tests via a scripted-chat seam. docs/ASK.md.

Retrieve layer — hybrid search (kibble index / kibble search)

  • kibble index [path] builds a retrieval index over the corpus (data/raw/data/ingest/ data/extracted, or an explicit path): chunks each document, embeds chunks via the shared data/embeddings cache ([understand.embed]), and builds a hand-rolled BM25 inverted index → data/index/{chunks.jsonl,bm25.json,meta.json}.
  • kibble search "<q>" [--k N] [--json] ranks passages by Reciprocal Rank Fusion of semantic cosine + BM25, returning score + provenance. Fail-soft: with no/unreachable embed backend it runs lexical-only (BM25). No new crate dep; deterministic offline tests. docs/RETRIEVE.md.

Understand layer — embeddings, semantic dedup & leakage

  • New kibble embedding backend ([understand.embed], OpenAI-compatible /embeddings) with a content-hash vector store/cache (data/embeddings/, reused across build→eval) and a hand-rolled SimHash LSH (random hyperplanes → cosine confirm → union-find). No new crate dep.
  • Semantic dedup in build ([curate].semantic_dedup, opt-in): collapses near-paraphrase answers, keeping the longest; reported as dropped_semantic_duplicates.
  • Semantic metrics in eval ([eval].semantic, opt-in): semantic_near_duplicate_rate + semantic_leakage_rate, gated (defaults 0.02 / 0.0 at cosine 0.90) into the score and --strict.
  • Backend: llama-server --embedding (nomic-embed-text, 768-dim) on localhost. Deterministic, offline tests via a stub embedder. docs/UNDERSTAND.md.

.env auto-loading

  • kibble now loads the nearest .env (searching the working directory upward) into the process environment at startup, so secrets (OCR_API_KEY, KIBBLE_API_TOKEN, OPENAI_API_KEY, …) no longer need manual source-ing. Real environment variables always win (a shell export overrides .env). Hand-rolled parser (KEY=VALUE, # comments, optional export prefix, quote-stripping) — no new crate dep; prints a one-line notice when it loads anything. Ships .env.example; .env is gitignored.

Soul / serving-artifact emitter — kibble soul build

  • kibble soul build [--soul] [--out] compiles a versioned soul.toml (persona + system prompt(s)
    • sampling + tools + optional base template) into serving artifacts: generation_config.json, system_prompt.txt, tools.json, chat_template.jinja (marker substitution — {{SOUL_DEFAULT_SYSTEM}} → JSON-escaped prompt), and manifest.json (sha256 — the registry seed). Pure/deterministic, no new deps; ships soul.example.toml + docs/SOUL.md. Automates/versions the hand-assembled serving bundle.

Web crawler — kibble crawl

  • kibble crawl <url> [--depth][--max-pages][--out][--all-hosts] — a Firecrawl-style discovery layer: BFS-crawls a site (same-host default, depth/page limits, robots.txt honored, politeness delay, per-page byte cap, SSRF-guarded, real User-Agent sent) and writes each page as markdown into data/ingest/crawl-<host>/ for build. [crawl] config; docs/CRAWL.md.
  • Static HTML only (no JS rendering — follow-up). Reuses web/serve-guard/scraper; no new deps. Verified against live sites.

MCP server — kibble mcp

  • kibble mcp runs a stdio JSON-RPC 2.0 server (hand-rolled, no MCP SDK dep) exposing the full measure/emit loop as tools an agent/harness (or Claude Code/Desktop) can call: build, eval, bench, soul_build, caps_list, caps_install. initialize/tools/list/tools/call/ping + notifications + correct error codes (-32601/-32602/-32700); tool failures → isError.
  • The harness integration surface (the harness itself is a separate project). stdio = local-trust, no listener; URL sources reuse the fetch SSRF guards; results are JSON. docs/MCP.md has client registration. No new crate deps.

Web-research / tool-execution bench — method = "research"

  • Adds an agentic benchmark method: the model calls web_search / fetch_page, the harness executes them (SearXNG JSON when [bench.search].base_url set, else a DuckDuckGo scrape with uddg decode; page fetch via web::fetch_url_capped + extract_main_text), feeds results back over the OpenAI tools protocol, and the final answer is scored (contains/judge) plus turns / tool-calls / wall-time (the measurable "beat-Perplexity"). max_turns cap; per-case faults score 0 (never abort).
  • Security: fetch_page is SSRF-guarded (serve::url_is_allowed + [bench.search].allow_hosts bypass — model-chosen URLs can't reach loopback/private) and byte-capped. No new deps.

Model benchmark harness — kibble bench

  • kibble bench [--strict] benchmarks a served model (any OpenAI-compatible endpoint) against an extensible suite: [[bench.benchmark]] + a cases.jsonl, scored by contains / mcq / judge, with per-request latency + tokens/sec. Per-benchmark thresholds → overall PASS + 0–100 score; --strict exits nonzero for CI. Writes bench/report.{json,md}; ships a starter suite + docs/BENCH.md; keys env-only; panic-safe on malformed endpoint responses.
  • This is the self-improvement loop's steering signal ("did the model get better / fast enough?"). KIBBLE is the tool, not the harness/model. A live web-research / tool-execution ("beat-Perplexity") benchmark is the flagship follow-up.

Active curation — build produces gate-passing datasets

  • Dedup + leakage-safe split: build now collapses rows with the same normalized answer and splits by answer content (split_bucket(answer_key)) instead of doc_id, so identical answers can never land in two splits — exact train↔valid/test leakage is impossible by construction. stats.json gains dropped_duplicates. On by default; [curate] (dedup, leakage_safe_split) to tune. Consolidated the per-source split/stat logic into one curate_split stage.
  • On the real 59,632-row corpus: duplicate 8.5% → 0%, leakage 14.1% → 0% (5,694 dupes dropped).
  • Curation filters: build also drops malformed (non-SFT-valid), degenerate (empty/echo), and too-short rows ([curate] drop_malformed/drop_degenerate/min_answer_chars, default on/on/20), reported as dropped_filtered. The degeneracy check is clean-aware (matches the post-clean_text content eval reads), so a filtered build clears eval's malformed_rate/degenerate_rate/short_rate by construction (verified: real build → all 0).
  • Fixed eval's balance metric to read stats.json's object-shaped sources (was silently null).
  • Eval near-dup metrics (opt-in [eval].near_dup): MinHash near_duplicate_rate + near_leakage_rate, gated, so the quality gate certifies fuzzy dedup/leakage (build fixes, eval certifies).
  • Opt-in fuzzy near-dup ([curate].near_dedup, default off): a deterministic MinHash + LSH + union-find pass collapses near-paraphrase/boilerplate answers (char k-grams, threshold 0.85), reported as dropped_near_duplicates. Conservative on real prose (~0.6% dropped, raises distinct2); no new deps.
  • eval distinct2 perf: hashes bigrams to u64 instead of storing String pairs — full 53k-row eval went from timing out (>180s) to ~45s.
  • Next (deferred): source rebalancing; DPO/pretraining formats; the MCP/harness track.

Repositioning — mass knowledge & data

  • Dropped the original personal "voice/style" framing. KIBBLE is a general knowledge & data acquisition and curation platform; voice tuning was only the first use-case. Docs reframed.
  • eval repurposed from a voice-fidelity scorer into a dataset-quality gate: kibble eval [--strict] reads the built train/valid/test.jsonl + stats.json, validates SFT readiness, and scores duplication, train↔valid/test leakage, length, source balance, degeneracy, and diversity → report.{json,md} with a 0–100 score; --strict exits nonzero on FAIL so CI can block a bad dataset. The voice scorer (VoiceProfile/LLM judge) is removed. Dataset-shaping prompt defaults de-voiced. (Real-corpus run caught 14% leakage + 8.5% dupes.)

Capability registry — Layer 7.1 (agentic harness, skills-first)

Skill installation applied to the agent itself: consume a source → gain its skills.

  • kibble caps install|scan|list|remove — detect SKILL.md skills in a source (local path or URL; URL reuses fetch::resolve), install them under <root>/skills/<name>/, and maintain <root>/registry.json. [caps].root (default ~/.kibble/caps), --force, --project.
  • Detection prunes nested skills (a skill dir is atomic); frontmatter name/description with dir-name fallback; names sanitized.
  • Install is copy + register only (skill scripts never executed): dests confined under the skills root, all symlinks skipped (no escape, no recursion), cumulative byte cap, sha256 per skill, atomic registry write, collision skip-unless---force.
  • Foundation for the rest of the harness layer (MCP exposure of the registry, capability assembly, autonomous build/tune loop + emry).

Extractor — Layer 3

Turn any acquired artifact into clean text, staged under data/extracted/ for build (fetch → extract → build). Each backend tries a configured OpenAI-compatible endpoint, then a local CLI, then errors. Keys from env only; CLIs via arg-vectors with canonicalized input paths (no shell, no argv-flag injection); inputs size-capped.

  • 3.3 — video + office: ffmpeg extracts a video's audio → transcription; pandoc converts office docs → plain text. extract now covers all artifact kinds (no unsupported gap).
  • 3.2 — PDF: text-layer extraction via pdf-extract; scanned/sparse PDFs render pages with pdftoppm → OCR each → concatenate. Errors (instead of emitting empty) if every page fails.
  • 3.1 — framework + OCR + transcription: kibble extract + artifact detection; image OCR (vision endpoint, tesseract fallback); audio transcription (/audio/transcriptions, whisper fallback); html/text passthrough.

Ingester — Layer 2 (acquire)

  • mega.nz: download + decrypt public file and folder links — hand-rolled MEGA protocol (AES-128 ECB/CBC/CTR, key derivation, node-tree walk) with no heavy crate; offline known-answer tested.
  • 3c — file hosts: Dropbox (dl=1), Google Drive (uc?export), MediaFire (scrape), gofile (guest-token API).
  • 3b — kibble serve: axum HTTP API — POST /ingest → job id, GET /jobs/{id}, GET /health. Bearer-token auth (fail-closed), an SSRF guard (blocks loopback/private/link-local incl. IPv4-mapped IPv6 and userinfo tricks; allow_hosts bypass), in-memory job registry.
  • 3a — kibble fetch: pluggable handler layer — git clone, HuggingFace dataset, open HTTP directory crawl, direct HTTP file, archive extraction (zip / tar.gz, zip-slip & tar-symlink safe). Streamed max_bytes cap; traversal-safe filenames; proxy-aware.

Eval & Kaggle wiring

  • kibble eval: blended voice-fidelity score — deterministic heuristic profile + a configurable LLM judge (Anthropic or OpenAI-compatible) anchored on real held-out posts; writes report.json + report.md. Generate-or-consume samples; keys from env only; judge failures non-fatal.
  • kibble pack: package the build output into a Kaggle dataset bundle ([pack] config: splits, includes, metadata) + zip; kaggle/RUNBOOK.md documents build → pack → upload → train.
  • Polish: configurable eval reference_max; auth header only sent when a key is set (local endpoints need none); refresh = true updates a cached git clone via git pull.

Build pipeline & config

  • [paths] config: data_root / dataset_dir configurable (defaults preserve the layout).
  • Phase 2.6: extract_main_text refinements (<pre> preserved, nested-block de-dupe); host-anchored github detection; Files (.txt/.md dir) and Blog (.html) source types; dataset preserve_code (bypass the prose cleaner for code-bearing datasets).
  • Phase 2.5 (a–c): tokio runtime + proxy-aware net client + kibble.toml source spine; external chat-dataset merge; local codebase scan; git-URL clone; web/URL fetch + main-text extraction.
  • Phase 2 — kibble build: raw docs + configured sources → catalog (role/topic) → chunk → deterministic 78/12/10 split (SHA256(doc_id)) → {messages} JSONL + stats.json; code bypasses the prose cleaner, prose is cleaned at write.

Foundation

  • Phase 1 — kibble clean + kibble ingest: the text-cleaning rules (strip mentions/URLs/hashtags/tickers/subreddits/invites/zero-width/empty-brackets, emails preserved) and local ingest (Twitter GDPR archive, textfiles tree).

Cross-cutting throughout: Rust orchestrator routing to external backends; everything offline-tested (localhost mocks + stub binaries, no real network/tools in tests); responsible acquisition (SSRF/size guards). Design specs and implementation plans live in docs/superpowers/ in the source project.