docs(spec): RSS feed support (#163)

* docs(spec): design RSS support as native source + skill; demote fetcher demo to supplement

The main spec now specifies the complete project: a native read-only
type: rss source (two fixed tables, per-feed TTL cache + conditional
requests, partition-per-feed execution, visible per-feed degradation,
typed config, rss_scan UDTF, html_to_markdown UDF, compatibility
strategy) and the auto_news_base skill. The Python-fetcher demo is
repositioned as a non-gating exploratory probe whose pain log seeds
the parser-compatibility fixture corpus.

* docs(spec): add dialect conformance checking and dialect->schema field-mapping annotations to RSS design

- Decision 16: detect declared vs parsed dialect, verify spec-required
  fields, surface deviations in queryable feeds.conformance_notes
- Decision 17: dialect -> unified-schema mapping as documented contract,
  shipped in docs/rss.md and as semantics-overlay column descriptions
- feeds table: +dialect, +dialect_declared, +conformance_notes
- Field Mapping table (RSS 2.0 / RSS 1.0 / Atom 1.0 / JSON Feed)
- Testing + acceptance criteria extended accordingly

* docs(spec): drop fetcher probe supplement; fold parser-tolerance evidence loop into main spec

* docs(spec): rewrite RSS spec as standalone first version, dropping earlier-draft comparisons

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(spec): add architecture-first v2 of RSS design per review feedback

- system-context diagram + 5-step interaction walkthrough before any decision
- decisions grouped into 7 logical blocks (data model, freshness, execution,
  fault tolerance/conformance, configuration, parsing, surfaces)
- v1 retained as normative reference for schemas, alternatives, acceptance

* docs(spec): restructure RSS v2 following the Open Connector doc architecture

- decisions become one-line statements; rationale moves to topical sections
  (Scan Execution, Freshness and Caching, Parsing/Sanitation/Conformance, ...)
- mermaid architecture + sequence diagrams replace ASCII art
- fully standalone: schemas, field mapping, failure modes, acceptance
  criteria all in-document; adds Research Findings and Observability

* docs(spec): define auto_news_base two-table archive contract and skill lifecycle

The archive the skill renders gains a wire-faithful news_items table
alongside news_chunks, so search results stay citable after entries
fall out of the live window and history can be re-chunked/re-embedded.
Adds a skill-lifecycle contract (subscription edits are config-only;
parameter changes rebuild from retained content; re-runs are
idempotent and diff-first) plus matching testing/acceptance criteria.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(spec): align Goals/Decisions with the downstream contract; show provider in composition diagram

Goals now promise post-assembly maintainability and citation durability;
Decisions gains a "Downstream contract" group indexing the two-table
archive, subscription-agnostic artifacts, and idempotent rendering. The
composition diagram makes the provider and the engine/user-space
boundary explicit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* delete history file

* docs(spec): cut rss_scan from scope — registered subscriptions only

Per review: every feed Skardi reads is declared in configuration first.
Registration is zero-I/O and the feeds health table covers the preview
need. Recorded in Non-goals, Decisions, and Future Extensions; the
skill's subscribe-time preview becomes register-then-verify.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(spec): fold HTML→Markdown into chunk('html') mode instead of a UDF

The conversion is a bridge between the provider's wire-faithful HTML
and markdown-aware chunking, so it lives inside chunk()'s existing
mode dispatch as a pre-pass — no new user-facing function registered.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(spec): give RSS degradation in-band visibility and a push path

Absorb review feedback on silent incompleteness after exhausted retries:

- Stamp every items row with window_status (fresh | revalidated |
  stale-error) so stale-window serving is visible in the result stream
- Prescribe the anti-join absence check in SQL Interfaces and state the
  polling contract explicitly (nobody polls feeds; consumption is reactive)
- Make feeds a pure observation surface (never fetches) and re-arm the
  TTL on failure too (negative caching, bounded failure fuse); two-step
  the preview/verify flows accordingly (scan items, then read feeds)
- End sync's ingest pipeline with a closing health SELECT whose rows are
  the response (empty = all healthy, report-only); declare the
  multi-statement pipeline extension as an M3 dependency
- Record result-level warnings as a deferred alternative
- Extend unit/mock-HTTP/e2e tests and acceptance criteria (4, 13, 14)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(spec): add default-deny egress (SSRF) policy and HTML-sanitization contract

Feed URLs are agent-authored configuration, so a subscription URL is
attacker-influenceable: a prompt-injected agent could add an internal or
cloud-metadata address as a "feed". The spec was silent on this.

Egress (SSRF): the fetcher default-denies hosts resolving into reserved
ranges (loopback/link-local/private/CGNAT/ULA), re-validates on each
redirect hop, and connects to the validated IP against DNS rebinding. New
logic kept local to the RSS fetcher — no existing helper filters by
resolved IP (llm_extract gates by scheme, not address). An egress
allowlist for intentional internal feeds is deferred to Future Extensions.

Stored content: item content stays wire-faithful HTML; the sanitization
obligation is stated to sit with any consumer that renders it — Skardi
neither executes nor sanitizes it.

Threaded through Decisions, Fetcher, a new Security section, Failure
Modes, Acceptance, Testing, Repo Shape, and Documentation Commitments.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(spec): bound parse-time DoS and document feed-content trust residuals

Follows the egress/SSRF pass (0b000e2) with the remaining content-axis
threats surfaced in review.

Parse-time DoS: the response-size cap is enforced on the decompressed
stream (a compressed payload cannot inflate past it) and the parser runs
with DTD/entity expansion disabled (billion-laughs class). New Failure
Modes rows and fixture/mock-HTTP tests cover both.

Content trust residuals the provider cannot fix, stated as contracts and
delegated to the consumer:
- LLM consumption: feed content is untrusted input to any reasoning agent;
  prompt-injection mitigation is the consumer/harness's (treat-as-data,
  least-privilege, human-confirm side effects). Provider contributes
  consequence-limiting only (egress + config-diff visibility).
- Content authenticity: guid/link/author/published are feed-asserted and
  unverifiable; item identity is feed-scoped and archive ingest is
  append-only, so a feed cannot collide across feeds or rewrite history.

Enclosure/link fetching: the provider never fetches them; any future
feature that does must route through the same egress policy.

The bundled semantics overlay now flags content/summary as untrusted so an
agent discovers the trust boundary from the schema.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(spec): reframe content-risk notes around concrete defenses

Marking feed content "untrusted" is not a mitigation — external content is
untrusted by definition, an injected agent ignores a label, and the
semantics overlay is for data meaning, not trust posture. Dropped the
schema-level untrusted-content flag and the "untrusted / delegate to
consumer" framing.

Each content-risk note now names the actual defense and the layer that owns
it:
- XSS: contextual output encoding at the render point (the standard,
  sink-dependent defense the provider cannot perform for the consumer).
- Prompt injection: containment, not prevention — least-privilege reading
  agent + deterministic action gating; egress and human-visible config
  edits are the provider-side rails.
- Authenticity: feed-asserted fields are unverifiable, but identity is
  feed-scoped and archive ingest is append-only (structural), with
  provenance surfaced.

Parse-time DoS remains the one content risk the provider bounds
mechanically (unchanged).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(spec): make LIMIT pruning's semantics and side-effect reach explicit

A bare LIMIT (no ORDER BY) short-circuits partition launch, so it serves a
nondeterministic subset of feeds — and because fetch and health refresh ride
on the scan, pruning also bounds the side effects: un-launched feeds keep
aging past the one-TTL observation bound a full scan restores.

Documented rather than mechanically "fixed": LIMIT is the caller's work
bound, and honoring it is the point — this is requested truncation, not
concealed failure. The result's feed column is the scan's coverage manifest;
completeness-sensitive reads (the absence check, sync's ingest) use no bare
LIMIT; ORDER BY … LIMIT is Top-K and prunes nothing.

Absence diagnosis gains a third case (not-scanned, next to legitimately-empty
and dead), and the semantics overlay teaches the bare-LIMIT caveat so agents
learn it from the schema alone.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(spec): pin dialect column to feed-rs FeedType granularity

feed-rs exposes the parsed dialect (Feed.feed_type: Atom/JSON/RSS0/RSS1/
RSS2) but family-level only, and its XML dispatch keys on exactly root
element + version attribute — so in-band declared-vs-parsed comparison
is tautological. dialect's value set drops to feed_type granularity
(atom loses the version split), dialect_declared keeps the version
detail (atom-0.3 vs atom-1.0), and the conformance mismatch axis moves
to Content-Type vs parsed family.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(spec): specify the ampersand repair, stage sanitation, define wire-faithful

"Repair naked ampersands" was four words wide open to the classic
corruption footgun: text-level rewriting mangles CDATA and already-valid
references. Three layers close it:

- Behavioral spec: the repair is lexical, not textual — CDATA sections,
  comments, and PIs pass through untouched; only an & that cannot open a
  valid reference is rewritten (the five predefined entities and numeric
  charrefs are left alone; undefined HTML names become &amp;-escaped).
- Staged retry: sanitation runs as a cumulative ladder — re-encode, strip
  control characters, repair ampersands — re-parsing after each rung and
  stopping at the first success, so a feed receives the minimal repair set
  and conformance_notes records exactly that set.
- Contract: every rung is a byte-level no-op on well-formed input, and
  fixtures rescued by sanitation pin their expected extracted content —
  parse-success alone would miss silent corruption. Acceptance criterion 16.

wire-faithful is now defined at its storage-contract site: faithful to the
HTML fragment the parser extracted — transport encoding removed at
extraction, repairs pre-extraction and queryable, nothing altered after — a
fragment-level fidelity claim, not byte-identity with the wire, covering
items and the news_items archive alike.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(spec): name the external-orchestration dependency behind "continuously updating"

The motivation promised continuity without saying who supplies it; the
scheduler cut lived only in Non-goals. Now named at the promise and at
the operating surface: the provider fetches only when read — query-time
freshness is self-serve — while unattended continuity (archive capture,
health observation) is the caller's cadence, a scheduled skardi sync
running faster than the fastest feed's window roll. The missed-cadence
cost becomes a Failure Modes row — entries that scroll out between
scans are a permanent capture gap, diagnosed by aging feeds.last_fetch —
and the rendered skill README teaches the scheduling note.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(spec): version the engine↔skill contract and anchor its tests in-repo

The archive DDL and pipelines render from skardi-skills yet embed the
engine's feeds/items column names, and a rendered pipeline, once in a
user's context, outlives both repositories — version skew is this design's
default failure mode, not an edge case (unlike Open Connector's compiled-in
source packs). Four layers close it:

- Declared surface (v1): feeds/items evolve additively; an enumerated list
  of breaking changes bumps an integer rss surface version.
- Visible at registration (v1): the active version reaches logs and table
  metadata — the Open Connector convention (visible, never silent).
- Pinned consumer fixture (v1): a canonical render is vendored into the
  engine's fixture corpus as a representative consumer; acceptance
  criteria 6/11 run against it in engine CI, so engine-initiated breakage
  fails in-repo with no cross-repo plumbing.
- Load-time handshake (M3, with the skill): rendered pipelines stamp
  requires: rss/<version>; the loader refuses a mismatch with an error
  naming both versions and the re-render remedy.

Standing cross-repo CI rejected: HEAD×HEAD only — protects neither
released pairings nor already-rendered artifacts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(spec): store item content as Markdown, converted inside the provider

Reverse the wire-faithful HTML storage decision: items.content/summary
(and the archived news_items rows) now hold Markdown produced by a
deterministic HTML-to-Markdown pass that runs provider-internally at
extraction time, between field extraction and Arrow conversion.

Why: the live consumers of item content are prompts, fts5, and
embeddings - nobody renders feed HTML - so markup fidelity taxed every
read (token noise, markup in indexes, per-query conversion) while
serving no one. Converting once at extraction makes results
prompt-ready, lets pipelines use the already-implemented
chunk('markdown') mode, and narrows the rendering surface: the
converter emits no raw HTML, so stored content carries no executable
markup. The old design is recorded under Alternatives Considered with
the accepted loss stated (source HTML not retained; history cannot be
re-converted, only re-chunked/re-embedded).

The chunk('html') bridge mode loses its only in-spec consumer and moves
to Future Extensions; M2 sheds its engine change. New acceptance
criterion 18 pins the converter contract (no raw HTML out, determinism,
plain-text passthrough) via hostile-markup fixtures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: gracexmatin