docs(spec): RSS feed support by gracexmatin · Pull Request #163 · SkardiLabs/skardi

gracexmatin · 2026-07-21T14:35:37Z

Body

This PR contributes the design spec for first-class RSS/Atom support (docs/superpowers/specs/2026-07-20-rss-feed-support-design.md). Design-only; implementation follows in three milestones (M1 provider core, M2 surfaces, M3 skill).

What's designed

Many dialects, one representation. RSS 0.9x/1.0/2.0, Atom 0.3/1.0, and JSON Feed all normalize to a single protocol-pinned Arrow schema via feed-rs (Decisions 10, 12).
Conformance checking of fetched data. After every parse, the provider records what the feed claimed to be vs what it parsed as, plus spec-required-field violations and sanitation repairs, in queryable feeds.dialect / dialect_declared / conformance_notes columns — tolerance made visible, never silent (Decision 16).
Annotated schema mapping. A normative dialect → unified-schema Field Mapping table ships in docs/rss.md and as semantics-overlay column descriptions, so agents see each column's per-dialect provenance (Decision 17).
One table-provider, many feeds. A single source binds a subscription list and exposes two fixed tables — feeds (health) and items (live union, feed discriminator column); per-feed tables explicitly rejected (Decisions 1–2).
TTL cache + conditional-request validators. Scan-time fetch through a per-feed cache: within TTL → serve cached, zero network; expired + stored ETag/Last-Modified → revalidate, 304 re-arms cache without reparsing; otherwise full fetch/parse. Only complete parses are ever cached (Decision 5).
Plus: partition-per-feed execution with per-feed fault isolation, rss_scan(url) UDTF, html_to_markdown() UDF, fixture-corpus compatibility ratchet, and the auto_news_base skill contract.

…er demo to supplement The main spec now specifies the complete project: a native read-only type: rss source (two fixed tables, per-feed TTL cache + conditional requests, partition-per-feed execution, visible per-feed degradation, typed config, rss_scan UDTF, html_to_markdown UDF, compatibility strategy) and the auto_news_base skill. The Python-fetcher demo is repositioned as a non-gating exploratory probe whose pain log seeds the parser-compatibility fixture corpus.

…d-mapping annotations to RSS design - Decision 16: detect declared vs parsed dialect, verify spec-required fields, surface deviations in queryable feeds.conformance_notes - Decision 17: dialect -> unified-schema mapping as documented contract, shipped in docs/rss.md and as semantics-overlay column descriptions - feeds table: +dialect, +dialect_declared, +conformance_notes - Field Mapping table (RSS 2.0 / RSS 1.0 / Atom 1.0 / JSON Feed) - Testing + acceptance criteria extended accordingly

…ence loop into main spec

codecov · 2026-07-21T14:47:25Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…rlier-draft comparisons Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- system-context diagram + 5-step interaction walkthrough before any decision - decisions grouped into 7 logical blocks (data model, freshness, execution, fault tolerance/conformance, configuration, parsing, surfaces) - v1 retained as normative reference for schemas, alternatives, acceptance

…tecture - decisions become one-line statements; rationale moves to topical sections (Scan Execution, Freshness and Caching, Parsing/Sanitation/Conformance, ...) - mermaid architecture + sequence diagrams replace ASCII art - fully standalone: schemas, field mapping, failure modes, acceptance criteria all in-document; adds Research Findings and Observability

…l lifecycle The archive the skill renders gains a wire-faithful news_items table alongside news_chunks, so search results stay citable after entries fall out of the live window and history can be re-chunked/re-embedded. Adds a skill-lifecycle contract (subscription edits are config-only; parameter changes rebuild from retained content; re-runs are idempotent and diff-first) plus matching testing/acceptance criteria. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…provider in composition diagram Goals now promise post-assembly maintainability and citation durability; Decisions gains a "Downstream contract" group indexing the two-table archive, subscription-agnostic artifacts, and idempotent rendering. The composition diagram makes the provider and the engine/user-space boundary explicit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Per review: every feed Skardi reads is declared in configuration first. Registration is zero-I/O and the feeds health table covers the preview need. Recorded in Non-goals, Decisions, and Future Extensions; the skill's subscribe-time preview becomes register-then-verify. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The conversion is a bridge between the provider's wire-faithful HTML and markdown-aware chunking, so it lives inside chunk()'s existing mode dispatch as a pre-pass — no new user-facing function registered. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Absorb review feedback on silent incompleteness after exhausted retries: - Stamp every items row with window_status (fresh | revalidated | stale-error) so stale-window serving is visible in the result stream - Prescribe the anti-join absence check in SQL Interfaces and state the polling contract explicitly (nobody polls feeds; consumption is reactive) - Make feeds a pure observation surface (never fetches) and re-arm the TTL on failure too (negative caching, bounded failure fuse); two-step the preview/verify flows accordingly (scan items, then read feeds) - End sync's ingest pipeline with a closing health SELECT whose rows are the response (empty = all healthy, report-only); declare the multi-statement pipeline extension as an M3 dependency - Record result-level warnings as a deferred alternative - Extend unit/mock-HTTP/e2e tests and acceptance criteria (4, 13, 14) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…on contract Feed URLs are agent-authored configuration, so a subscription URL is attacker-influenceable: a prompt-injected agent could add an internal or cloud-metadata address as a "feed". The spec was silent on this. Egress (SSRF): the fetcher default-denies hosts resolving into reserved ranges (loopback/link-local/private/CGNAT/ULA), re-validates on each redirect hop, and connects to the validated IP against DNS rebinding. New logic kept local to the RSS fetcher — no existing helper filters by resolved IP (llm_extract gates by scheme, not address). An egress allowlist for intentional internal feeds is deferred to Future Extensions. Stored content: item content stays wire-faithful HTML; the sanitization obligation is stated to sit with any consumer that renders it — Skardi neither executes nor sanitizes it. Threaded through Decisions, Fetcher, a new Security section, Failure Modes, Acceptance, Testing, Repo Shape, and Documentation Commitments. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

abbccdda · 2026-07-24T14:29:04Z

@gracexmatin Any update?

…duals Follows the egress/SSRF pass (0b000e2) with the remaining content-axis threats surfaced in review. Parse-time DoS: the response-size cap is enforced on the decompressed stream (a compressed payload cannot inflate past it) and the parser runs with DTD/entity expansion disabled (billion-laughs class). New Failure Modes rows and fixture/mock-HTTP tests cover both. Content trust residuals the provider cannot fix, stated as contracts and delegated to the consumer: - LLM consumption: feed content is untrusted input to any reasoning agent; prompt-injection mitigation is the consumer/harness's (treat-as-data, least-privilege, human-confirm side effects). Provider contributes consequence-limiting only (egress + config-diff visibility). - Content authenticity: guid/link/author/published are feed-asserted and unverifiable; item identity is feed-scoped and archive ingest is append-only, so a feed cannot collide across feeds or rewrite history. Enclosure/link fetching: the provider never fetches them; any future feature that does must route through the same egress policy. The bundled semantics overlay now flags content/summary as untrusted so an agent discovers the trust boundary from the schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Marking feed content "untrusted" is not a mitigation — external content is untrusted by definition, an injected agent ignores a label, and the semantics overlay is for data meaning, not trust posture. Dropped the schema-level untrusted-content flag and the "untrusted / delegate to consumer" framing. Each content-risk note now names the actual defense and the layer that owns it: - XSS: contextual output encoding at the render point (the standard, sink-dependent defense the provider cannot perform for the consumer). - Prompt injection: containment, not prevention — least-privilege reading agent + deterministic action gating; egress and human-visible config edits are the provider-side rails. - Authenticity: feed-asserted fields are unverifiable, but identity is feed-scoped and archive ingest is append-only (structural), with provenance surfaced. Parse-time DoS remains the one content risk the provider bounds mechanically (unchanged). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…icit A bare LIMIT (no ORDER BY) short-circuits partition launch, so it serves a nondeterministic subset of feeds — and because fetch and health refresh ride on the scan, pruning also bounds the side effects: un-launched feeds keep aging past the one-TTL observation bound a full scan restores. Documented rather than mechanically "fixed": LIMIT is the caller's work bound, and honoring it is the point — this is requested truncation, not concealed failure. The result's feed column is the scan's coverage manifest; completeness-sensitive reads (the absence check, sync's ingest) use no bare LIMIT; ORDER BY … LIMIT is Top-K and prunes nothing. Absence diagnosis gains a third case (not-scanned, next to legitimately-empty and dead), and the semantics overlay teaches the bare-LIMIT caveat so agents learn it from the schema alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

feed-rs exposes the parsed dialect (Feed.feed_type: Atom/JSON/RSS0/RSS1/ RSS2) but family-level only, and its XML dispatch keys on exactly root element + version attribute — so in-band declared-vs-parsed comparison is tautological. dialect's value set drops to feed_type granularity (atom loses the version split), dialect_declared keeps the version detail (atom-0.3 vs atom-1.0), and the conformance mismatch axis moves to Content-Type vs parsed family. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…re-faithful "Repair naked ampersands" was four words wide open to the classic corruption footgun: text-level rewriting mangles CDATA and already-valid references. Three layers close it: - Behavioral spec: the repair is lexical, not textual — CDATA sections, comments, and PIs pass through untouched; only an & that cannot open a valid reference is rewritten (the five predefined entities and numeric charrefs are left alone; undefined HTML names become &-escaped). - Staged retry: sanitation runs as a cumulative ladder — re-encode, strip control characters, repair ampersands — re-parsing after each rung and stopping at the first success, so a feed receives the minimal repair set and conformance_notes records exactly that set. - Contract: every rung is a byte-level no-op on well-formed input, and fixtures rescued by sanitation pin their expected extracted content — parse-success alone would miss silent corruption. Acceptance criterion 16. wire-faithful is now defined at its storage-contract site: faithful to the HTML fragment the parser extracted — transport encoding removed at extraction, repairs pre-extraction and queryable, nothing altered after — a fragment-level fidelity claim, not byte-identity with the wire, covering items and the news_items archive alike. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…uously updating" The motivation promised continuity without saying who supplies it; the scheduler cut lived only in Non-goals. Now named at the promise and at the operating surface: the provider fetches only when read — query-time freshness is self-serve — while unattended continuity (archive capture, health observation) is the caller's cadence, a scheduled skardi sync running faster than the fastest feed's window roll. The missed-cadence cost becomes a Failure Modes row — entries that scroll out between scans are a permanent capture gap, diagnosed by aging feeds.last_fetch — and the rendered skill README teaches the scheduling note. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…-repo The archive DDL and pipelines render from skardi-skills yet embed the engine's feeds/items column names, and a rendered pipeline, once in a user's context, outlives both repositories — version skew is this design's default failure mode, not an edge case (unlike Open Connector's compiled-in source packs). Four layers close it: - Declared surface (v1): feeds/items evolve additively; an enumerated list of breaking changes bumps an integer rss surface version. - Visible at registration (v1): the active version reaches logs and table metadata — the Open Connector convention (visible, never silent). - Pinned consumer fixture (v1): a canonical render is vendored into the engine's fixture corpus as a representative consumer; acceptance criteria 6/11 run against it in engine CI, so engine-initiated breakage fails in-repo with no cross-repo plumbing. - Load-time handshake (M3, with the skill): rendered pipelines stamp requires: rss/<version>; the loader refuses a mismatch with an error naming both versions and the re-render remedy. Standing cross-repo CI rejected: HEAD×HEAD only — protects neither released pairings nor already-rendered artifacts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ider Reverse the wire-faithful HTML storage decision: items.content/summary (and the archived news_items rows) now hold Markdown produced by a deterministic HTML-to-Markdown pass that runs provider-internally at extraction time, between field extraction and Arrow conversion. Why: the live consumers of item content are prompts, fts5, and embeddings - nobody renders feed HTML - so markup fidelity taxed every read (token noise, markup in indexes, per-query conversion) while serving no one. Converting once at extraction makes results prompt-ready, lets pipelines use the already-implemented chunk('markdown') mode, and narrows the rendering surface: the converter emits no raw HTML, so stored content carries no executable markup. The old design is recorded under Alternatives Considered with the accepted loss stated (source HTML not retained; history cannot be re-converted, only re-chunked/re-embedded). The chunk('html') bridge mode loses its only in-spec consumer and moves to Future Extensions; M2 sheds its engine change. New acceptance criterion 18 pins the converter contract (no raw HTML out, determinism, plain-text passthrough) via hostile-markup fixtures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

abbccdda · 2026-07-27T02:59:32Z

@@ -0,0 +1,564 @@
+# RSS Feed Support Design


nit: this file should be v1, not v2

Brings in the Open Connector GitHub source pack (#168), UDTF string-arg extractor refactor (#169), POST /query endpoint (#158), and the squash-merged spec snapshot from #163 — the branch's newer spec revision is kept as-is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

gracexmatin added 3 commits July 20, 2026 18:11

docs(spec): drop fetcher probe supplement; fold parser-tolerance evid…

e01c8f6

…ence loop into main spec

docs(spec): rewrite RSS spec as standalone first version, dropping ea…

1c7bd83

…rlier-draft comparisons Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

abbccdda reviewed Jul 21, 2026

View reviewed changes

Comment thread docs/superpowers/specs/2026-07-20-rss-feed-support-design.md Outdated

gracexmatin and others added 5 commits July 22, 2026 10:08

delete history file

e73cfa2

BtXin reviewed Jul 22, 2026

View reviewed changes

Comment thread docs/superpowers/specs/2026-07-22-rss-feed-support-design-v2.md Outdated

gracexmatin and others added 2 commits July 22, 2026 22:12

abbccdda self-requested a review July 22, 2026 17:47