Commit 3faf6d4
feat(sources): rss provider phase 2/4 — sanitation ladder, HTML→Markdown, parsing, Arrow schemas, TTL cache
Content chain of the RSS/Atom provider (plan Tasks 5-10):
- byte-level sanitation ladder: encoding repair (incl. a lying declaration
over valid UTF-8), control-char stripping, internal-DTD refusal run on
BOTH raw and sanitized bytes (case-insensitive)
- HTML→Markdown via htmd behind a lexical MAX_HTML_DEPTH=100 pre-scan —
htmd's DOM walk recurses unboundedly and a ~7KB feed with 600 nested
divs aborts the process on a 2MiB Tokio stack; above the ceiling the
content degrades to tag-stripped text
- dialect conformance (RSS 0.9x/1.0/2.0, Atom, JSON Feed) + the feed-rs
parse driver; field extraction per the spec's Field Mapping table
- fixed Arrow schemas: items (17 cols), feeds (15 cols), surface_version
metadata; batch builders routed through the FeedStatus domain
- per-feed TTL cache: LRU-evicted windows, health observations never
evicted, HTTP validators bundled with the window they validate
- docs/rss/semantics.yaml ships here because a schema test pins the
overlay against the two schemas
Part 2 of 4 stacked PRs replacing #175 (base: rss-m1-phase1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>1 parent 6d7067d commit 3faf6d4
11 files changed
Lines changed: 5474 additions & 0 deletions
File tree
- crates/skardi
- src/sources/providers/rss
- fixtures
- docs/rss
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
58 | 61 | | |
59 | 62 | | |
60 | 63 | | |
| |||
128 | 131 | | |
129 | 132 | | |
130 | 133 | | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
131 | 140 | | |
132 | 141 | | |
133 | 142 | | |
| |||
0 commit comments