Skip to content

Commit 6312d4d

Browse files
gracexmatinclaude
andcommitted
feat(sources): rss provider phase 2/4 — sanitation ladder, HTML→Markdown, parsing, Arrow schemas, TTL cache
Content chain of the RSS/Atom provider (plan Tasks 5-10): - byte-level sanitation ladder: encoding repair (incl. a lying declaration over valid UTF-8), control-char stripping, internal-DTD refusal run on BOTH raw and sanitized bytes (case-insensitive) - HTML→Markdown via htmd behind a lexical MAX_HTML_DEPTH=100 pre-scan — htmd's DOM walk recurses unboundedly and a ~7KB feed with 600 nested divs aborts the process on a 2MiB Tokio stack; above the ceiling the content degrades to tag-stripped text - dialect conformance (RSS 0.9x/1.0/2.0, Atom, JSON Feed) + the feed-rs parse driver; field extraction per the spec's Field Mapping table - fixed Arrow schemas: items (17 cols), feeds (15 cols), surface_version metadata; batch builders routed through the FeedStatus domain - per-feed TTL cache: LRU-evicted windows, health observations never evicted, HTTP validators bundled with the window they validate - docs/rss/semantics.yaml ships here because a schema test pins the overlay against the two schemas Part 2 of 4 stacked PRs replacing #175 (base: rss-m1-phase1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent ad2a5b7 commit 6312d4d

11 files changed

Lines changed: 5474 additions & 0 deletions

File tree

Cargo.lock

Lines changed: 177 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

crates/skardi/Cargo.toml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,9 @@ documents = ["dep:liteparse", "dep:glob", "dep:object_store"]
5555
rss = [
5656
"dep:quick-xml",
5757
"reqwest/gzip",
58+
"dep:encoding_rs",
59+
"dep:htmd",
60+
"dep:feed-rs",
5861
]
5962

6063
[dependencies]
@@ -128,6 +131,12 @@ liteparse = { version = "2.4.0", default-features = false, optional = true }
128131
blake3 = "1"
129132
# include_globs matching for the documents connector.
130133
glob = { version = "0.3", optional = true }
134+
# Label lookup + transcoding for the rss sanitation ladder's re-encode rung.
135+
encoding_rs = { version = "0.8", optional = true }
136+
# Deterministic HTML→Markdown for rss item content.
137+
htmd = { version = "0.5.5", optional = true }
138+
# RSS 0.9x/1.0/2.0, Atom, and JSON Feed parsing into one model.
139+
feed-rs = { version = "2.4", optional = true }
131140
# object-store (S3/GCS/Azure) backend for the documents connector's source `path`
132141
# and `image_store`. Only the `aws` scheme is wired in v1 (see blob.rs).
133142
object_store = { workspace = true, optional = true }

0 commit comments

Comments
 (0)