Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
122dc61
docs(spec): design RSS support as native source + skill; demote fetch…
gracexmatin Jul 20, 2026
10e390d
docs(spec): add dialect conformance checking and dialect->schema fiel…
gracexmatin Jul 21, 2026
e01c8f6
docs(spec): drop fetcher probe supplement; fold parser-tolerance evid…
gracexmatin Jul 21, 2026
1c7bd83
docs(spec): rewrite RSS spec as standalone first version, dropping ea…
gracexmatin Jul 21, 2026
c7862e6
docs(spec): add architecture-first v2 of RSS design per review feedback
gracexmatin Jul 22, 2026
6662cfe
docs(spec): restructure RSS v2 following the Open Connector doc archi…
gracexmatin Jul 22, 2026
80db8e5
docs(spec): define auto_news_base two-table archive contract and skil…
gracexmatin Jul 22, 2026
93a2aae
docs(spec): align Goals/Decisions with the downstream contract; show …
gracexmatin Jul 22, 2026
e73cfa2
delete history file
gracexmatin Jul 22, 2026
0ac2d2a
docs(spec): cut rss_scan from scope — registered subscriptions only
gracexmatin Jul 22, 2026
cce0bb5
docs(spec): fold HTML→Markdown into chunk('html') mode instead of a UDF
gracexmatin Jul 22, 2026
7a5c63a
Merge remote-tracking branch 'origin/main' into add_RSS_Feed
gracexmatin Jul 23, 2026
6d01285
docs(spec): give RSS degradation in-band visibility and a push path
gracexmatin Jul 23, 2026
0b000e2
docs(spec): add default-deny egress (SSRF) policy and HTML-sanitizati…
gracexmatin Jul 24, 2026
aed30bc
docs(spec): bound parse-time DoS and document feed-content trust resi…
gracexmatin Jul 26, 2026
c2a4f71
docs(spec): reframe content-risk notes around concrete defenses
gracexmatin Jul 26, 2026
ecd6394
docs(spec): make LIMIT pruning's semantics and side-effect reach expl…
gracexmatin Jul 26, 2026
e500a2c
docs(spec): pin dialect column to feed-rs FeedType granularity
gracexmatin Jul 26, 2026
d5fdda4
docs(spec): specify the ampersand repair, stage sanitation, define wi…
gracexmatin Jul 26, 2026
8ff1fb1
docs(spec): name the external-orchestration dependency behind "contin…
gracexmatin Jul 26, 2026
394cd34
docs(spec): version the engine↔skill contract and anchor its tests in…
gracexmatin Jul 26, 2026
469c4b7
docs(spec): store item content as Markdown, converted inside the prov…
gracexmatin Jul 27, 2026
8c2e279
docs(plan): RSS provider M1+M2 implementation plan
gracexmatin Jul 27, 2026
0eab318
feat(sources): add typed RssConfig and rss feature skeleton
gracexmatin Jul 27, 2026
c042341
fix(sources): assert unknown-field errors name the offending key
gracexmatin Jul 27, 2026
5cfa451
docs(plan): require negative tests to assert the failure reason
gracexmatin Jul 27, 2026
2a43da7
Merge origin/main into add_RSS_Feed
gracexmatin Jul 28, 2026
eccb5e1
change name
gracexmatin Jul 28, 2026
0ca1277
test(sources): cover the opml-only validate path flagged by codecov
gracexmatin Jul 28, 2026
3e80ccc
docs: follow the spec file rename (drop -v2 suffix) in code and plan …
gracexmatin Jul 28, 2026
1eb644f
feat(sources): resolve rss subscriptions from inline list or OPML
gracexmatin Jul 28, 2026
165c131
feat(sources): rss sanitation rungs — reencode, control-strip, lexica…
gracexmatin Jul 28, 2026
0c0cd12
fix(sources): cover the OPML well-formedness branch and the resolve-t…
gracexmatin Jul 28, 2026
826b021
docs(plan): fix the malformed-OPML test input for quick-xml's actual …
gracexmatin Jul 28, 2026
b206a38
feat(sources): deterministic HTML-to-Markdown conversion for rss item…
gracexmatin Jul 28, 2026
c08a926
feat(sources): rss sanitation-ladder parse driver and dialect conform…
gracexmatin Jul 28, 2026
f118631
feat(sources): rss default-deny egress policy with rebinding-safe res…
gracexmatin Jul 28, 2026
de9c27f
fix(sources): rss egress — classify embedded-IPv4 v6 forms, cfg(test)…
gracexmatin Jul 28, 2026
5ca0f6d
docs(plan): close the egress plan's own SSRF gaps and require verifie…
gracexmatin Jul 28, 2026
8048cba
fix(sources): correct the febf::1 test comment's mask arithmetic
gracexmatin Jul 28, 2026
3385c0e
feat(sources): rss field extraction to unified rows with Markdown con…
gracexmatin Jul 28, 2026
faf3066
feat(sources): rss bounded fetcher with conditional GET, retries, egr…
gracexmatin Jul 28, 2026
1ac2edc
fix(sources): keep rss attachments out of item link and identity
gracexmatin Jul 28, 2026
c5621e3
fix(sources): cap retry waits, reject unconditional 304, cover hostna…
gracexmatin Jul 28, 2026
1c132fb
feat(sources): rss feeds/items Arrow schemas with surface-version met…
gracexmatin Jul 28, 2026
a4ec853
fix(sources): correct gzip comment mechanism, harden retry-cap test, …
gracexmatin Jul 28, 2026
a634d81
Merge branch 'rss-task5-sanitize' into add_RSS_Feed
gracexmatin Jul 28, 2026
f6f1a94
docs(plan): extend the ceiling constraint to recursion depth; fix a v…
gracexmatin Jul 28, 2026
a2d663d
wip(sources): partial fixes for the Tasks 5-9 batch review — 2 tests …
gracexmatin Jul 28, 2026
12b44c6
fix(sources): correct extensions_json source/language test fixtures
gracexmatin Jul 29, 2026
837d5b8
fix(sources): bump quick-xml to 0.41 and correct the pin's comment
gracexmatin Jul 29, 2026
6c6b23a
fix(sources): make the HTML depth scan an actual upper bound on DOM d…
gracexmatin Jul 29, 2026
bcfdd62
fix(sources): keep rung 1 a no-op on a truthful legacy encoding decla…
gracexmatin Jul 29, 2026
d4270c7
fix(sources): stop trusting a stale `foreign` count to close a tag
gracexmatin Jul 29, 2026
ae84ea0
fix(sources): treat `\r` as tag whitespace and correct the margin claim
gracexmatin Jul 29, 2026
eeaeb31
fix(sources): decide rung 1 by decoding, not by `is_ascii_compatible`
gracexmatin Jul 29, 2026
7221f8d
Merge the Tasks 5-9 fix round into add_RSS_Feed
gracexmatin Jul 29, 2026
1e9409d
feat(sources): rss per-feed TTL cache with negative caching and obser…
gracexmatin Jul 29, 2026
78ecc1f
fix(sources): bound rss cache map size and correct a literal-exclusiv…
gracexmatin Jul 29, 2026
e415dd9
docs(plan): carry Task 10's two handoffs into Task 11's requirements
gracexmatin Jul 29, 2026
386d0cd
fix(sources): floor max_observations so max_entries=0 keeps observati…
gracexmatin Jul 29, 2026
5d29edd
feat(sources): rss engine freshness state machine with per-feed degra…
gracexmatin Jul 29, 2026
4314208
refactor(sources): drop the rss allow(dead_code) suppressions
gracexmatin Jul 29, 2026
c6f1dfe
fix(sources): re-arm the rss TTL on a 304 whose window was evicted
gracexmatin Jul 29, 2026
fb8cf9d
test(sources): pin the rss engine's uncovered paths and correct the r…
gracexmatin Jul 29, 2026
aa0be48
docs(sources): restate the rss last_error property instead of enumera…
gracexmatin Jul 29, 2026
d381f48
docs(sources): scope the last_error audit's claims to what a grep can…
gracexmatin Jul 29, 2026
6693c7b
fix(sources): resolve rustdoc private_intra_doc_links in rss cache/opml
gracexmatin Jul 29, 2026
932ad5c
docs(plan): state the last_error property with its exception; add car…
gracexmatin Jul 29, 2026
7e63d47
feat(sources): rss partition-per-feed execution plan with limit gating
gracexmatin Jul 30, 2026
5c79d18
feat(sources): rss table providers with exact feed pushdown and parti…
gracexmatin Jul 30, 2026
e94fe71
fix(sources): reset rss scan state on plan re-execution
gracexmatin Jul 30, 2026
6ce305b
feat(sources): register rss sources as two-table catalogs with zero-I…
gracexmatin Jul 30, 2026
629fe97
docs(plan): the sequential limit test does not pin the post-acquire gate
gracexmatin Jul 30, 2026
28a769a
feat(server): wire type=rss through validation, dispatch, and catalog…
gracexmatin Jul 30, 2026
b67868a
feat(cli): register type=rss sources from ctx.yaml
gracexmatin Jul 30, 2026
5a42939
test(sources): rss fixture corpus contract tests with pinned Markdown…
gracexmatin Jul 30, 2026
43c720d
docs: correct two measured feed-rs facts in the spec and plan
gracexmatin Jul 30, 2026
2f05e67
test(sources): rss mock-HTTP integration suite covering acceptance cr…
gracexmatin Jul 30, 2026
d4eb94e
test(sources): pin the latin-1 fixture's recovered text, not just its…
gracexmatin Jul 30, 2026
5fcf291
test(sources): rss end-to-end composition — federated join, idempoten…
gracexmatin Jul 30, 2026
86b5a61
docs(rss): reference doc for the RSS/Atom provider
gracexmatin Jul 30, 2026
8330370
docs(rss): bundled semantics overlay and a sample context
gracexmatin Jul 30, 2026
35ac4af
docs(readme): supported-sources row and architecture note for rss
gracexmatin Jul 30, 2026
4d93c59
feat(rss): prune on a disjunction of feed equalities
gracexmatin Jul 31, 2026
9176f23
fix(rss): a within-TTL feed whose window was evicted refetches
gracexmatin Jul 31, 2026
b6d4c42
fix(rss): state what last_error actually carries, and bound the two w…
gracexmatin Jul 31, 2026
a469eb2
docs(rss): four false published claims and the documentation gaps
gracexmatin Jul 31, 2026
70ab0a1
test(rss): pin the semantics overlay against the schemas, the degrada…
gracexmatin Jul 31, 2026
ef3ce1a
test(rss): pin the three self-validating constants against something …
gracexmatin Jul 31, 2026
34a6121
test(rss): correct three claims about which mechanism catches what
gracexmatin Jul 31, 2026
a7def3a
test(rss): exercise the two unreached retryable statuses and the dept…
gracexmatin Jul 31, 2026
e90de55
docs(rss): drop the intra-doc link to opml, which dangles in featurel…
gracexmatin Aug 1, 2026
31cfb59
Merge origin/main into add_RSS_Feed
gracexmatin Aug 1, 2026
ef87eb5
test(rss): skip the candle composition test when no model is configur…
gracexmatin Aug 1, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
187 changes: 187 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,7 @@ For end-to-end walkthroughs — RAG, recommendations, an agent-native wiki, a si
| InfluxDB 3 | Read | No | Time-series measurements over Arrow Flight SQL | [docs/influxdb/](docs/influxdb/) |
| Open Connector | Read | Yes | SaaS resources as stable SQL tables via a self-hosted [Open Connector](https://github.com/oomol-lab/open-connector) gateway; GitHub pack (repos, issues, PRs, reviews, commits, workflow runs, releases — [guide](docs/open-connector-github.md)), Slack pack (conversations, users, files — [guide](docs/open-connector-slack.md)), `open_connector_query` / `open_connector_scan` UDTFs, filter + limit pushdown, bounded TTL cache (more provider packs rolling out) | [docs/open-connector.md](docs/open-connector.md), [demo](docs/open-connector/) |
| Documents | Read | No | PDF/Office/ODF/image -> per-page markdown, tables, images (local directories or S3 prefixes; `documents` feature) | [docs/documents.md](docs/documents.md) |
| RSS / Atom | Read | Yes | RSS 0.9x/1.0/2.0, Atom, JSON Feed subscriptions as `feeds` (health) + `items` (live window); content stored as Markdown, per-feed TTL cache with conditional GETs, per-feed fault isolation, default-deny SSRF egress guard (`rss` feature) | [docs/rss.md](docs/rss.md) |

---

Expand All @@ -363,6 +364,8 @@ For end-to-end walkthroughs — RAG, recommendations, an agent-native wiki, a si

</details>

Most sources read a store that already holds rows. The **RSS/Atom** source is the one that reaches out over the open web at query time: it fetches each subscription through a per-feed TTL cache with conditional GETs, runs one DataFusion partition per feed so a dead feed degrades alone instead of failing the scan, and guards the fetch with a default-deny SSRF egress policy — because feed URLs are agent-authored input. See [docs/rss.md](docs/rss.md).

---

## Docker
Expand Down
4 changes: 4 additions & 0 deletions crates/server/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ chunking = ["skardi/chunking"]
rag = ["embedding", "chunking"]
# documents source connector (mirrors skardi/documents).
documents = ["skardi/documents"]
# rss source connector (mirrors skardi/rss). The typed `rss:` config block on
# `data_sources[]` parses without it; this flag is what makes a `type: rss`
# source register instead of failing with a build-capability error.
rss = ["skardi/rss"]

[dependencies]
better-auth = { workspace = true }
Expand Down
Loading
Loading