Skip to content

Commit 53131fb

Browse files
gracexmatinclaude
andcommitted
feat(sources): rss provider phase 3/4 — freshness engine, partition-per-feed exec, table providers, catalog registration
Query surface of the RSS/Atom provider (plan Tasks 11-14): - RssEngine: the per-feed freshness state machine (TTL hit, conditional revalidation, 304 re-arm, refetch, degrade-to-stale) — sole production consumer of egress/fetch/cache, whose dead_code allowances drop here - RssScanExec: one partition per feed, concurrency bounded by max_concurrent, LIMIT gates further fetch launches after acquire, scan deadline degrades a partition to zero rows; reset_state overridden so re-execution (RecursiveQueryExec) starts clean - feeds/items TableProviders: exact pushdown for single-feed equality, IN lists, and single-column equality disjunctions (what DataFusion's simplifier rewrites short IN lists into); pruning matches pushdown - register_rss_tables: one source = one catalog (<name>.main.feeds / <name>.main.items), zero network I/O at registration; the egress policy is constructed in exactly one production place Part 3 of 4 stacked PRs replacing #175 (base: rss-m1-phase2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent cee644d commit 53131fb

7 files changed

Lines changed: 4743 additions & 16 deletions

File tree

crates/skardi/src/sources/providers/rss/egress.rs

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -65,14 +65,6 @@
6565
//! handled means adding a check inside `check_v4`/`check_v6` themselves,
6666
//! not changing anything about how or when they're called.
6767
68-
// This module's only consumer today is `fetch.rs`, and `fetch.rs` in turn
69-
// has no production caller yet — the engine (a later PR in this stack) is
70-
// the first one, and hasn't landed. Until then, everything here outside of
71-
// this module's own tests is unreferenced from a build that excludes test
72-
// code, and `cargo check`/`cargo build` would otherwise flag the whole
73-
// file. Remove this once the engine wires `FeedFetcher` in.
74-
#![allow(dead_code)]
75-
7668
use std::fmt;
7769
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr};
7870
use std::sync::Arc;

crates/skardi/src/sources/providers/rss/engine.rs

Lines changed: 2106 additions & 0 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)